Orionnnn/hub-stats huggingface dataset:
SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
Authors: name: Kunlun Zhu, name: Jiaxun Zhang, name: Ziheng Qi, name: Nuoxing Shang, name: Zijia Liu, name: Peixuan Han, name: Yue Su, name: Haofei Yu, name: Jiaxuan You
Published: 2025-05-29 15:35:58

Recent advancements in large language model (LLM) agents have significantly
accelerated scientific discovery automation, yet concurrently raised critical
ethical and safety concerns. To systematically address these challenges, we
introduce SafeScientist, an innovative AI scientist framework
explicitly designed to enhance safety and ethical responsibility in AI-driven
scientific exploration. SafeScientist proactively refuses ethically
inappropriate or high-risk tasks and rigorously emphasizes safety throughout
the research process. To achieve comprehensive safety oversight, we integrate
multiple defensive mechanisms, including prompt monitoring, agent-collaboration
monitoring, tool-use monitoring, and an ethical reviewer component.
Complementing SafeScientist, we propose SciSafetyBench, a novel
benchmark specifically designed to evaluate AI safety in scientific contexts,
comprising 240 high-risk scientific tasks across 6 domains, alongside 30
specially designed scientific tools and 120 tool-related risk tasks. Extensive
experiments demonstrate that SafeScientist significantly improves safety
performance by 35\% compared to traditional AI scientist frameworks, without
compromising scientific output quality. Additionally, we rigorously validate
the robustness of our safety pipeline against diverse adversarial attack
methods, further confirming the effectiveness of our integrated approach. The
code and data will be available at https://github.com/ulab-uiuc/SafeScientist.
red{Warning: this paper contains example data that may be offensive
or harmful.}
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC
  Videos
Authors: name: Tingyu Song, name: Tongyan Hu, name: Guo Gan, name: Yilun Zhao
Published: 2025-05-29 17:31:13

MLLMs have been widely studied for video question answering recently.
However, most existing assessments focus on natural videos, overlooking
synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in
video generation rely on MLLMs to evaluate the quality of generated videos, but
the capabilities of MLLMs on interpreting AIGC videos remain largely
underexplored. To address this, we propose a new benchmark, VF-Eval, which
introduces four tasks-coherence validation, error awareness, error type
detection, and reasoning evaluation-to comprehensively evaluate the abilities
of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that
even the best-performing model, GPT-4.1, struggles to achieve consistently good
performance across all tasks. This highlights the challenging nature of our
benchmark. Additionally, to investigate the practical applications of VF-Eval
in improving video generation, we conduct an experiment, RePrompt,
demonstrating that aligning MLLMs more closely with human feedback can benefit
video generation.
CXReasonBench: A Benchmark for Evaluating Structured Diagnostic
  Reasoning in Chest X-rays
Authors: name: Hyungyung Lee, name: Geon Choi, name: Jung-Oh Lee, name: Hangyul Yoon, name: Hyuk Gi Hong, name: Edward Choi
Published: 2025-05-23 16:44:21

Recent progress in Large Vision-Language Models (LVLMs) has enabled promising
applications in medical tasks, such as report generation and visual question
answering. However, existing benchmarks focus mainly on the final diagnostic
answer, offering limited insight into whether models engage in clinically
meaningful reasoning. To address this, we present CheXStruct and CXReasonBench,
a structured pipeline and benchmark built on the publicly available
MIMIC-CXR-JPG dataset. CheXStruct automatically derives a sequence of
intermediate reasoning steps directly from chest X-rays, such as segmenting
anatomical regions, deriving anatomical landmarks and diagnostic measurements,
computing diagnostic indices, and applying clinical thresholds. CXReasonBench
leverages this pipeline to evaluate whether models can perform clinically valid
reasoning steps and to what extent they can learn from structured guidance,
enabling fine-grained and transparent assessment of diagnostic reasoning. The
benchmark comprises 18,988 QA pairs across 12 diagnostic tasks and 1,200 cases,
each paired with up to 4 visual inputs, and supports multi-path, multi-stage
evaluation including visual grounding via anatomical region selection and
diagnostic measurements. Even the strongest of 10 evaluated LVLMs struggle with
structured reasoning and generalization, often failing to link abstract
knowledge with anatomically grounded visual interpretation. The code is
available at https://github.com/ttumyche/CXReasonBench
Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study
  of Conditional Effectiveness
Authors: name: Yongjin Yang, name: Euiin Yi, name: Jongwoo Ko, name: Kimin Lee, name: Zhijing Jin, name: Se-Young Yun
Published: 2025-05-29 01:02:55

The remarkable growth in large language model (LLM) capabilities has spurred
exploration into multi-agent systems, with debate frameworks emerging as a
promising avenue for enhanced problem-solving. These multi-agent debate (MAD)
approaches, where agents collaboratively present, critique, and refine
arguments, potentially offer improved reasoning, robustness, and diverse
perspectives over monolithic models. Despite prior studies leveraging MAD, a
systematic understanding of its effectiveness compared to self-agent methods,
particularly under varying conditions, remains elusive. This paper seeks to
fill this gap by conceptualizing MAD as a test-time computational scaling
technique, distinguished by collaborative refinement and diverse exploration
capabilities. We conduct a comprehensive empirical investigation comparing MAD
with strong self-agent test-time scaling baselines on mathematical reasoning
and safety-related tasks. Our study systematically examines the influence of
task difficulty, model scale, and agent diversity on MAD's performance. Key
findings reveal that, for mathematical reasoning, MAD offers limited advantages
over self-agent scaling but becomes more effective with increased problem
difficulty and decreased model capability, while agent diversity shows little
benefit. Conversely, for safety tasks, MAD's collaborative refinement can
increase vulnerability, but incorporating diverse agent configurations
facilitates a gradual reduction in attack success through the collaborative
refinement process. We believe our findings provide critical guidance for the
future development of more effective and strategically deployed MAD systems.
PixelThink: Towards Efficient Chain-of-Pixel Reasoning
Authors: name: Song Wang, name: Gongfan Fang, name: Lingdong Kong, name: Xiangtai Li, name: Jianyun Xu, name: Sheng Yang, name: Qiang Li, name: Jianke Zhu, name: Xinchao Wang
Published: 2025-05-29 17:55:49

Existing reasoning segmentation approaches typically fine-tune multimodal
large language models (MLLMs) using image-text pairs and corresponding mask
labels. However, they exhibit limited generalization to out-of-distribution
scenarios without an explicit reasoning process. Although recent efforts
leverage reinforcement learning through group-relative policy optimization
(GRPO) to enhance reasoning ability, they often suffer from overthinking -
producing uniformly verbose reasoning chains irrespective of task complexity.
This results in elevated computational costs and limited control over reasoning
quality. To address this problem, we propose PixelThink, a simple yet effective
scheme that integrates externally estimated task difficulty and internally
measured model uncertainty to regulate reasoning generation within a
reinforcement learning paradigm. The model learns to compress reasoning length
in accordance with scene complexity and predictive confidence. To support
comprehensive evaluation, we introduce ReasonSeg-Diff, an extended benchmark
with annotated reasoning references and difficulty scores, along with a suite
of metrics designed to assess segmentation accuracy, reasoning quality, and
efficiency jointly. Experimental results demonstrate that the proposed approach
improves both reasoning efficiency and overall segmentation performance. Our
work contributes novel perspectives towards efficient and interpretable
multimodal understanding. The code and model will be publicly available.
IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ
  Tests
Authors: name: Tan-Hanh Pham, name: Phu-Vinh Nguyen, name: Dang The Hung, name: Bui Trong Duong, name: Vu Nguyen Thanh, name: Chris Ngo, name: Tri Quang Truong, name: Truong-Son Hy
Published: 2025-05-17 13:24:08

Although large Vision-Language Models (VLMs) have demonstrated remarkable
performance in a wide range of multimodal tasks, their true reasoning
capabilities on human IQ tests remain underexplored. To advance research on the
fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed
to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the
reasoning capabilities of VLMs, which we argue are more important than the
accuracy of the final prediction. **Our benchmark is visually centric,
minimizing the dependence on unnecessary textual content**, thus encouraging
models to derive answers primarily from image-based information rather than
learned textual knowledge. To this end, we manually collected and annotated 500
visual IQ questions to **prevent unintentional data leakage during training**.
Unlike prior work that focuses primarily on the accuracy of the final answer,
we evaluate the reasoning ability of the models by assessing their explanations
and the patterns used to solve each problem, along with the accuracy of the
final prediction and human evaluation. Our experiments show that there are
substantial performance disparities between tasks, with models such as
`o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest
average accuracies of 0.615, 0.578, and 0.548, respectively. However, all
models struggle with 3D spatial and anagram reasoning tasks, highlighting
significant limitations in current VLMs' general reasoning abilities. In terms
of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet`
achieved top averages of 0.696, 0.586, and 0.516, respectively. These results
highlight inconsistencies between the reasoning processes of the models and
their final answers, emphasizing the importance of evaluating the accuracy of
the reasoning in addition to the final predictions.
Can Large Language Models Infer Causal Relationships from Real-World
  Text?
Authors: name: Ryan Saklad, name: Aman Chadha, name: Oleg Pavlov, name: Raha Moraffah
Published: 2025-05-25 01:50:05

Understanding and inferring causal relationships from texts is a core aspect
of human cognition and is essential for advancing large language models (LLMs)
towards artificial general intelligence. Existing work primarily focuses on
synthetically generated texts which involve simple causal relationships
explicitly mentioned in the text. This fails to reflect the complexities of
real-world tasks. In this paper, we investigate whether LLMs are capable of
inferring causal relationships from real-world texts. We develop a benchmark
drawn from real-world academic literature which includes diverse texts with
respect to length, complexity of relationships (different levels of
explicitness, number of events, and causal relationships), and domains and
sub-domains. To the best of our knowledge, our benchmark is the first-ever
real-world dataset for this task. Our experiments on state-of-the-art LLMs
evaluated on our proposed benchmark demonstrate significant challenges, with
the best-performing model achieving an average F1 score of only 0.477. Analysis
reveals common pitfalls: difficulty with implicitly stated information, in
distinguishing relevant causal factors from surrounding contextual details, and
with connecting causally relevant information spread across lengthy textual
passages. By systematically characterizing these deficiencies, our benchmark
offers targeted insights for further research into advancing LLM causal
reasoning.
Right Side Up? Disentangling Orientation Understanding in MLLMs with
  Fine-grained Multi-axis Perception Tasks
Authors: name: Keanu Nichols, name: Nazia Tasnim, name: Yan Yuting, name: Nicholas Ikechukwu, name: Elva Zou, name: Deepti Ghadiyaram, name: Bryan Plummer
Published: 2025-05-27 18:22:44

Object orientation understanding represents a fundamental challenge in visual
perception critical for applications like robotic manipulation and augmented
reality. Current vision-language benchmarks fail to isolate this capability,
often conflating it with positional relationships and general scene
understanding. We introduce DORI (Discriminative Orientation Reasoning
Intelligence), a comprehensive benchmark establishing object orientation
perception as a primary evaluation target. DORI assesses four dimensions of
orientation comprehension: frontal alignment, rotational transformations,
relative directional relationships, and canonical orientation understanding.
Through carefully curated tasks from 11 datasets spanning 67 object categories
across synthetic and real-world scenarios, DORI provides insights on how
multi-modal systems understand object orientations. Our evaluation of 15
state-of-the-art vision-language models reveals critical limitations: even the
best models achieve only 54.2% accuracy on coarse tasks and 33.0% on granular
orientation judgments, with performance deteriorating for tasks requiring
reference frame shifts or compound rotations. These findings demonstrate the
need for dedicated orientation representation mechanisms, as models show
systematic inability to perform precise angular estimations, track orientation
changes across viewpoints, and understand compound rotations - suggesting
limitations in their internal 3D spatial representations. As the first
diagnostic framework specifically designed for orientation awareness in
multimodal systems, DORI offers implications for improving robotic control, 3D
scene reconstruction, and human-AI interaction in physical environments. DORI
data: https://huggingface.co/datasets/appledora/DORI-Benchmark
Precise In-Parameter Concept Erasure in Large Language Models
Authors: name: Yoav Gur-Arieh, name: Clara Suslik, name: Yihuai Hong, name: Fazl Barez, name: Mor Geva
Published: 2025-05-28 16:58:23

Large language models (LLMs) often acquire knowledge during pretraining that
is undesirable in downstream deployments, e.g., sensitive information or
copyrighted content. Existing approaches for removing such knowledge rely on
fine-tuning, training low-rank adapters or fact-level editing, but these are
either too coarse, too shallow, or ineffective. In this work, we propose PISCES
(Precise In-parameter Suppression for Concept EraSure), a novel framework for
precisely erasing entire concepts from model parameters by directly editing
directions that encode them in parameter space. PISCES uses a disentangler
model to decompose MLP vectors into interpretable features, identifies those
associated with a target concept using automated interpretability techniques,
and removes them from model parameters. Experiments on Gemma 2 and Llama 3.1
over various concepts show that PISCES achieves modest gains in efficacy over
leading erasure methods, reducing accuracy on the target concept to as low as
7.7%, while dramatically improving erasure specificity (by up to 31%) and
robustness (by up to 38%). Overall, these results demonstrate that
feature-based in-parameter editing enables a more precise and reliable approach
for removing conceptual knowledge in language models.
Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit
  Assignment
Authors: name: Siliang Zeng, name: Quan Wei, name: William Brown, name: Oana Frunza, name: Yuriy Nevmyvaka, name: Mingyi Hong
Published: 2025-05-17 04:09:46

This paper investigates approaches to enhance the reasoning capabilities of
Large Language Model (LLM) agents using Reinforcement Learning (RL).
Specifically, we focus on multi-turn tool-use scenarios, which can be naturally
modeled as Markov Decision Processes (MDPs). While existing approaches often
train multi-turn LLM agents with trajectory-level advantage estimation in
bandit settings, they struggle with turn-level credit assignment across
multiple decision steps, limiting their performance on multi-turn reasoning
tasks. To address this, we introduce a fine-grained turn-level advantage
estimation strategy to enable more precise credit assignment in multi-turn
agent interactions. The strategy is general and can be incorporated into
various RL algorithms such as Group Relative Preference Optimization (GRPO).
Our experimental evaluation on multi-turn reasoning and search-based tool-use
tasks with GRPO implementations highlights the effectiveness of the MDP
framework and the turn-level credit assignment in advancing the multi-turn
reasoning capabilities of LLM agents in complex decision-making settings. Our
method achieves 100% success in tool execution and 50% accuracy in exact answer
matching, significantly outperforming baselines, which fail to invoke tools and
achieve only 20-30% exact match accuracy.
Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent
  Approach
Authors: name: Yuchen Wu, name: Edward Sun, name: Kaijie Zhu, name: Jianxun Lian, name: Jose Hernandez-Orallo, name: Aylin Caliskan, name: Jindong Wang
Published: 2025-05-24 21:37:10

Large language models (LLMs) typically generate identical or similar
responses for all users given the same prompt, posing serious safety risks in
high-stakes applications where user vulnerabilities differ widely. Existing
safety evaluations primarily rely on context-independent metrics - such as
factuality, bias, or toxicity - overlooking the fact that the same response may
carry divergent risks depending on the user's background or condition. We
introduce personalized safety to fill this gap and present PENGUIN - a
benchmark comprising 14,000 scenarios across seven sensitive domains with both
context-rich and context-free variants. Evaluating six leading LLMs, we
demonstrate that personalized user information significantly improves safety
scores by 43.2%, confirming the effectiveness of personalization in safety
alignment. However, not all context attributes contribute equally to safety
enhancement. To address this, we develop RAISE - a training-free, two-stage
agent framework that strategically acquires user-specific background. RAISE
improves safety scores by up to 31.6% over six vanilla LLMs, while maintaining
a low interaction cost of just 2.7 user queries on average. Our findings
highlight the importance of selective information gathering in safety-critical
domains and offer a practical solution for personalizing LLM responses without
model retraining. This work establishes a foundation for safety research that
adapts to individual user contexts rather than assuming a universal harm
standard.
FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid
  Control
Authors: name: Younggyo Seo, name: Carmelo Sferrazza, name: Haoran Geng, name: Michal Nauman, name: Zhao-Heng Yin, name: Pieter Abbeel
Published: 2025-05-28 17:55:26

Reinforcement learning (RL) has driven significant progress in robotics, but
its complexity and long training times remain major bottlenecks. In this
report, we introduce FastTD3, a simple, fast, and capable RL algorithm that
significantly speeds up training for humanoid robots in popular suites such as
HumanoidBench, IsaacLab, and MuJoCo Playground. Our recipe is remarkably
simple: we train an off-policy TD3 agent with several modifications -- parallel
simulation, large-batch updates, a distributional critic, and carefully tuned
hyperparameters. FastTD3 solves a range of HumanoidBench tasks in under 3 hours
on a single A100 GPU, while remaining stable during training. We also provide a
lightweight and easy-to-use implementation of FastTD3 to accelerate RL research
in robotics.
Towards Scalable Language-Image Pre-training for 3D Medical Imaging
Authors: name: Chenhui Zhao, name: Yiwei Lyu, name: Asadur Chowdury, name: Edward Harake, name: Akhil Kondepudi, name: Akshay Rao, name: Xinhai Hou, name: Honglak Lee, name: Todd Hollon
Published: 2025-05-28 01:16:34

Language-image pre-training has demonstrated strong performance in 2D medical
imaging, but its success in 3D modalities such as CT and MRI remains limited
due to the high computational demands of volumetric data, which pose a
significant barrier to training on large-scale, uncurated clinical studies. In
this study, we introduce Hierarchical attention for Language-Image Pre-training
(HLIP), a scalable pre-training framework for 3D medical imaging. HLIP adopts a
lightweight hierarchical attention mechanism inspired by the natural hierarchy
of radiology data: slice, scan, and study. This mechanism exhibits strong
generalizability, e.g., +4.3% macro AUC on the Rad-ChestCT benchmark when
pre-trained on CT-RATE. Moreover, the computational efficiency of HLIP enables
direct training on uncurated datasets. Trained on 220K patients with 3.13
million scans for brain MRI and 240K patients with 1.44 million scans for head
CT, HLIP achieves state-of-the-art performance, e.g., +32.4% balanced ACC on
the proposed publicly available brain MRI benchmark Pub-Brain-5; +1.4% and
+6.9% macro AUC on head CT benchmarks RSNA and CQ500, respectively. These
results demonstrate that, with HLIP, directly pre-training on uncurated
clinical datasets is a scalable and effective direction for language-image
pre-training in 3D medical imaging. The code is available at
https://github.com/Zch0414/hlip
FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich
  Document Understanding
Authors: name: Amit Agarwal, name: Srikant Panda, name: Kulbhushan Pachauri
Published: 2025-05-22 22:53:58

In this work, we propose Few Shot Domain Adapting Graph (FS-DAG), a scalable
and efficient model architecture for visually rich document understanding
(VRDU) in few-shot settings. FS-DAG leverages domain-specific and
language/vision specific backbones within a modular framework to adapt to
diverse document types with minimal data. The model is robust to practical
challenges such as handling OCR errors, misspellings, and domain shifts, which
are critical in real-world deployments. FS-DAG is highly performant with less
than 90M parameters, making it well-suited for complex real-world applications
for Information Extraction (IE) tasks where computational resources are
limited. We demonstrate FS-DAG's capability through extensive experiments for
information extraction task, showing significant improvements in convergence
speed and performance compared to state-of-the-art methods. Additionally, this
work highlights the ongoing progress in developing smaller, more efficient
models that do not compromise on performance. Code :
https://github.com/oracle-samples/fs-dag