Orionnnn/hub-stats huggingface dataset:
Measuring Epistemic Humility in Multimodal Large Language Models
Authors: name: Bingkui Tong, name: Jiaer Xia, name: Sifeng Shang, name: Kaiyang Zhou
Published: 2025-09-11 17:54:00

Hallucinations in multimodal large language models (MLLMs) -- where the model
generates content inconsistent with the input image -- pose significant risks
in real-world applications, from misinformation in visual question answering to
unsafe errors in decision-making. Existing benchmarks primarily test
recognition accuracy, i.e., evaluating whether models can select the correct
answer among distractors. This overlooks an equally critical capability for
trustworthy AI: recognizing when none of the provided options are correct, a
behavior reflecting epistemic humility. We present HumbleBench, a new
hallucination benchmark designed to evaluate MLLMs' ability to reject plausible
but incorrect answers across three hallucination types: object, relation, and
attribute. Built from a panoptic scene graph dataset, we leverage fine-grained
scene graph annotations to extract ground-truth entities and relations, and
prompt GPT-4-Turbo to generate multiple-choice questions, followed by a
rigorous manual filtering process. Each question includes a "None of the above"
option, requiring models not only to recognize correct visual information but
also to identify when no provided answer is valid. We evaluate a variety of
state-of-the-art MLLMs -- including both general-purpose and specialized
reasoning models -- on HumbleBench and share valuable findings and insights
with the community. By incorporating explicit false-option rejection,
HumbleBench fills a key gap in current evaluation suites, providing a more
realistic measure of MLLM reliability in safety-critical settings. Our code and
dataset are released publicly and can be accessed at
https://github.com/maifoundations/HumbleBench.
UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
Authors: name: Zhengxi Lu, name: Jiabo Ye, name: Fei Tang, name: Yongliang Shen, name: Haiyang Xu, name: Ziwei Zheng, name: Weiming Lu, name: Ming Yan, name: Fei Huang, name: Jun Xiao, name: Yueting Zhuang
Published: 2025-09-15 03:24:08

Graphical User Interface (GUI) agents have demonstrated remarkable progress
in automating complex user interface interactions through reinforcement
learning. However, current approaches face a fundamental dilemma: offline RL
enables stable training on pre-collected trajectories, but struggles with
multi-step task execution for lack of trajectory-level reward signals; online
RL captures these signals through environment interaction, but suffers from
sparse rewards and prohibitive deployment costs. To address it, we present
Semi-online Reinforcement Learning, a novel paradigm that simulates online RL
on offline trajectories. During each rollout process, we preserve the original
model output within the multi-turn dialogue, where a Patch Module adaptively
recovers the divergence between rollout and expert trajectories. To capture
long-term training signals, Semi-online RL introduces discounted future returns
into the reward computation and optimizes the policy with weighted step-level
and episode-level advantages. We further introduce Semi-Online Performance
(SOP), a metric that aligns better with true online performance, serving as a
practical and effective proxy for real-world evaluation. Experiments show that
ours Semi-online RL achieves SOTA performance among 7B models across four
dynamic benchmarks, with significant gains over the base model (e.g., +12.0% on
AndroidWorld, +23.8% on AITW), demonstrating significant progress in bridging
the gap between offline training efficiency and online multi-turn reasoning.
The code is available at https://github.com/X-PLUG/MobileAgent/tree/main/UI-S1.
Learning to Optimize Multi-Objective Alignment Through Dynamic Reward
  Weighting
Authors: name: Yining Lu, name: Zilong Wang, name: Shiyang Li, name: Xin Liu, name: Changlong Yu, name: Qingyu Yin, name: Zhan Shi, name: Zixuan Zhang, name: Meng Jiang
Published: 2025-09-14 21:56:35

Prior works in multi-objective reinforcement learning typically use linear
reward scalarization with fixed weights, which provably fail to capture
non-convex Pareto fronts and thus yield suboptimal results. This limitation
becomes especially critical in online preference alignment for large language
models. Here, stochastic trajectories generated by parameterized policies
create highly non-linear and non-convex mappings from parameters to objectives
that no single static weighting scheme can find optimal trade-offs. We address
this limitation by introducing dynamic reward weighting, which adaptively
adjusts reward weights during the online reinforcement learning process. Unlike
existing approaches that rely on fixed-weight interpolation, our dynamic
weighting continuously balances and prioritizes objectives in training,
facilitating effective exploration of Pareto fronts in objective space. We
introduce two approaches of increasing sophistication and generalizability: (1)
hypervolume-guided weight adaptation and (2) gradient-based weight
optimization, offering a versatile toolkit for online multi-objective
alignment. Our extensive experiments demonstrate their compatibility with
commonly used online reinforcement learning algorithms (including GRPO,
REINFORCE, and RLOO), effectiveness across multiple mathematical reasoning
datasets, and applicability to different model families, consistently achieving
Pareto dominant solutions with fewer training steps than fixed-weight linear
scalarization baselines.
CognitiveSky: Scalable Sentiment and Narrative Analysis for
  Decentralized Social Media
Authors: name: Gaurab Chhetri, name: Anandi Dutta, name: Subasish Das
Published: 2025-09-14 21:37:24

The emergence of decentralized social media platforms presents new
opportunities and challenges for real-time analysis of public discourse. This
study introduces CognitiveSky, an open-source and scalable framework designed
for sentiment, emotion, and narrative analysis on Bluesky, a federated Twitter
or X.com alternative. By ingesting data through Bluesky's Application
Programming Interface (API), CognitiveSky applies transformer-based models to
annotate large-scale user-generated content and produces structured and
analyzable outputs. These summaries drive a dynamic dashboard that visualizes
evolving patterns in emotion, activity, and conversation topics. Built entirely
on free-tier infrastructure, CognitiveSky achieves both low operational cost
and high accessibility. While demonstrated here for monitoring mental health
discourse, its modular design enables applications across domains such as
disinformation detection, crisis response, and civic sentiment analysis. By
bridging large language models with decentralized networks, CognitiveSky offers
a transparent, extensible tool for computational social science in an era of
shifting digital ecosystems.
EthicsMH: A Pilot Benchmark for Ethical Reasoning in Mental Health AI
Authors: name: Sai Kartheek Reddy Kasu
Published: 2025-09-15 07:35:35

The deployment of large language models (LLMs) in mental health and other
sensitive domains raises urgent questions about ethical reasoning, fairness,
and responsible alignment. Yet, existing benchmarks for moral and clinical
decision-making do not adequately capture the unique ethical dilemmas
encountered in mental health practice, where confidentiality, autonomy,
beneficence, and bias frequently intersect. To address this gap, we introduce
Ethical Reasoning in Mental Health (EthicsMH), a pilot dataset of 125 scenarios
designed to evaluate how AI systems navigate ethically charged situations in
therapeutic and psychiatric contexts. Each scenario is enriched with structured
fields, including multiple decision options, expert-aligned reasoning, expected
model behavior, real-world impact, and multi-stakeholder viewpoints. This
structure enables evaluation not only of decision accuracy but also of
explanation quality and alignment with professional norms. Although modest in
scale and developed with model-assisted generation, EthicsMH establishes a task
framework that bridges AI ethics and mental health decision-making. By
releasing this dataset, we aim to provide a seed resource that can be expanded
through community and expert contributions, fostering the development of AI
systems capable of responsibly handling some of society's most delicate
decisions.
Interpretable Physics Reasoning and Performance Taxonomy in
  Vision-Language Models
Authors: name: Pranav Pawar, name: Kavish Shah, name: Akshat Bhalani, name: Komal Kasat, name: Dev Mittal, name: Hadi Gala, name: Deepali Patil, name: Nikita Raichada, name: Monali Deshmukh
Published: 2025-09-10 04:15:01

As Vision-Language Models (VLMs) grow in sophistication, their ability to
perform reasoning is coming under increasing supervision. While they excel at
many tasks, their grasp of fundamental scientific principles, such as physics,
remains an underexplored frontier. To reflect the advancements in these
capabilities, we introduce a novel and accessible framework designed to
rigorously evaluate VLMs on their understanding of 2D physics. Our framework
features a pragmatic scenario generator that creates a diverse testbed of over
400 problems across four core domains: Projectile Motion, Collision Dynamics,
Mechanics, and Fluid Dynamics. Through comprehensive evaluation of four
state-of-the-art VLMs, we demonstrate a strong correlation between model scale
and reasoning ability, with our top-performing model, Qwen2.5-VL-7B, achieving
an overall score of 0.815. We find that while models excel at formulaic
problems, they struggle significantly with domains requiring abstract spatial
reasoning. By designing this framework, we aim to democratize the study of
scientific reasoning in VLMs and foster deeper insights into their capabilities
and limitations.
Context Engineering for Trustworthiness: Rescorla Wagner Steering Under
  Mixed and Inappropriate Contexts
Authors: name: Rushi Wang, name: Jiateng Liu, name: Cheng Qian, name: Yifan Shen, name: Yanzhou Pan, name: Zhaozhuo Xu, name: Ahmed Abbasi, name: Heng Ji, name: Denghui Zhang
Published: 2025-09-02 00:40:34

Incorporating external context can significantly enhance the response quality
of Large Language Models (LLMs). However, real-world contexts often mix
relevant information with disproportionate inappropriate content, posing
reliability risks. How do LLMs process and prioritize mixed context? To study
this, we introduce the Poisoned Context Testbed, pairing queries with
real-world contexts containing relevant and inappropriate content. Inspired by
associative learning in animals, we adapt the Rescorla-Wagner (RW) model from
neuroscience to quantify how competing contextual signals influence LLM
outputs. Our adapted model reveals a consistent behavioral pattern: LLMs
exhibit a strong tendency to incorporate information that is less prevalent in
the context. This susceptibility is harmful in real-world settings, where small
amounts of inappropriate content can substantially degrade response quality.
Empirical evaluations on our testbed further confirm this vulnerability. To
tackle this, we introduce RW-Steering, a two-stage finetuning-based approach
that enables the model to internally identify and ignore inappropriate signals.
Unlike prior methods that rely on extensive supervision across diverse context
mixtures, RW-Steering generalizes robustly across varying proportions of
inappropriate content. Experiments show that our best fine-tuned model improves
response quality by 39.8% and reverses the undesirable behavior curve,
establishing RW-Steering as a robust, generalizable context engineering
solution for improving LLM safety in real-world use.
Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images
Authors: name: Boammani Aser Lompo, name: Marc Haraoui
Published: 2025-09-09 17:52:26

Visual reasoning over structured data such as tables is a critical capability
for modern vision-language models (VLMs), yet current benchmarks remain limited
in scale, diversity, or reasoning depth, especially when it comes to rendered
table images. Addressing this gap, we introduce Visual-TableQA, a large-scale,
open-domain multimodal dataset specifically designed to evaluate and enhance
visual reasoning over complex tabular data. Our generation pipeline is modular,
scalable, and fully autonomous, involving multiple reasoning LLMs collaborating
across distinct roles: generation, validation, and inspiration. Visual-TableQA
comprises 2.5k richly structured LaTeX-rendered tables and 6k
reasoning-intensive QA pairs, all produced at a cost of under USD 100. To
promote diversity and creativity, our pipeline performs multi-model
collaborative data generation via cross-model prompting ('inspiration') and
LLM-jury filtering. Stronger models seed layouts and topics that weaker models
elaborate, collectively distilling diverse reasoning patterns and visual
structures into the dataset. Empirical results show that models fine-tuned on
Visual-TableQA generalize robustly to external benchmarks, outperforming
several proprietary models despite the dataset's synthetic nature. The full
pipeline and resources are publicly available at
https://github.com/AI-4-Everyone/Visual-TableQA.
World Modeling with Probabilistic Structure Integration
Authors: name: Klemen Kotar, name: Wanhee Lee, name: Rahul Venkatesh, name: Honglin Chen, name: Daniel Bear, name: Jared Watrous, name: Simon Kim, name: Khai Loong Aw, name: Lilian Naing Chen, name: Stefan Stojanov, name: Kevin Feigelis, name: Imran Thobani, name: Alex Durango, name: Khaled Jedoui, name: Atlas Kazemian, name: Dan Yamins
Published: 2025-09-10 18:01:04

We present Probabilistic Structure Integration (PSI), a system for learning
richly controllable and flexibly promptable world models from data. PSI
consists of a three-step cycle. The first step, Probabilistic prediction,
involves building a probabilistic graphical model Psi of the data, in the form
of a random-access autoregressive sequence model. Psi supports a complete set
of learned conditional distributions describing the dependence of any variables
in the data on any other set of variables. In step 2, Structure extraction, we
show how to extract underlying low-dimensional properties in the data,
corresponding to a diverse set of meaningful "intermediate structures", in a
zero-shot fashion via causal inference on Psi. Step 3, Integration, completes
the cycle by converting these structures into new token types that are then
continually mixed back into the training diet as conditioning signals and
prediction targets. Each such cycle augments the capabilities of Psi, both
allowing it to model the underlying data better, and creating new control
handles -- akin to an LLM-like universal prompting language. We train an
instance of Psi on 1.4 trillion tokens of internet video data; we use it to
perform a variety of useful video prediction and understanding inferences; we
extract state-of-the-art optical flow, self-supervised depth and object
segmentation; and we use these structures to support a full cycle of predictive
improvements.
CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge
  into Large Language Models
Authors: name: Kairong Han, name: Wenshuo Zhao, name: Ziyu Zhao, name: JunJian Ye, name: Lujia Pan, name: Kun Kuang
Published: 2025-09-01 15:13:15

Large Language Models (LLMs) have achieved remarkable success across various
domains. However, a fundamental question remains: Can LLMs effectively utilize
causal knowledge for prediction and generation? Through empirical studies, we
find that LLMs trained directly on large-scale data often capture spurious
correlations rather than true causal relationships, leading to suboptimal
performance, especially in out-of-distribution (OOD) scenarios. To address this
challenge, we propose Causal Attention Tuning (CAT), a novel approach that
injects fine-grained causal knowledge into the attention mechanism. We propose
an automated pipeline that leverages human priors to automatically generate
token-level causal signals and introduce the Re-Attention mechanism to guide
training, helping the model focus on causal structures while mitigating noise
and biases in attention scores. Experimental results on our proposed Spurious
Token Game (STG) benchmark and multiple downstream tasks demonstrate that our
approach effectively leverages causal knowledge for prediction and remains
robust in OOD scenarios. Implementation details can be found at
https://github.com/Kairong-Han/CAT.
Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs
  for Text Annotation
Authors: name: Joachim Baumann, name: Paul Röttger, name: Aleksandra Urman, name: Albert Wendsjö, name: Flor Miriam Plaza-del-Arco, name: Johannes B. Gruber, name: Dirk Hovy
Published: 2025-09-10 17:58:53

Large language models (LLMs) are rapidly transforming social science research
by enabling the automation of labor-intensive tasks like data annotation and
text analysis. However, LLM outputs vary significantly depending on the
implementation choices made by researchers (e.g., model selection, prompting
strategy, or temperature settings). Such variation can introduce systematic
biases and random errors, which propagate to downstream analyses and cause Type
I, Type II, Type S, or Type M errors. We call this LLM hacking.
  We quantify the risk of LLM hacking by replicating 37 data annotation tasks
from 21 published social science research studies with 18 different models.
Analyzing 13 million LLM labels, we test 2,361 realistic hypotheses to measure
how plausible researcher choices affect statistical conclusions. We find
incorrect conclusions based on LLM-annotated data in approximately one in three
hypotheses for state-of-the-art models, and in half the hypotheses for small
language models. While our findings show that higher task performance and
better general model capabilities reduce LLM hacking risk, even highly accurate
models do not completely eliminate it. The risk of LLM hacking decreases as
effect sizes increase, indicating the need for more rigorous verification of
findings near significance thresholds. Our extensive analysis of LLM hacking
mitigation techniques emphasizes the importance of human annotations in
reducing false positive findings and improving model selection. Surprisingly,
common regression estimator correction techniques are largely ineffective in
reducing LLM hacking risk, as they heavily trade off Type I vs. Type II errors.
  Beyond accidental errors, we find that intentional LLM hacking is
unacceptably simple. With few LLMs and just a handful of prompt paraphrases,
anything can be presented as statistically significant.
DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning
  and Label Distribution Learning
Authors: name: Daniil Ignatev, name: Nan Li, name: Hugh Mee Wong, name: Anh Dang, name: Shane Kaszefski Yaschuk
Published: 2025-09-11 15:04:42

This system paper presents the DeMeVa team's approaches to the third edition
of the Learning with Disagreements shared task (LeWiDi 2025; Leonardelli et
al., 2025). We explore two directions: in-context learning (ICL) with large
language models, where we compare example sampling strategies; and label
distribution learning (LDL) methods with RoBERTa (Liu et al., 2019b), where we
evaluate several fine-tuning methods. Our contributions are twofold: (1) we
show that ICL can effectively predict annotator-specific annotations
(perspectivist annotations), and that aggregating these predictions into soft
labels yields competitive performance; and (2) we argue that LDL methods are
promising for soft label predictions and merit further exploration by the
perspectivist community.
IntrEx: A Dataset for Modeling Engagement in Educational Conversations
Authors: name: Xingwei Tan, name: Mahathi Parvatham, name: Chiara Gambi, name: Gabriele Pergola
Published: 2025-09-08 13:07:35

Engagement and motivation are crucial for second-language acquisition, yet
maintaining learner interest in educational conversations remains a challenge.
While prior research has explored what makes educational texts interesting,
still little is known about the linguistic features that drive engagement in
conversations. To address this gap, we introduce IntrEx, the first large
dataset annotated for interestingness and expected interestingness in
teacher-student interactions. Built upon the Teacher-Student Chatroom Corpus
(TSCC), IntrEx extends prior work by incorporating sequence-level annotations,
allowing for the study of engagement beyond isolated turns to capture how
interest evolves over extended dialogues. We employ a rigorous annotation
process with over 100 second-language learners, using a comparison-based rating
approach inspired by reinforcement learning from human feedback (RLHF) to
improve agreement. We investigate whether large language models (LLMs) can
predict human interestingness judgments. We find that LLMs (7B/8B parameters)
fine-tuned on interestingness ratings outperform larger proprietary models like
GPT-4o, demonstrating the potential for specialised datasets to model
engagement in educational settings. Finally, we analyze how linguistic and
cognitive factors, such as concreteness, comprehensibility (readability), and
uptake, influence engagement in educational dialogues.
FLOWER: Democratizing Generalist Robot Policies with Efficient
  Vision-Language-Action Flow Policies
Authors: name: Moritz Reuss, name: Hongyi Zhou, name: Marcel Rühle, name: Ömer Erdinç Yağmurlu, name: Fabian Otto, name: Rudolf Lioutikov
Published: 2025-09-05 10:43:12

Developing efficient Vision-Language-Action (VLA) policies is crucial for
practical robotics deployment, yet current approaches face prohibitive
computational costs and resource requirements. Existing diffusion-based VLA
policies require multi-billion-parameter models and massive datasets to achieve
strong performance. We tackle this efficiency challenge with two contributions:
intermediate-modality fusion, which reallocates capacity to the diffusion head
by pruning up to 50% of LLM layers, and action-specific Global-AdaLN
conditioning, which cuts parameters by 20% through modular adaptation. We
integrate these advances into a novel 950 M-parameter VLA called FLOWER.
Pretrained in just 200 H100 GPU hours, FLOWER delivers competitive performance
with bigger VLAs across 190 tasks spanning ten simulation and real-world
benchmarks and demonstrates robustness across diverse robotic embodiments. In
addition, FLOWER achieves a new SoTA of 4.53 on the CALVIN ABC benchmark.
Demos, code and pretrained weights are available at
https://intuitive-robots.github.io/flower_vla/.
X-Part: high fidelity and structure coherent shape decomposition
Authors: name: Xinhao Yan, name: Jiachen Xu, name: Yang Li, name: Changfeng Ma, name: Yunhan Yang, name: Chunshi Wang, name: Zibo Zhao, name: Zeqiang Lai, name: Yunfei Zhao, name: Zhuo Chen, name: Chunchao Guo
Published: 2025-09-10 14:37:02

Generating 3D shapes at part level is pivotal for downstream applications
such as mesh retopology, UV mapping, and 3D printing. However, existing
part-based generation methods often lack sufficient controllability and suffer
from poor semantically meaningful decomposition. To this end, we introduce
X-Part, a controllable generative model designed to decompose a holistic 3D
object into semantically meaningful and structurally coherent parts with high
geometric fidelity. X-Part exploits the bounding box as prompts for the part
generation and injects point-wise semantic features for meaningful
decomposition. Furthermore, we design an editable pipeline for interactive part
generation. Extensive experimental results show that X-Part achieves
state-of-the-art performance in part-level shape generation. This work
establishes a new paradigm for creating production-ready, editable, and
structurally sound 3D assets. Codes will be released for public research.