Orionnnn/hub-stats huggingface dataset:
Hatevolution: What Static Benchmarks Don't Tell Us
Authors: name: Chiara Di Bonaventura, name: Barbara McGillivray, name: Yulan He, name: Albert Meroño-Peñuela
Published: 2025-06-13 18:08:19

Language changes over time, including in the hate speech domain, which
evolves quickly following social dynamics and cultural shifts. While NLP
research has investigated the impact of language evolution on model training
and has proposed several solutions for it, its impact on model benchmarking
remains under-explored. Yet, hate speech benchmarks play a crucial role to
ensure model safety. In this paper, we empirically evaluate the robustness of
20 language models across two evolving hate speech experiments, and we show the
temporal misalignment between static and time-sensitive evaluations. Our
findings call for time-sensitive linguistic benchmarks in order to correctly
and reliably evaluate language models in the hate speech domain.
DoTA-RAG: Dynamic of Thought Aggregation RAG
Authors: name: Saksorn Ruangtanusak, name: Natthapath Rungseesiripak, name: Peerawat Rojratchadakorn, name: Monthol Charattrakool, name: Natapong Nitarach
Published: 2025-06-14 16:56:00

In this paper, we introduce DoTA-RAG (Dynamic-of-Thought Aggregation RAG), a
retrieval-augmented generation system optimized for high-throughput,
large-scale web knowledge indexes. Traditional RAG pipelines often suffer from
high latency and limited accuracy over massive, diverse datasets. DoTA-RAG
addresses these challenges with a three-stage pipeline: query rewriting,
dynamic routing to specialized sub-indexes, and multi-stage retrieval and
ranking. We further enhance retrieval by evaluating and selecting a superior
embedding model, re-embedding the large FineWeb-10BT corpus. Moreover, we
create a diverse Q&A dataset of 500 questions generated via the DataMorgana
setup across a broad range of WebOrganizer topics and formats. DoTA-RAG
improves the answer correctness score from 0.752 (baseline, using LiveRAG
pre-built vector store) to 1.478 while maintaining low latency, and it achieves
a 0.929 correctness score on the Live Challenge Day. These results highlight
DoTA-RAG's potential for practical deployment in domains requiring fast,
reliable access to large and evolving knowledge sources.
Profiling News Media for Factuality and Bias Using LLMs and the
  Fact-Checking Methodology of Human Experts
Authors: name: Zain Muhammad Mujahid, name: Dilshod Azizov, name: Maha Tufail Agro, name: Preslav Nakov
Published: 2025-06-14 15:49:20

In an age characterized by the proliferation of mis- and disinformation
online, it is critical to empower readers to understand the content they are
reading. Important efforts in this direction rely on manual or automatic
fact-checking, which can be challenging for emerging claims with limited
information. Such scenarios can be handled by assessing the reliability and the
political bias of the source of the claim, i.e., characterizing entire news
outlets rather than individual claims or articles. This is an important but
understudied research direction. While prior work has looked into linguistic
and social contexts, we do not analyze individual articles or information in
social media. Instead, we propose a novel methodology that emulates the
criteria that professional fact-checkers use to assess the factuality and
political bias of an entire outlet. Specifically, we design a variety of
prompts based on these criteria and elicit responses from large language models
(LLMs), which we aggregate to make predictions. In addition to demonstrating
sizable improvements over strong baselines via extensive experiments with
multiple LLMs, we provide an in-depth error analysis of the effect of media
popularity and region on model performance. Further, we conduct an ablation
study to highlight the key components of our dataset that contribute to these
improvements. To facilitate future research, we released our dataset and code
at https://github.com/mbzuai-nlp/llm-media-profiling.
EgoPrivacy: What Your First-Person Camera Says About You?
Authors: name: Yijiang Li, name: Genpei Zhang, name: Jiacheng Cheng, name: Yi Li, name: Xiaojun Shan, name: Dashan Gao, name: Jiancheng Lyu, name: Yuan Li, name: Ning Bi, name: Nuno Vasconcelos
Published: 2025-06-13 22:19:54

While the rapid proliferation of wearable cameras has raised significant
concerns about egocentric video privacy, prior work has largely overlooked the
unique privacy threats posed to the camera wearer. This work investigates the
core question: How much privacy information about the camera wearer can be
inferred from their first-person view videos? We introduce EgoPrivacy, the
first large-scale benchmark for the comprehensive evaluation of privacy risks
in egocentric vision. EgoPrivacy covers three types of privacy (demographic,
individual, and situational), defining seven tasks that aim to recover private
information ranging from fine-grained (e.g., wearer's identity) to
coarse-grained (e.g., age group). To further emphasize the privacy threats
inherent to egocentric vision, we propose Retrieval-Augmented Attack, a novel
attack strategy that leverages ego-to-exo retrieval from an external pool of
exocentric videos to boost the effectiveness of demographic privacy attacks. An
extensive comparison of the different attacks possible under all threat models
is presented, showing that private information of the wearer is highly
susceptible to leakage. For instance, our findings indicate that foundation
models can effectively compromise wearer privacy even in zero-shot settings by
recovering attributes such as identity, scene, gender, and race with 70-80%
accuracy. Our code and data are available at
https://github.com/williamium3000/ego-privacy.
Uncertainty-Aware Remaining Lifespan Prediction from Images
Authors: name: Tristan Kenneweg, name: Philip Kenneweg, name: Barbara Hammer
Published: 2025-06-16 12:47:37

Predicting mortality-related outcomes from images offers the prospect of
accessible, noninvasive, and scalable health screening. We present a method
that leverages pretrained vision transformer foundation models to estimate
remaining lifespan from facial and whole-body images, alongside robust
uncertainty quantification. We show that predictive uncertainty varies
systematically with the true remaining lifespan, and that this uncertainty can
be effectively modeled by learning a Gaussian distribution for each sample. Our
approach achieves state-of-the-art mean absolute error (MAE) of 7.48 years on
an established Dataset, and further improves to 4.79 and 5.07 years MAE on two
new, higher-quality datasets curated and published in this work. Importantly,
our models provide well-calibrated uncertainty estimates, as demonstrated by a
bucketed expected calibration error of 0.62 years. While not intended for
clinical deployment, these results highlight the potential of extracting
medically relevant signals from images. We make all code and datasets available
to facilitate further research.
VGR: Visual Grounded Reasoning
Authors: name: Jiacong Wang, name: Zijiang Kang, name: Haochen Wang, name: Haiyong Jiang, name: Jiawen Li, name: Bohong Wu, name: Ya Wang, name: Jiao Ran, name: Xiao Liang, name: Chao Feng, name: Jun Xiao
Published: 2025-06-13 17:47:43

In the field of multimodal chain-of-thought (CoT) reasoning, existing
approaches predominantly rely on reasoning on pure language space, which
inherently suffers from language bias and is largely confined to math or
science domains. This narrow focus limits their ability to handle complex
visual reasoning tasks that demand comprehensive understanding of image
details. To address these limitations, this paper introduces VGR, a novel
reasoning multimodal large language model (MLLM) with enhanced fine-grained
visual perception capabilities. Unlike traditional MLLMs that answer the
question or reasoning solely on the language space, our VGR first detects
relevant regions that may help to solve problems, and then provides precise
answers based on replayed image regions. To achieve this, we conduct a
large-scale SFT dataset called VGR -SFT that contains reasoning data with mixed
vision grounding and language deduction. The inference pipeline of VGR allows
the model to choose bounding boxes for visual reference and a replay stage is
introduced to integrates the corresponding regions into the reasoning process,
enhancing multimodel comprehension. Experiments on the LLaVA-NeXT-7B baseline
show that VGR achieves superior performance on multi-modal benchmarks requiring
comprehensive image detail understanding. Compared to the baseline, VGR uses
only 30\% of the image token count while delivering scores of +4.1 on MMStar,
+7.1 on AI2D, and a +12.9 improvement on ChartQA.
SRLAgent: Enhancing Self-Regulated Learning Skills through Gamification
  and LLM Assistance
Authors: name: Wentao Ge, name: Yuqing Sun, name: Ziyan Wang, name: Haoyue Zheng, name: Weiyang He, name: Piaohong Wang, name: Qianyu Zhu, name: Benyou Wang
Published: 2025-06-11 17:45:03

Self-regulated learning (SRL) is crucial for college students navigating
increased academic demands and independence. Insufficient SRL skills can lead
to disorganized study habits, low motivation, and poor time management,
undermining learners ability to thrive in challenging environments. Through a
formative study involving 59 college students, we identified key challenges
students face in developing SRL skills, including difficulties with
goal-setting, time management, and reflective learning. To address these
challenges, we introduce SRLAgent, an LLM-assisted system that fosters SRL
skills through gamification and adaptive support from large language models
(LLMs). Grounded in Zimmermans three-phase SRL framework, SRLAgent enables
students to engage in goal-setting, strategy execution, and self-reflection
within an interactive game-based environment. The system offers real-time
feedback and scaffolding powered by LLMs to support students independent study
efforts. We evaluated SRLAgent using a between-subjects design, comparing it to
a baseline system (SRL without Agent features) and a traditional multimedia
learning condition. Results showed significant improvements in SRL skills
within the SRLAgent group (p < .001, Cohens d = 0.234) and higher engagement
compared to the baselines. This work highlights the value of embedding SRL
scaffolding and real-time AI support within gamified environments, offering
design implications for educational technologies that aim to promote deeper
learning and metacognitive skill development.
Incorporating Domain Knowledge into Materials Tokenization
Authors: name: Yerim Oh, name: Jun-Hyung Park, name: Junho Kim, name: SungHo Kim, name: SangKeun Lee
Published: 2025-06-09 04:59:13

While language models are increasingly utilized in materials science, typical
models rely on frequency-centric tokenization methods originally developed for
natural language processing. However, these methods frequently produce
excessive fragmentation and semantic loss, failing to maintain the structural
and semantic integrity of material concepts. To address this issue, we propose
MATTER, a novel tokenization approach that integrates material knowledge into
tokenization. Based on MatDetector trained on our materials knowledge base and
a re-ranking method prioritizing material concepts in token merging, MATTER
maintains the structural integrity of identified material concepts and prevents
fragmentation during tokenization, ensuring their semantic meaning remains
intact. The experimental results demonstrate that MATTER outperforms existing
tokenization methods, achieving an average performance gain of 4% and 2%
in the generation and classification tasks, respectively. These results
underscore the importance of domain knowledge for tokenization strategies in
scientific text processing. Our code is available at
https://github.com/yerimoh/MATTER
QGuard:Question-based Zero-shot Guard for Multi-modal LLM Safety
Authors: name: Taegyeong Lee, name: Jeonghwa Yoo, name: Hyoungseo Cho, name: Soo Yong Kim, name: Yunho Maeng
Published: 2025-06-14 01:23:50

The recent advancements in Large Language Models(LLMs) have had a significant
impact on a wide range of fields, from general domains to specialized areas.
However, these advancements have also significantly increased the potential for
malicious users to exploit harmful and jailbreak prompts for malicious attacks.
Although there have been many efforts to prevent harmful prompts and jailbreak
prompts, protecting LLMs from such malicious attacks remains an important and
challenging task. In this paper, we propose QGuard, a simple yet effective
safety guard method, that utilizes question prompting to block harmful prompts
in a zero-shot manner. Our method can defend LLMs not only from text-based
harmful prompts but also from multi-modal harmful prompt attacks. Moreover, by
diversifying and modifying guard questions, our approach remains robust against
the latest harmful prompts without fine-tuning. Experimental results show that
our model performs competitively on both text-only and multi-modal harmful
datasets. Additionally, by providing an analysis of question prompting, we
enable a white-box analysis of user inputs. We believe our method provides
valuable insights for real-world LLM services in mitigating security risks
associated with harmful prompts.
LETS Forecast: Learning Embedology for Time Series Forecasting
Authors: name: Abrar Majeedi, name: Viswanatha Reddy Gajjala, name: Satya Sai Srinath Namburi GNVV, name: Nada Magdi Elkordi, name: Yin Li
Published: 2025-06-06 18:24:12

Real-world time series are often governed by complex nonlinear dynamics.
Understanding these underlying dynamics is crucial for precise future
prediction. While deep learning has achieved major success in time series
forecasting, many existing approaches do not explicitly model the dynamics. To
bridge this gap, we introduce DeepEDM, a framework that integrates nonlinear
dynamical systems modeling with deep neural networks. Inspired by empirical
dynamic modeling (EDM) and rooted in Takens' theorem, DeepEDM presents a novel
deep model that learns a latent space from time-delayed embeddings, and employs
kernel regression to approximate the underlying dynamics, while leveraging
efficient implementation of softmax attention and allowing for accurate
prediction of future time steps. To evaluate our method, we conduct
comprehensive experiments on synthetic data of nonlinear dynamical systems as
well as real-world time series across domains. Our results show that DeepEDM is
robust to input noise, and outperforms state-of-the-art methods in forecasting
accuracy. Our code is available at: https://abrarmajeedi.github.io/deep_edm.
TaskCraft: Automated Generation of Agentic Tasks
Authors: name: Dingfeng Shi, name: Jingyi Cao, name: Qianben Chen, name: Weichen Sun, name: Weizhen Li, name: Hongxuan Lu, name: Fangchen Dong, name: Tianrui Qin, name: King Zhu, name: Minghao Yang, name: Jian Yang, name: Ge Zhang, name: Jiaheng Liu, name: Changwang Zhang, name: Jun Wang, name: Yuchen Eleanor Jiang, name: Wangchunshu Zhou
Published: 2025-06-11 17:58:14

Agentic tasks, which require multi-step problem solving with autonomy, tool
use, and adaptive reasoning, are becoming increasingly central to the
advancement of NLP and AI. However, existing instruction data lacks tool
interaction, and current agentic benchmarks rely on costly human annotation,
limiting their scalability. We introduce TaskCraft, an automated
workflow for generating difficulty-scalable, multi-tool, and verifiable agentic
tasks with execution trajectories. TaskCraft expands atomic tasks using
depth-based and width-based extensions to create structurally and
hierarchically complex challenges. Empirical results show that these tasks
improve prompt optimization in the generation workflow and enhance supervised
fine-tuning of agentic foundation models. We present a large-scale synthetic
dataset of approximately 36,000 tasks with varying difficulty to support future
research on agent tuning and evaluation.
ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm
  Engineering
Authors: name: Yuki Imajuku, name: Kohki Horie, name: Yoichi Iwata, name: Kensho Aoki, name: Naohiro Takahashi, name: Takuya Akiba
Published: 2025-06-10 17:59:56

How well do AI systems perform in algorithm engineering for hard optimization
problems in domains such as package-delivery routing, crew scheduling, factory
production planning, and power-grid balancing? We introduce ALE-Bench, a new
benchmark for evaluating AI systems on score-based algorithmic programming
contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench
presents optimization problems that are computationally hard and admit no known
exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench
encourages iterative solution refinement over long time horizons. Our software
framework supports interactive agent architectures that leverage test-run
feedback and visualizations. Our evaluation of frontier LLMs revealed that
while they demonstrate high performance on specific problems, a notable gap
remains compared to humans in terms of consistency across problems and
long-horizon problem-solving capabilities. This highlights the need for this
benchmark to foster future AI advancements.
AI-Facilitated Analysis of Abstracts and Conclusions: Flagging Unsubstantiated Claims and Ambiguous Pronouns
Authors: name: Evgeny Markhasin
Published: 2025-06-16 07:34:31

We present and evaluate a suite of proof-of-concept (PoC), structured workflow prompts designed to elicit human-like hierarchical reasoning while guiding Large Language Models (LLMs) in the high-level semantic and linguistic analysis of scholarly manuscripts. The prompts target two non-trivial analytical tasks within academic summaries (abstracts and conclusions): identifying unsubstantiated claims (informational integrity) and flagging semantically confusing ambiguous pronoun references (linguistic clarity). We conducted a systematic, multi-run evaluation on two frontier models (Gemini Pro 2.5 Pro and ChatGPT Plus o3) under varied context conditions. Our results for the informational integrity task reveal a significant divergence in model performance: while both models successfully identified an unsubstantiated head of a noun phrase (95% success), ChatGPT consistently failed (0% success) to identify an unsubstantiated adjectival modifier that Gemini correctly flagged (95% success), raising a question regarding the potential influence of the target's syntactic role. For the linguistic analysis task, both models performed well (80-90% success) with full manuscript context. Surprisingly, in a summary-only setting, Gemini's performance was substantially degraded, while ChatGPT achieved a perfect (100%) success rate. Our findings suggest that while structured prompting is a viable methodology for complex textual analysis, prompt performance may be highly dependent on the interplay between the model, task type, and context, highlighting the need for rigorous, model-specific testing.
AI Agent Behavioral Science
Authors: name: Lin Chen, name: Yunke Zhang, name: Jie Feng, name: Haoye Chai, name: Honglin Zhang, name: Bingbing Fan, name: Yibo Ma, name: Shiyuan Zhang, name: Nian Li, name: Tianhui Liu, name: Nicholas Sukiennik, name: Keyu Zhao, name: Yu Li, name: Ziyi Liu, name: Fengli Xu, name: Yong Li
Published: 2025-06-04 08:12:32

Recent advances in large language models (LLMs) have enabled the development
of AI agents that exhibit increasingly human-like behaviors, including
planning, adaptation, and social dynamics across diverse, interactive, and
open-ended scenarios. These behaviors are not solely the product of the
internal architectures of the underlying models, but emerge from their
integration into agentic systems operating within specific contexts, where
environmental factors, social cues, and interaction feedbacks shape behavior
over time. This evolution necessitates a new scientific perspective: AI Agent
Behavioral Science. Rather than focusing only on internal mechanisms, this
perspective emphasizes the systematic observation of behavior, design of
interventions to test hypotheses, and theory-guided interpretation of how AI
agents act, adapt, and interact over time. We systematize a growing body of
research across individual agent, multi-agent, and human-agent interaction
settings, and further demonstrate how this perspective informs responsible AI
by treating fairness, safety, interpretability, accountability, and privacy as
behavioral properties. By unifying recent findings and laying out future
directions, we position AI Agent Behavioral Science as a necessary complement
to traditional model-centric approaches, providing essential tools for
understanding, evaluating, and governing the real-world behavior of
increasingly autonomous AI systems.
Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
Authors: name: Shulin Tian, name: Ruiqi Wang, name: Hongming Guo, name: Penghao Wu, name: Yuhao Dong, name: Xiuying Wang, name: Jingkang Yang, name: Hao Zhang, name: Hongyuan Zhu, name: Ziwei Liu
Published: 2025-06-16 16:17:08

We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e.,
in days and weeks) egocentric videos, which leverages a structured
Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained
via reinforcement learning (RL). Inspired by human problem-solving strategies,
CoTT decomposes complex reasoning into modular steps, with the RL agent
invoking specific tools, one per step, to iteratively and collaboratively
answer sub-questions tackling such tasks as temporal retrieval and multi-modal
understanding. We design a two-stage training paradigm involving supervised
finetuning (SFT) of a pretrained language model using CoTT data and RL to
enable our agent to dynamically propose step-by-step tools for long-range
reasoning. To facilitate training, we construct a dataset called Ego-R1 Data,
which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our
Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark,
Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources.
Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought
reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of
understanding ultra-long egocentric videos, significantly extending the time
coverage from few hours to a week.