Orionnnn/hub-stats huggingface dataset:
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism
  Elucidation and Reasoning
Authors: name: Ruiling Xu, name: Yifan Zhang, name: Qingyun Wang, name: Carl Edwards, name: Heng Ji
Published: 2025-10-09 03:13:31

Organic reaction mechanisms are the stepwise elementary reactions by which
reactants form intermediates and products, and are fundamental to understanding
chemical reactivity and designing new molecules and reactions. Although large
language models (LLMs) have shown promise in understanding chemical tasks such
as synthesis design, it is unclear to what extent this reflects genuine
chemical reasoning capabilities, i.e., the ability to generate valid
intermediates, maintain chemical consistency, and follow logically coherent
multi-step pathways. We address this by introducing oMeBench, the first
large-scale, expert-curated benchmark for organic mechanism reasoning in
organic chemistry. It comprises over 10,000 annotated mechanistic steps with
intermediates, type labels, and difficulty ratings. Furthermore, to evaluate
LLM capability more precisely and enable fine-grained scoring, we propose oMeS,
a dynamic evaluation framework that combines step-level logic and chemical
similarity. We analyze the performance of state-of-the-art LLMs, and our
results show that although current models display promising chemical intuition,
they struggle with correct and consistent multi-step reasoning. Notably, we
find that using prompting strategy and fine-tuning a specialist model on our
proposed dataset increases performance by 50% over the leading closed-source
model. We hope that oMeBench will serve as a rigorous foundation for advancing
AI systems toward genuine chemical reasoning.
The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses
  Against Llm Jailbreaks and Prompt Injections
Authors: name: Milad Nasr, name: Nicholas Carlini, name: Chawin Sitawarin, name: Sander V. Schulhoff, name: Jamie Hayes, name: Michael Ilie, name: Juliette Pluto, name: Shuang Song, name: Harsh Chaudhari, name: Ilia Shumailov, name: Abhradeep Thakurta, name: Kai Yuanqing Xiao, name: Andreas Terzis, name: Florian Tramèr
Published: 2025-10-10 05:51:04

How should we evaluate the robustness of language model defenses? Current
defenses against jailbreaks and prompt injections (which aim to prevent an
attacker from eliciting harmful knowledge or remotely triggering malicious
actions, respectively) are typically evaluated either against a static set of
harmful attack strings, or against computationally weak optimization methods
that were not designed with the defense in mind. We argue that this evaluation
process is flawed.
  Instead, we should evaluate defenses against adaptive attackers who
explicitly modify their attack strategy to counter a defense's design while
spending considerable resources to optimize their objective. By systematically
tuning and scaling general optimization techniques-gradient descent,
reinforcement learning, random search, and human-guided exploration-we bypass
12 recent defenses (based on a diverse set of techniques) with attack success
rate above 90% for most; importantly, the majority of defenses originally
reported near-zero attack success rates. We believe that future defense work
must consider stronger attacks, such as the ones we describe, in order to make
reliable and convincing claims of robustness.
A Tale of LLMs and Induced Small Proxies: Scalable Agents for Knowledge
  Mining
Authors: name: Sipeng Zhang, name: Longfei Yun, name: Zilong Wang, name: Jingbo Shang, name: Letian Peng
Published: 2025-10-01 20:06:48

At the core of Deep Research is knowledge mining, the task of extracting
structured information from massive unstructured text in response to user
instructions. Large language models (LLMs) excel at interpreting such
instructions but are prohibitively expensive to deploy at scale, while
traditional pipelines of classifiers and extractors remain efficient yet
brittle and unable to generalize to new tasks. We introduce Falconer, a
collaborative framework that combines the agentic reasoning of LLMs with
lightweight proxy models for scalable knowledge mining. In Falconer, LLMs act
as planners, decomposing user instructions into executable pipelines, and as
annotators, generating supervision to train small proxies. The framework
unifies classification and extraction into two atomic operations, get label and
get span, enabling a single instruction-following model to replace multiple
task-specific components. To evaluate the consistency between proxy models
incubated by Falconer and annotations provided by humans and large models, we
construct new benchmarks covering both planning and end-to-end execution.
Experiments show that Falconer closely matches state-of-the-art LLMs in
instruction-following accuracy while reducing inference cost by up to 90% and
accelerating large-scale knowledge mining by more than 20x, offering an
efficient and scalable foundation for Deep Research.
MultiCOIN: Multi-Modal COntrollable Video INbetweening
Authors: name: Maham Tanveer, name: Yang Zhou, name: Simon Niklaus, name: Ali Mahdavi Amiri, name: Hao Zhang, name: Krishna Kumar Singh, name: Nanxuan Zhao
Published: 2025-10-09 17:59:27

Video inbetweening creates smooth and natural transitions between two image
frames, making it an indispensable tool for video editing and long-form video
synthesis. Existing works in this domain are unable to generate large, complex,
or intricate motions. In particular, they cannot accommodate the versatility of
user intents and generally lack fine control over the details of intermediate
frames, leading to misalignment with the creative mind. To fill these gaps, we
introduce MultiCOIN, a video inbetweening framework that allows multi-modal
controls, including depth transition and layering, motion trajectories, text
prompts, and target regions for movement localization, while achieving a
balance between flexibility, ease of use, and precision for fine-grained video
interpolation. To achieve this, we adopt the Diffusion Transformer (DiT)
architecture as our video generative model, due to its proven capability to
generate high-quality long videos. To ensure compatibility between DiT and our
multi-modal controls, we map all motion controls into a common sparse and
user-friendly point-based representation as the video/noise input. Further, to
respect the variety of controls which operate at varying levels of granularity
and influence, we separate content controls and motion controls into two
branches to encode the required features before guiding the denoising process,
resulting in two generators, one for motion and the other for content. Finally,
we propose a stage-wise training strategy to ensure that our model learns the
multi-modal controls smoothly. Extensive qualitative and quantitative
experiments demonstrate that multi-modal controls enable a more dynamic,
customizable, and contextually accurate visual narrative.
Multimodal Policy Internalization for Conversational Agents
Authors: name: Zhenhailong Wang, name: Jiateng Liu, name: Amin Fazel, name: Ritesh Sarkhel, name: Xing Fan, name: Xiang Li, name: Chenlei Guo, name: Heng Ji, name: Ruhi Sarikaya
Published: 2025-10-10 15:28:30

Modern conversational agents like ChatGPT and Alexa+ rely on predefined
policies specifying metadata, response styles, and tool-usage rules. As these
LLM-based systems expand to support diverse business and user queries, such
policies, often implemented as in-context prompts, are becoming increasingly
complex and lengthy, making faithful adherence difficult and imposing large
fixed computational costs. With the rise of multimodal agents, policies that
govern visual and multimodal behaviors are critical but remain understudied.
Prior prompt-compression work mainly shortens task templates and
demonstrations, while existing policy-alignment studies focus only on
text-based safety rules. We introduce Multimodal Policy Internalization (MPI),
a new task that internalizes reasoning-intensive multimodal policies into model
parameters, enabling stronger policy-following without including the policy
during inference. MPI poses unique data and algorithmic challenges. We build
two datasets spanning synthetic and real-world decision-making and tool-using
tasks and propose TriMPI, a three-stage training framework. TriMPI first
injects policy knowledge via continual pretraining, then performs supervised
finetuning, and finally applies PolicyRollout, a GRPO-style reinforcement
learning extension that augments rollouts with policy-aware responses for
grounded exploration. TriMPI achieves notable gains in end-to-end accuracy,
generalization, and robustness to forgetting. As the first work on multimodal
policy internalization, we provide datasets, training recipes, and
comprehensive evaluations to foster future research. Project page:
https://mikewangwzhl.github.io/TriMPI.
AndesVL Technical Report: An Efficient Mobile-side Multimodal Large
  Language Model
Authors: name: Zhiwei Jin, name: Xiaohui Song, name: Nan Wang, name: Yafei Liu, name: Chao Li, name: Xin Li, name: Ruichen Wang, name: Zhihao Li, name: Qi Qi, name: Long Cheng, name: Dongze Hao, name: Quanlong Zheng, name: Yanhao Zhang, name: Haobo Ji, name: Jian Ma, name: Zhitong Zheng, name: Zhenyi Lin, name: Haolin Deng, name: Xin Zou, name: Xiaojie Yin, name: Ruilin Wang, name: Liankai Cai, name: Haijing Liu, name: Yuqing Qiu, name: Ke Chen, name: Zixian Li, name: Chi Xie, name: Huafei Li, name: Chenxing Li, name: Chuangchuang Wang, name: Kai Tang, name: Zhiguang Zhu, name: Kai Tang, name: Wenmei Gao, name: Rui Wang, name: Jun Wu, name: Chao Liu, name: Qin Xie, name: Chen Chen, name: Haonan Lu
Published: 2025-10-13 15:04:38

In recent years, while cloud-based MLLMs such as QwenVL, InternVL, GPT-4o,
Gemini, and Claude Sonnet have demonstrated outstanding performance with
enormous model sizes reaching hundreds of billions of parameters, they
significantly surpass the limitations in memory, power consumption, and
computing capacity of edge devices such as mobile phones. This paper introduces
AndesVL, a suite of mobile-side MLLMs with 0.6B to 4B parameters based on
Qwen3's LLM and various visual encoders. We comprehensively outline the model
architectures, training pipeline, and training data of AndesVL, which achieves
first-tier performance across a wide range of open-source benchmarks, including
fields such as text-rich image understanding, reasoning and math, multi-image
comprehension, general VQA, hallucination mitigation, multilingual
understanding, and GUI-related tasks when compared with state-of-the-art models
of a similar scale. Furthermore, we introduce a 1+N LoR
SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive
  Reasoning
Authors: name: Ruohao Li, name: Hongjun Liu, name: Leyi Zhao, name: Zisu Li, name: Jiawei Li, name: Jiajun Jiang, name: Linning Xu, name: Chen Zhao, name: Mingming Fan, name: Chen Liang
Published: 2025-10-11 06:28:22

Large language model (LLM) agents have shown remarkable reasoning abilities.
However, existing multi-agent frameworks often rely on fixed roles or
centralized control, limiting scalability and adaptability in long-horizon
reasoning. We introduce SwarmSys, a closed-loop framework for distributed
multi-agent reasoning inspired by swarm intelligence. Coordination in SwarmSys
emerges through iterative interactions among three specialized roles,
Explorers, Workers, and Validators, that continuously cycle through
exploration, exploitation, and validation. To enable scalable and adaptive
collaboration, we integrate adaptive agent and event profiles, embedding-based
probabilistic matching, and a pheromone-inspired reinforcement mechanism,
supporting dynamic task allocation and self-organizing convergence without
global supervision. Across symbolic reasoning, research synthesis, and
scientific programming tasks, SwarmSys consistently outperforms baselines,
improving both accuracy and reasoning stability. These findings highlight
swarm-inspired coordination as a promising paradigm for scalable, robust, and
adaptive multi-agent reasoning, suggesting that coordination scaling may rival
model scaling in advancing LLM intelligence.
World-To-Image: Grounding Text-to-Image Generation with Agent-Driven
  World Knowledge
Authors: name: Moo Hyun Son, name: Jintaek Oh, name: Sun Bin Mun, name: Jaechul Roh, name: Sehyun Choi
Published: 2025-10-05 13:35:30

While text-to-image (T2I) models can synthesize high-quality images, their
performance degrades significantly when prompted with novel or
out-of-distribution (OOD) entities due to inherent knowledge cutoffs. We
introduce World-To-Image, a novel framework that bridges this gap by empowering
T2I generation with agent-driven world knowledge. We design an agent that
dynamically searches the web to retrieve images for concepts unknown to the
base model. This information is then used to perform multimodal prompt
optimization, steering powerful generative backbones toward an accurate
synthesis. Critically, our evaluation goes beyond traditional metrics,
utilizing modern assessments like LLMGrader and ImageReward to measure true
semantic fidelity. Our experiments show that World-To-Image substantially
outperforms state-of-the-art methods in both semantic alignment and visual
aesthetics, achieving +8.1% improvement in accuracy-to-prompt on our curated
NICE benchmark. Our framework achieves these results with high efficiency in
less than three iterations, paving the way for T2I systems that can better
reflect the ever-changing real world. Our demo code is available
herehttps://github.com/mhson-kyle/World-To-Image.
ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large
  Vision-and-Language Models
Authors: name: Yuqi Liu, name: Liangyu Chen, name: Jiazhen Liu, name: Mingkang Zhu, name: Zhisheng Zhong, name: Bei Yu, name: Jiaya Jia
Published: 2025-10-12 13:42:55

Typical post-training paradigms for Large Vision-and-Language Models (LVLMs)
include Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable
Rewards (RLVR). SFT leverages external guidance to inject new knowledge,
whereas RLVR utilizes internal reinforcement to enhance reasoning capabilities
and overall performance. However, our analysis reveals that SFT often leads to
sub-optimal performance, while RLVR struggles with tasks that exceed the
model's internal knowledge base. To address these limitations, we propose
ViSurf (Visual Supervised-and-Reinforcement
Fine-Tuning), a unified post-training paradigm that integrates the
strengths of both SFT and RLVR within a single stage. We analyze the derivation
of the SFT and RLVR objectives to establish the ViSurf objective, providing a
unified perspective on these two paradigms. The core of ViSurf involves
injecting ground-truth labels into the RLVR rollouts, thereby providing
simultaneous external supervision and internal reinforcement. Furthermore, we
introduce three novel reward control strategies to stabilize and optimize the
training process. Extensive experiments across several diverse benchmarks
demonstrate the effectiveness of ViSurf, outperforming both individual SFT,
RLVR, and two-stage SFT \textrightarrow RLVR. In-depth analysis corroborates
these findings, validating the derivation and design principles of ViSurf.
The Curious Case of Factual (Mis)Alignment between LLMs' Short- and
  Long-Form Answers
Authors: name: Saad Obaid ul Islam, name: Anne Lauscher, name: Goran Glavaš
Published: 2025-10-13 10:00:58

Large language models (LLMs) can correctly answer "When was Einstein born?"
yet fail to provide the same date when writing about Einstein's life revealing
a fundamental inconsistency in how models access factual knowledge across task
complexities. While models display impressive accuracy on factual
question-answering benchmarks, the reliability gap between simple and complex
queries remains poorly understood, eroding their trustworthiness. In this work,
we introduce Short-Long Form Alignment for Factual Question Answering (SLAQ), a
controlled evaluation framework that compares LLMs' answers to the same factual
questions asked (a) in isolation (short) vs. (b) integrated into complex
queries (long). Looking at 16 LLMs across 600 queries, we find a systematic
misalignment of answers to the corresponding short and long queries. We further
uncover position-dependent accuracy loss and momentum effects where consecutive
correct or incorrect answers create self-reinforcing patterns. Through
mechanistic analysis, we find that aligned facts activate overlapping model
internals, and that metrics based on mechanistic similarity can predict
short-long answer alignment with up to 78% accuracy. Our work establishes
factual consistency over query complexity as an important aspect of LLMs'
trustworthiness and challenges current evaluation practices, which implicitly
assume that good performance for simple factual queries implies reliability in
more complex knowledge-seeking tasks too.
Through the Perspective of LiDAR: A Feature-Enriched and
  Uncertainty-Aware Annotation Pipeline for Terrestrial Point Cloud
  Segmentation
Authors: name: Fei Zhang, name: Rob Chancia, name: Josie Clapp, name: Amirhossein Hassanzadeh, name: Dimah Dera, name: Richard MacKenzie, name: Jan van Aardt
Published: 2025-10-08 02:25:59

Accurate semantic segmentation of terrestrial laser scanning (TLS) point
clouds is limited by costly manual annotation. We propose a semi-automated,
uncertainty-aware pipeline that integrates spherical projection, feature
enrichment, ensemble learning, and targeted annotation to reduce labeling
effort, while sustaining high accuracy. Our approach projects 3D points to a 2D
spherical grid, enriches pixels with multi-source features, and trains an
ensemble of segmentation networks to produce pseudo-labels and uncertainty
maps, the latter guiding annotation of ambiguous regions. The 2D outputs are
back-projected to 3D, yielding densely annotated point clouds supported by a
three-tier visualization suite (2D feature maps, 3D colorized point clouds, and
compact virtual spheres) for rapid triage and reviewer guidance. Using this
pipeline, we build Mangrove3D, a semantic segmentation TLS dataset for mangrove
forests. We further evaluate data efficiency and feature importance to address
two key questions: (1) how much annotated data are needed and (2) which
features matter most. Results show that performance saturates after ~12
annotated scans, geometric features contribute the most, and compact
nine-channel stacks capture nearly all discriminative power, with the mean
Intersection over Union (mIoU) plateauing at around 0.76. Finally, we confirm
the generalization of our feature-enrichment strategy through cross-dataset
tests on ForestSemantic and Semantic3D.
  Our contributions include: (i) a robust, uncertainty-aware TLS annotation
pipeline with visualization tools; (ii) the Mangrove3D dataset; and (iii)
empirical guidance on data efficiency and feature importance, thus enabling
scalable, high-quality segmentation of TLS point clouds for ecological
monitoring and beyond. The dataset and processing scripts are publicly
available at https://fz-rit.github.io/through-the-lidars-eye/.
CoBia: Constructed Conversations Can Trigger Otherwise Concealed
  Societal Biases in LLMs
Authors: name: Nafiseh Nikeghbal, name: Amir Hossein Kargaran, name: Jana Diesner
Published: 2025-10-10 21:09:01

Improvements in model construction, including fortified safety guardrails,
allow Large language models (LLMs) to increasingly pass standard safety checks.
However, LLMs sometimes slip into revealing harmful behavior, such as
expressing racist viewpoints, during conversations. To analyze this
systematically, we introduce CoBia, a suite of lightweight adversarial attacks
that allow us to refine the scope of conditions under which LLMs depart from
normative or ethical behavior in conversations. CoBia creates a constructed
conversation where the model utters a biased claim about a social group. We
then evaluate whether the model can recover from the fabricated bias claim and
reject biased follow-up questions. We evaluate 11 open-source as well as
proprietary LLMs for their outputs related to six socio-demographic categories
that are relevant to individual safety and fair treatment, i.e., gender, race,
religion, nationality, sex orientation, and others. Our evaluation is based on
established LLM-based bias metrics, and we compare the results against human
judgments to scope out the LLMs' reliability and alignment. The results suggest
that purposefully constructed conversations reliably reveal bias amplification
and that LLMs often fail to reject biased follow-up questions during dialogue.
This form of stress-testing highlights deeply embedded biases that can be
surfaced through interaction. Code and artifacts are available at
https://github.com/nafisenik/CoBia.
Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by
  Refining Belief States
Authors: name: Qinglin Zhu, name: Yizhen Yao, name: Runcong Zhao, name: Yanzheng Xiang, name: Amrutha Saseendran, name: Chen Jin, name: Philip Alexander Teare, name: Bin Liang, name: Yulan He, name: Lin Gui
Published: 2025-10-13 06:38:13

Autoregressive (AR) models remain the standard for natural language
generation but still suffer from high latency due to strictly sequential
decoding. Recent diffusion-inspired approaches, such as LlaDA and Dream,
mitigate this by generating in parallel, yet they suffer from two core
limitations: information loss, as predictive distributions for non-finalized
tokens are discarded at each step, and premature commitment, where local
decisions are made without sufficient global coordination. We introduce Latent
Refinement Decoding (LRD), a two-stage framework with Latent Refinement and a
Predictive Feedback Loop. The first stage maintains masked positions as
distributional mixtures of predicted tokens and the mask embedding, allowing
the model to establish more globally consistent beliefs. The second stage
progressively finalizes confident tokens while retaining uncertain ones for
iterative feedback. KL-divergence dynamics provide a principled and reliable
criterion for convergence and early stopping. Experiments across coding
(HumanEval +6.3, MBPP +2.6) and reasoning (GSM8K +2.9, MATH500 +3.8) show that
LRD improves accuracy while delivering speedups of up to 10.6x, making it a
strong and versatile alternative for parallel sequence generation.
From Data to Rewards: a Bilevel Optimization Perspective on Maximum
  Likelihood Estimation
Authors: name: Abdelhakim Benechehab, name: Gabriel Singer, name: Corentin Léger, name: Youssef Attia El Hili, name: Giuseppe Paolo, name: Albert Thomas, name: Maurizio Filippone, name: Balázs Kégl
Published: 2025-10-08 23:45:37

Generative models form the backbone of modern machine learning, underpinning
state-of-the-art systems in text, vision, and multimodal applications. While
Maximum Likelihood Estimation has traditionally served as the dominant training
paradigm, recent work have highlighted its limitations, particularly in
generalization and susceptibility to catastrophic forgetting compared to
Reinforcement Learning techniques, such as Policy Gradient methods. However,
these approaches depend on explicit reward signals, which are often unavailable
in practice, leaving open the fundamental problem of how to align generative
models when only high-quality datasets are accessible. In this work, we address
this challenge via a Bilevel Optimization framework, where the reward function
is treated as the optimization variable of an outer-level problem, while a
policy gradient objective defines the inner-level. We then conduct a
theoretical analysis of this optimization problem in a tractable setting and
extract insights that, as we demonstrate, generalize to applications such as
tabular classification and model-based reinforcement learning. We release the
code at https://github.com/abenechehab/nll_to_po .