Orionnnn/hub-stats huggingface dataset:
Vision Transformers with Self-Distilled Registers
Authors: name: Yinjie Chen, name: Zipeng Yan, name: Chong Zhou, name: Bo Dai, name: Andrew F. Luo
Published: 2025-05-27 17:59:41

Vision Transformers (ViTs) have emerged as the dominant architecture for
visual processing tasks, demonstrating excellent scalability with increased
training data and model size. However, recent work has identified the emergence
of artifact tokens in ViTs that are incongruous with the local semantics. These
anomalous tokens degrade ViT performance in tasks that require fine-grained
localization or structural coherence. An effective mitigation of this issue is
to the addition of register tokens to ViTs, which implicitly "absorb" the
artifact term during training. Given the availability of various large-scale
pre-trained ViTs, in this paper we aim at equipping them with such register
tokens without the need of re-training them from scratch, which is infeasible
considering their size. Specifically, we propose Post Hoc Registers (PH-Reg),
an efficient self-distillation method that integrates registers into an
existing ViT without requiring additional labeled data and full retraining.
PH-Reg initializes both teacher and student networks from the same pre-trained
ViT. The teacher remains frozen and unmodified, while the student is augmented
with randomly initialized register tokens. By applying test-time augmentation
to the teacher's inputs, we generate denoised dense embeddings free of
artifacts, which are then used to optimize only a small subset of unlocked
student weights. We show that our approach can effectively reduce the number of
artifact tokens, improving the segmentation and depth prediction of the student
ViT under zero-shot and linear probing.
CoreMatching: A Co-adaptive Sparse Inference Framework with Token and
  Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
Authors: name: Qinsi Wang, name: Hancheng Ye, name: Ming-Yu Chung, name: Yudong Liu, name: Yueqian Lin, name: Martin Kuo, name: Mingyuan Ma, name: Jianyi Zhang, name: Yiran Chen
Published: 2025-05-25 17:16:34

Vision-Language Models (VLMs) excel across diverse tasks but suffer from high
inference costs in time and memory. Token sparsity mitigates inefficiencies in
token usage, while neuron sparsity reduces high-dimensional computations, both
offering promising solutions to enhance efficiency. Recently, these two
sparsity paradigms have evolved largely in parallel, fostering the prevailing
assumption that they function independently. However, a fundamental yet
underexplored question remains: Do they truly operate in isolation, or is there
a deeper underlying interplay that has yet to be uncovered? In this paper, we
conduct the first comprehensive investigation into this question. By
introducing and analyzing the matching mechanism between Core Neurons and Core
Tokens, we found that key neurons and tokens for inference mutually influence
and reinforce each other. Building on this insight, we propose CoreMatching, a
co-adaptive sparse inference framework, which leverages the synergy between
token and neuron sparsity to enhance inference efficiency. Through theoretical
analysis and efficiency evaluations, we demonstrate that the proposed method
surpasses state-of-the-art baselines on ten image understanding tasks and three
hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs
reduction and a 10x overall speedup. Code is released at
https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.
Thinker: Learning to Think Fast and Slow
Authors: name: Stephen Chung, name: Wenyu Du, name: Jie Fu
Published: 2025-05-27 12:22:46

Recent studies show that the reasoning capabilities of Large Language Models
(LLMs) can be improved by applying Reinforcement Learning (RL) to
question-answering (QA) tasks in areas such as math and coding. With a long
context length, LLMs may learn to perform search, as indicated by the
self-correction behavior observed in DeepSeek R1. However, this search behavior
is often imprecise and lacks confidence, resulting in long, redundant responses
and highlighting deficiencies in intuition and verification. Inspired by the
Dual Process Theory in psychology, we introduce a simple modification to the QA
task that includes four stages: Fast Thinking, where the LLM must answer within
a strict token budget; Verification, where the model evaluates its initial
response; Slow Thinking, where it refines the initial response with more
deliberation; and Summarization, where it distills the refinement from the
previous stage into precise steps. Our proposed task improves average accuracy
from 24.9% to 27.9% for Qwen2.5-1.5B, and from 45.9% to 49.8% for
DeepSeek-R1-Qwen-1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone
achieves 26.8% accuracy using fewer than 1000 tokens, demonstrating substantial
inference efficiency gains. These findings suggest that intuition and
deliberative reasoning are distinct, complementary systems benefiting from
targeted training.
PreMoe: Lightening MoEs on Constrained Memory by Expert Pruning and
  Retrieval
Authors: name: Zehua Pei, name: Ying Zhang, name: Hui-Ling Zhen, name: Xianzhi Yu, name: Wulong Liu, name: Sinno Jialin Pan, name: Mingxuan Yuan, name: Bei Yu
Published: 2025-05-23 08:59:16

Mixture-of-experts (MoE) architectures enable scaling large language models
(LLMs) to vast parameter counts without a proportional rise in computational
costs. However, the significant memory demands of large MoE models hinder their
deployment across various computational environments, from cloud servers to
consumer devices. This study first demonstrates pronounced task-specific
specialization in expert activation patterns within MoE layers. Building on
this, we introduce PreMoe, a novel framework that enables efficient deployment
of massive MoE models in memory-constrained environments. PreMoe features two
main components: probabilistic expert pruning (PEP) and task-adaptive expert
retrieval (TAER). PEP employs a new metric, the task-conditioned expected
selection score (TCESS), derived from router logits to quantify expert
importance for specific tasks, thereby identifying a minimal set of critical
experts. TAER leverages these task-specific expert importance profiles for
efficient inference. It pre-computes and stores compact expert patterns for
diverse tasks. When a user query is received, TAER rapidly identifies the most
relevant stored task pattern and reconstructs the model by loading only the
small subset of experts crucial for that task. This approach dramatically
reduces the memory footprint across all deployment scenarios. DeepSeek-R1 671B
maintains 97.2\% accuracy on MATH500 when pruned to 8/128 configuration (50\%
expert reduction), and still achieves 72.0\% with aggressive 8/32 pruning
(87.5\% expert reduction). Pangu-Ultra-MoE 718B achieves 97.15\% on MATH500 and
81.3\% on AIME24 with 8/128 pruning, while even more aggressive pruning to 4/64
(390GB memory) preserves 96.95\% accuracy on MATH500. We make our code publicly
available at https://github.com/JarvisPei/PreMoe.
MotionPro: A Precise Motion Controller for Image-to-Video Generation
Authors: name: Zhongwei Zhang, name: Fuchen Long, name: Zhaofan Qiu, name: Yingwei Pan, name: Wu Liu, name: Ting Yao, name: Tao Mei
Published: 2025-05-26 17:59:03

Animating images with interactive motion control has garnered popularity for
image-to-video (I2V) generation. Modern approaches typically rely on large
Gaussian kernels to extend motion trajectories as condition without explicitly
defining movement region, leading to coarse motion control and failing to
disentangle object and camera moving. To alleviate these, we present MotionPro,
a precise motion controller that novelly leverages region-wise trajectory and
motion mask to regulate fine-grained motion synthesis and identify target
motion category (i.e., object or camera moving), respectively. Technically,
MotionPro first estimates the flow maps on each training video via a tracking
model, and then samples the region-wise trajectories to simulate inference
scenario. Instead of extending flow through large Gaussian kernels, our
region-wise trajectory approach enables more precise control by directly
utilizing trajectories within local regions, thereby effectively characterizing
fine-grained movements. A motion mask is simultaneously derived from the
predicted flow maps to capture the holistic motion dynamics of the movement
regions. To pursue natural motion control, MotionPro further strengthens video
denoising by incorporating both region-wise trajectories and motion mask
through feature modulation. More remarkably, we meticulously construct a
benchmark, i.e., MC-Bench, with 1.1K user-annotated image-trajectory pairs, for
the evaluation of both fine-grained and object-level I2V motion control.
Extensive experiments conducted on WebVid-10M and MC-Bench demonstrate the
effectiveness of MotionPro. Please refer to our project page for more results:
https://zhw-zhang.github.io/MotionPro-page/.
Rendering-Aware Reinforcement Learning for Vector Graphics Generation
Authors: name: Juan A. Rodriguez, name: Haotian Zhang, name: Abhay Puri, name: Aarash Feizi, name: Rishav Pramanik, name: Pascal Wichmann, name: Arnab Mondal, name: Mohammad Reza Samsami, name: Rabiul Awal, name: Perouz Taslakian, name: Spandana Gella, name: Sai Rajeswar, name: David Vazquez, name: Christopher Pal, name: Marco Pedersoli
Published: 2025-05-27 06:56:00

Scalable Vector Graphics (SVG) offer a powerful format for representing
visual designs as interpretable code. Recent advances in vision-language models
(VLMs) have enabled high-quality SVG generation by framing the problem as a
code generation task and leveraging large-scale pretraining. VLMs are
particularly suitable for this task as they capture both global semantics and
fine-grained visual patterns, while transferring knowledge across vision,
natural language, and code domains. However, existing VLM approaches often
struggle to produce faithful and efficient SVGs because they never observe the
rendered images during training. Although differentiable rendering for
autoregressive SVG code generation remains unavailable, rendered outputs can
still be compared to original inputs, enabling evaluative feedback suitable for
reinforcement learning (RL). We introduce RLRF(Reinforcement Learning from
Rendering Feedback), an RL method that enhances SVG generation in
autoregressive VLMs by leveraging feedback from rendered SVG outputs. Given an
input image, the model generates SVG roll-outs that are rendered and compared
to the original image to compute a reward. This visual fidelity feedback guides
the model toward producing more accurate, efficient, and semantically coherent
SVGs. RLRF significantly outperforms supervised fine-tuning, addressing common
failure modes and enabling precise, high-quality SVG generation with strong
structural understanding and generalization.
Reinforcing General Reasoning without Verifiers
Authors: name: Xiangxin Zhou, name: Zichen Liu, name: Anya Sims, name: Haonan Wang, name: Tianyu Pang, name: Chongxuan Li, name: Liang Wang, name: Min Lin, name: Chao Du
Published: 2025-05-27 17:56:27

The recent paradigm shift towards training large language models (LLMs) using
DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has
led to impressive advancements in code and mathematical reasoning. However,
this methodology is limited to tasks where rule-based answer verification is
possible and does not naturally extend to real-world domains such as chemistry,
healthcare, engineering, law, biology, business, and economics. Current
practical workarounds use an additional LLM as a model-based verifier; however,
this introduces issues such as reliance on a strong verifier LLM,
susceptibility to reward hacking, and the practical burden of maintaining the
verifier model in memory during training. To address this and extend
DeepSeek-R1-Zero-style training to general reasoning domains, we propose a
verifier-free method (VeriFree) that bypasses answer verification and instead
uses RL to directly maximize the probability of generating the reference
answer. We compare VeriFree with verifier-based methods and demonstrate that,
in addition to its significant practical benefits and reduced compute
requirements, VeriFree matches and even surpasses verifier-based methods on
extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related
benchmarks. Moreover, we provide insights into this method from multiple
perspectives: as an elegant integration of training both the policy and
implicit verifier in a unified model, and as a variational optimization
approach. Code is available at https://github.com/sail-sg/VeriFree.
Capability-Based Scaling Laws for LLM Red-Teaming
Authors: name: Alexander Panfilov, name: Paul Kassianik, name: Maksym Andriushchenko, name: Jonas Geiping
Published: 2025-05-26 16:05:41

As large language models grow in capability and agency, identifying
vulnerabilities through red-teaming becomes vital for safe deployment. However,
traditional prompt-engineering approaches may prove ineffective once
red-teaming turns into a weak-to-strong problem, where target models surpass
red-teamers in capabilities. To study this shift, we frame red-teaming through
the lens of the capability gap between attacker and target. We evaluate more
than 500 attacker-target pairs using LLM-based jailbreak attacks that mimic
human red-teamers across diverse families, sizes, and capability levels. Three
strong trends emerge: (i) more capable models are better attackers, (ii) attack
success drops sharply once the target's capability exceeds the attacker's, and
(iii) attack success rates correlate with high performance on social science
splits of the MMLU-Pro benchmark. From these trends, we derive a jailbreaking
scaling law that predicts attack success for a fixed target based on
attacker-target capability gap. These findings suggest that fixed-capability
attackers (e.g., humans) may become ineffective against future models,
increasingly capable open-source models amplify risks for existing systems, and
model providers must accurately measure and control models' persuasive and
manipulative abilities to limit their effectiveness as attackers.
MMPerspective: Do MLLMs Understand Perspective? A Comprehensive
  Benchmark for Perspective Perception, Reasoning, and Robustness
Authors: name: Yunlong Tang, name: Pinxin Liu, name: Mingqian Feng, name: Zhangyun Tan, name: Rui Mao, name: Chao Huang, name: Jing Bi, name: Yunzhong Xiao, name: Susan Liang, name: Hang Hua, name: Ali Vosoughi, name: Luchuan Song, name: Zeliang Zhang, name: Chenliang Xu
Published: 2025-05-26 18:20:22

Understanding perspective is fundamental to human visual perception, yet the
extent to which multimodal large language models (MLLMs) internalize
perspective geometry remains unclear. We introduce MMPerspective, the first
benchmark specifically designed to systematically evaluate MLLMs' understanding
of perspective through 10 carefully crafted tasks across three complementary
dimensions: Perspective Perception, Reasoning, and Robustness. Our benchmark
comprises 2,711 real-world and synthetic image instances with 5,083
question-answer pairs that probe key capabilities, such as vanishing point
perception and counting, perspective type reasoning, line relationship
understanding in 3D space, invariance to perspective-preserving
transformations, etc. Through a comprehensive evaluation of 43 state-of-the-art
MLLMs, we uncover significant limitations: while models demonstrate competence
on surface-level perceptual tasks, they struggle with compositional reasoning
and maintaining spatial consistency under perturbations. Our analysis further
reveals intriguing patterns between model architecture, scale, and perspective
capabilities, highlighting both robustness bottlenecks and the benefits of
chain-of-thought prompting. MMPerspective establishes a valuable testbed for
diagnosing and advancing spatial understanding in vision-language systems.
Resources available at: https://yunlong10.github.io/MMPerspective/
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic
  Scientific Workflows
Authors: name: Qiushi Sun, name: Zhoumianze Liu, name: Chang Ma, name: Zichen Ding, name: Fangzhi Xu, name: Zhangyue Yin, name: Haiteng Zhao, name: Zhenyu Wu, name: Kanzhi Cheng, name: Zhaoyang Liu, name: Jianing Wang, name: Qintong Li, name: Xiangru Tang, name: Tianbao Xie, name: Xiachong Feng, name: Xiang Li, name: Ben Kao, name: Wenhai Wang, name: Biqing Qi, name: Lingpeng Kong, name: Zhiyong Wu
Published: 2025-05-26 12:27:27

Large Language Models (LLMs) have extended their impact beyond Natural
Language Processing, substantially fostering the development of
interdisciplinary research. Recently, various LLM-based agents have been
developed to assist scientific discovery progress across multiple aspects and
domains. Among these, computer-using agents, capable of interacting with
operating systems as humans do, are paving the way to automated scientific
problem-solving and addressing routines in researchers' workflows. Recognizing
the transformative potential of these agents, we introduce ScienceBoard, which
encompasses two complementary contributions: (i) a realistic, multi-domain
environment featuring dynamic and visually rich scientific workflows with
integrated professional software, where agents can autonomously interact via
different interfaces to accelerate complex research tasks and experiments; and
(ii) a challenging benchmark of 169 high-quality, rigorously validated
real-world tasks curated by humans, spanning scientific-discovery workflows in
domains such as biochemistry, astronomy, and geoinformatics. Extensive
evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude
3.7, UI-TARS) show that, despite some promising results, they still fall short
of reliably assisting scientists in complex workflows, achieving only a 15%
overall success rate. In-depth analysis further provides valuable insights for
addressing current agent limitations and more effective design principles,
paving the way to build more capable agents for scientific discovery. Our code,
environment, and benchmark are at
https://qiushisun.github.io/ScienceBoard-Home/.
VideoGameBench: Can Vision-Language Models complete popular video games?
Authors: name: Alex L. Zhang, name: Thomas L. Griffiths, name: Karthik R. Narasimhan, name: Ofir Press
Published: 2025-05-23 17:43:27

Vision-language models (VLMs) have achieved strong results on coding and math
benchmarks that are challenging for humans, yet their ability to perform tasks
that come naturally to humans--such as perception, spatial navigation, and
memory management--remains understudied. Real video games are crafted to be
intuitive for humans to learn and master by leveraging innate inductive biases,
making them an ideal testbed for evaluating such capabilities in VLMs. To this
end, we introduce VideoGameBench, a benchmark consisting of 10 popular video
games from the 1990s that VLMs directly interact with in real-time.
VideoGameBench challenges models to complete entire games with access to only
raw visual inputs and a high-level description of objectives and controls, a
significant departure from existing setups that rely on game-specific
scaffolding and auxiliary information. We keep three of the games secret to
encourage solutions that generalize to unseen environments. Our experiments
show that frontier vision-language models struggle to progress beyond the
beginning of each game. We find inference latency to be a major limitation of
frontier models in the real-time setting; therefore, we introduce
VideoGameBench Lite, a setting where the game pauses while waiting for the LM's
next action. The best performing model, Gemini 2.5 Pro, completes only 0.48% of
VideoGameBench and 1.6% of VideoGameBench Lite. We hope that the formalization
of the human skills mentioned above into this benchmark motivates progress in
these research directions.
BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge
  Bases
Authors: name: Mathew J. Koretsky, name: Maya Willey, name: Adi Asija, name: Owen Bianchi, name: Chelsea X. Alvarado, name: Tanay Nayak, name: Nicole Kuznetsov, name: Sungwon Kim, name: Mike A. Nalls, name: Daniel Khashabi, name: Faraz Faghri
Published: 2025-05-23 17:58:07

Biomedical researchers increasingly rely on large-scale structured databases
for complex analytical tasks. However, current text-to-SQL systems often
struggle to map qualitative scientific questions into executable SQL,
particularly when implicit domain reasoning is required. We introduce
BiomedSQL, the first benchmark explicitly designed to evaluate scientific
reasoning in text-to-SQL generation over a real-world biomedical knowledge
base. BiomedSQL comprises 68,000 question/SQL query/answer triples grounded in
a harmonized BigQuery knowledge base that integrates gene-disease associations,
causal inference from omics data, and drug approval records. Each question
requires models to infer domain-specific criteria, such as genome-wide
significance thresholds, effect directionality, or trial phase filtering,
rather than rely on syntactic translation alone. We evaluate a range of open-
and closed-source LLMs across prompting strategies and interaction paradigms.
Our results reveal a substantial performance gap: GPT-o3-mini achieves 59.0%
execution accuracy, while our custom multi-step agent, BMSQL, reaches 62.6%,
both well below the expert baseline of 90.0%. BiomedSQL provides a new
foundation for advancing text-to-SQL systems capable of supporting scientific
discovery through robust reasoning over structured biomedical knowledge bases.
Our dataset is publicly available at
https://huggingface.co/datasets/NIH-CARD/BiomedSQL, and our code is open-source
at https://github.com/NIH-CARD/biomedsql.
Explaining Sources of Uncertainty in Automated Fact-Checking
Authors: name: Jingyi Sun, name: Greta Warren, name: Irina Shklovski, name: Isabelle Augenstein
Published: 2025-05-23 13:06:43

Understanding sources of a model's uncertainty regarding its predictions is
crucial for effective human-AI collaboration. Prior work proposes using
numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain
uncertainty that arises from conflicting evidence, leaving users unable to
resolve disagreements or rely on the output. We introduce CLUE
(Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the
first framework to generate natural language explanations of model uncertainty
by (i) identifying relationships between spans of text that expose
claim-evidence or inter-evidence conflicts and agreements that drive the
model's predictive uncertainty in an unsupervised way, and (ii) generating
explanations via prompting and attention steering that verbalize these critical
interactions. Across three language models and two fact-checking datasets, we
show that CLUE produces explanations that are more faithful to the model's
uncertainty and more consistent with fact-checking decisions than prompting for
uncertainty explanations without span-interaction guidance. Human evaluators
judge our explanations to be more helpful, more informative, less redundant,
and more logically consistent with the input than this baseline. CLUE requires
no fine-tuning or architectural changes, making it plug-and-play for any
white-box language model. By explicitly linking uncertainty to evidence
conflicts, it offers practical support for fact-checking and generalises
readily to other tasks that require reasoning over complex information.