Orionnnn/hub-stats huggingface dataset:
Adaptive Computation Pruning for the Forgetting Transformer
Authors: name: Zhixuan Lin, name: Johan Obando-Ceron, name: Xu Owen He, name: Aaron Courville
Published: 2025-04-09 14:57:55

The recently proposed Forgetting Transformer (FoX) incorporates a forget gate
into softmax attention and has shown consistently better or on-par performance
compared to the standard RoPE-based Transformer. Notably, many attention heads
in FoX tend to forget quickly, causing their output at each timestep to rely
primarily on the local context. Based on this observation, we propose Adaptive
Computation Pruning (ACP) for FoX, a method that dynamically prunes
computations involving input-output dependencies that are strongly decayed by
the forget gate. This is achieved using a dynamically set pruning threshold
that ensures that the pruned attention weights remain negligible. We apply ACP
to language model pretraining with FoX and show it consistently reduces the
number of FLOPs in softmax attention by around 70% across different model sizes
and context lengths, resulting in a roughly 10% to 35% improvement in training
throughput. Furthermore, longer context lengths yield greater computational
savings. All these speed improvements are achieved without any performance
degradation. We also perform several analyses to provide deeper insights into
our method, such as examining the pruning patterns and analyzing the
distribution of FLOP savings across different attention heads. Our code is
available at https://github.com/zhixuan-lin/arctic-fox.
NormalCrafter: Learning Temporally Consistent Normals from Video
  Diffusion Priors
Authors: name: Yanrui Bin, name: Wenbo Hu, name: Haoyuan Wang, name: Xinya Chen, name: Bing Wang
Published: 2025-04-15 17:39:07

Surface normal estimation serves as a cornerstone for a spectrum of computer
vision applications. While numerous efforts have been devoted to static image
scenarios, ensuring temporal coherence in video-based normal estimation remains
a formidable challenge. Instead of merely augmenting existing methods with
temporal components, we present NormalCrafter to leverage the inherent temporal
priors of video diffusion models. To secure high-fidelity normal estimation
across sequences, we propose Semantic Feature Regularization (SFR), which
aligns diffusion features with semantic cues, encouraging the model to
concentrate on the intrinsic semantics of the scene. Moreover, we introduce a
two-stage training protocol that leverages both latent and pixel space learning
to preserve spatial accuracy while maintaining long temporal context. Extensive
evaluations demonstrate the efficacy of our method, showcasing a superior
performance in generating temporally consistent normal sequences with intricate
details from diverse videos.
Seedream 3.0 Technical Report
Authors: name: Yu Gao, name: Lixue Gong, name: Qiushan Guo, name: Xiaoxia Hou, name: Zhichao Lai, name: Fanshi Li, name: Liang Li, name: Xiaochen Lian, name: Chao Liao, name: Liyang Liu, name: Wei Liu, name: Yichun Shi, name: Shiqi Sun, name: Yu Tian, name: Zhi Tian, name: Peng Wang, name: Rui Wang, name: Xuanda Wang, name: Xun Wang, name: Ye Wang, name: Guofeng Wu, name: Jie Wu, name: Xin Xia, name: Xuefeng Xiao, name: Zhonghua Zhai, name: Xinyu Zhang, name: Qi Zhang, name: Yuwei Zhang, name: Shijia Zhao, name: Jianchao Yang, name: Weilin Huang
Published: 2025-04-15 16:19:07

We present Seedream 3.0, a high-performance Chinese-English bilingual image
generation foundation model. We develop several technical improvements to
address existing challenges in Seedream 2.0, including alignment with
complicated prompts, fine-grained typography generation, suboptimal visual
aesthetics and fidelity, and limited image resolutions. Specifically, the
advancements of Seedream 3.0 stem from improvements across the entire pipeline,
from data construction to model deployment. At the data stratum, we double the
dataset using a defect-aware training paradigm and a dual-axis collaborative
data-sampling framework. Furthermore, we adopt several effective techniques
such as mixed-resolution training, cross-modality RoPE, representation
alignment loss, and resolution-aware timestep sampling in the pre-training
phase. During the post-training stage, we utilize diversified aesthetic
captions in SFT, and a VLM-based reward model with scaling, thereby achieving
outputs that well align with human preferences. Furthermore, Seedream 3.0
pioneers a novel acceleration paradigm. By employing consistent noise
expectation and importance-aware timestep sampling, we achieve a 4 to 8 times
speedup while maintaining image quality. Seedream 3.0 demonstrates significant
improvements over Seedream 2.0: it enhances overall capabilities, in particular
for text-rendering in complicated Chinese characters which is important to
professional typography generation. In addition, it provides native
high-resolution output (up to 2K), allowing it to generate images with high
visual quality.
TextArena
Authors: name: Leon Guertler, name: Bobby Cheng, name: Simon Yu, name: Bo Liu, name: Leshem Choshen, name: Cheston Tan
Published: 2025-04-15 17:55:20

TextArena is an open-source collection of competitive text-based games for
training and evaluation of agentic behavior in Large Language Models (LLMs). It
spans 57+ unique environments (including single-player, two-player, and
multi-player setups) and allows for easy evaluation of model capabilities via
an online-play system (against humans and other submitted models) with
real-time TrueSkill scores. Traditional benchmarks rarely assess dynamic social
skills such as negotiation, theory of mind, and deception, creating a gap that
TextArena addresses. Designed with research, community and extensibility in
mind, TextArena emphasizes ease of adding new games, adapting the framework,
testing models, playing against the models, and training models. Detailed
documentation of environments, games, leaderboard, and examples are available
on https://github.com/LeonGuertler/TextArena and https://www.textarena.ai/.
xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
Authors: name: Ding Chen, name: Qingchen Yu, name: Pengyuan Wang, name: Wentao Zhang, name: Bo Tang, name: Feiyu Xiong, name: Xinchi Li, name: Minchuan Yang, name: Zhiyu Li
Published: 2025-04-14 17:59:36

With the release of the o1 model by OpenAI, reasoning models adopting slow
thinking strategies have gradually emerged. As the responses generated by such
models often include complex reasoning, intermediate steps, and
self-reflection, existing evaluation methods are often inadequate. They
struggle to determine whether the LLM output is truly equivalent to the
reference answer, and also have difficulty identifying and extracting the final
answer from long, complex responses. To address this issue, we propose xVerify,
an efficient answer verifier for reasoning model evaluations. xVerify
demonstrates strong capability in equivalence judgment, enabling it to
effectively determine whether the answers produced by reasoning models are
equivalent to reference answers across various types of objective questions. To
train and evaluate xVerify, we construct the VAR dataset by collecting
question-answer pairs generated by multiple LLMs across various datasets,
leveraging multiple reasoning models and challenging evaluation sets designed
specifically for reasoning model assessment. A multi-round annotation process
is employed to ensure label accuracy. Based on the VAR dataset, we train
multiple xVerify models of different scales. In evaluation experiments
conducted on both the test set and generalization set, all xVerify models
achieve overall F1 scores and accuracy exceeding 95\%. Notably, the smallest
variant, xVerify-0.5B-I, outperforms all evaluation methods except GPT-4o,
while xVerify-3B-Ib surpasses GPT-4o in overall performance. These results
validate the effectiveness and generalizability of xVerify.
Heimdall: test-time scaling on the generative verification
Authors: name: Wenlei Shi, name: Xing Jin
Published: 2025-04-14 15:46:33

An AI system can create and maintain knowledge only to the extent that it can
verify that knowledge itself. Recent work on long Chain-of-Thought reasoning
has demonstrated great potential of LLMs on solving competitive problems, but
their verification ability remains to be weak and not sufficiently
investigated. In this paper, we propose Heimdall, the long CoT verification LLM
that can accurately judge the correctness of solutions. With pure reinforcement
learning, we boost the verification accuracy from 62.5% to 94.5% on competitive
math problems. By scaling with repeated sampling, the accuracy further
increases to 97.5%. Through human evaluation, Heimdall demonstrates impressive
generalization capabilities, successfully detecting most issues in challenging
math proofs, the type of which is not included during training. Furthermore, we
propose Pessimistic Verification to extend the functionality of Heimdall to
scaling up the problem solving. It calls Heimdall to judge the solutions from a
solver model and based on the pessimistic principle, selects the most likely
correct solution with the least uncertainty. Taking
DeepSeek-R1-Distill-Qwen-32B as the solver model, Pessimistic Verification
improves the solution accuracy on AIME2025 from 54.2% to 70.0% with 16x compute
budget and to 83.3% with more compute budget. With the stronger solver Gemini
2.5 Pro, the score reaches 93.0%. Finally, we prototype an automatic knowledge
discovery system, a ternary system where one poses questions, another provides
solutions, and the third verifies the solutions. Using the data synthesis work
NuminaMath for the first two components, Heimdall effectively identifies
problematic records within the dataset and reveals that nearly half of the data
is flawed, which interestingly aligns with the recent ablation studies from
NuminaMath.
How Instruction and Reasoning Data shape Post-Training: Data Quality
  through the Lens of Layer-wise Gradients
Authors: name: Ming Li, name: Yanhong Li, name: Ziyue Li, name: Tianyi Zhou
Published: 2025-04-14 23:53:47

As the post-training of large language models (LLMs) advances from
instruction-following to complex reasoning tasks, understanding how different
data affect finetuning dynamics remains largely unexplored. In this paper, we
present a spectral analysis of layer-wise gradients induced by low/high-quality
instruction and reasoning data for LLM post-training. Our analysis reveals that
widely-studied metrics for data evaluation, e.g., IFD, InsTag, Difficulty, and
Reward, can be explained and unified by spectral properties computed from
gradients' singular value decomposition (SVD). Specifically, higher-quality
data are usually associated with lower nuclear norms and higher effective
ranks. Notably, effective rank exhibits better robustness and resolution than
nuclear norm in capturing subtle quality differences. For example, reasoning
data achieves substantially higher effective ranks than instruction data,
implying richer gradient structures on more complex tasks. Our experiments also
highlight that models within the same family share similar gradient patterns
regardless of their sizes, whereas different model families diverge
significantly. Providing a unified view on the effects of data quality across
instruction and reasoning data, this work illuminates the interplay between
data quality and training stability, shedding novel insights into developing
better data exploration strategies for post-training.
Diffusion Distillation With Direct Preference Optimization For Efficient
  3D LiDAR Scene Completion
Authors: name: An Zhaol, name: Shengyuan Zhang, name: Ling Yang, name: Zejian Li, name: Jiale Wu, name: Haoran Xu, name: AnYang Wei, name: Perry Pengyun GU Lingyun Sun
Published: 2025-04-15 17:57:13

The application of diffusion models in 3D LiDAR scene completion is limited
due to diffusion's slow sampling speed. Score distillation accelerates
diffusion sampling but with performance degradation, while post-training with
direct policy optimization (DPO) boosts performance using preference data. This
paper proposes Distillation-DPO, a novel diffusion distillation framework for
LiDAR scene completion with preference aligment. First, the student model
generates paired completion scenes with different initial noises. Second, using
LiDAR scene evaluation metrics as preference, we construct winning and losing
sample pairs. Such construction is reasonable, since most LiDAR scene metrics
are informative but non-differentiable to be optimized directly. Third,
Distillation-DPO optimizes the student model by exploiting the difference in
score functions between the teacher and student models on the paired completion
scenes. Such procedure is repeated until convergence. Extensive experiments
demonstrate that, compared to state-of-the-art LiDAR scene completion diffusion
models, Distillation-DPO achieves higher-quality scene completion while
accelerating the completion speed by more than 5-fold. Our method is the first
to explore adopting preference learning in distillation to the best of our
knowledge and provide insights into preference-aligned distillation. Our code
is public available on https://github.com/happyw1nd/DistillationDPO.
MCP Safety Audit: LLMs with the Model Context Protocol Allow Major
  Security Exploits
Authors: name: Brandon Radosevich, name: John Halloran
Published: 2025-04-02 21:46:02

To reduce development overhead and enable seamless integration between
potential components comprising any given generative AI application, the Model
Context Protocol (MCP) (Anthropic, 2024) has recently been released and
subsequently widely adopted. The MCP is an open protocol that standardizes API
calls to large language models (LLMs), data sources, and agentic tools. By
connecting multiple MCP servers, each defined with a set of tools, resources,
and prompts, users are able to define automated workflows fully driven by LLMs.
However, we show that the current MCP design carries a wide range of security
risks for end users. In particular, we demonstrate that industry-leading LLMs
may be coerced into using MCP tools to compromise an AI developer's system
through various attacks, such as malicious code execution, remote access
control, and credential theft. To proactively mitigate these and related
attacks, we introduce a safety auditing tool, MCPSafetyScanner, the first
agentic tool to assess the security of an arbitrary MCP server. MCPScanner uses
several agents to (a) automatically determine adversarial samples given an MCP
server's tools and resources; (b) search for related vulnerabilities and
remediations based on those samples; and (c) generate a security report
detailing all findings. Our work highlights serious security issues with
general-purpose agentic workflows while also providing a proactive tool to
audit MCP server safety and address detected vulnerabilities before deployment.
  The described MCP server auditing tool, MCPSafetyScanner, is freely available
at: https://github.com/johnhalloran321/mcpSafetyScanner
Reasoning Models Can Be Effective Without Thinking
Authors: name: Wenjie Ma, name: Jingxuan He, name: Charlie Snell, name: Tyler Griggs, name: Sewon Min, name: Matei Zaharia
Published: 2025-04-14 04:08:16

Recent LLMs have significantly improved reasoning capabilities, primarily by
including an explicit, lengthy Thinking process as part of generation. In this
paper, we question whether this explicit thinking is necessary. Using the
state-of-the-art DeepSeek-R1-Distill-Qwen, we find that bypassing the thinking
process via simple prompting, denoted as NoThinking, can be surprisingly
effective. When controlling for the number of tokens, NoThinking outperforms
Thinking across a diverse set of seven challenging reasoning
datasets--including mathematical problem solving, formal theorem proving, and
coding--especially in low-budget settings, e.g., 51.3 vs. 28.9 on ACM 23 with
700 tokens. Notably, the performance of NoThinking becomes more competitive
with pass@k as k increases. Building on this observation, we demonstrate that a
parallel scaling approach that uses NoThinking to generate N outputs
independently and aggregates them is highly effective. For aggregation, we use
task-specific verifiers when available, or we apply simple best-of-N strategies
such as confidence-based selection. Our method outperforms a range of baselines
with similar latency using Thinking, and is comparable to Thinking with
significantly longer latency (up to 9x). Together, our research encourages a
reconsideration of the necessity of lengthy thinking processes, while also
establishing a competitive reference for achieving strong reasoning performance
in low-budget settings or at low latency using parallel scaling.
MIEB: Massive Image Embedding Benchmark
Authors: name: Chenghao Xiao, name: Isaac Chung, name: Imene Kerboua, name: Jamie Stirling, name: Xin Zhang, name: Márton Kardos, name: Roman Solomatin, name: Noura Al Moubayed, name: Kenneth Enevoldsen, name: Niklas Muennighoff
Published: 2025-04-14 17:54:28

Image representations are often evaluated through disjointed, task-specific
protocols, leading to a fragmented understanding of model capabilities. For
instance, it is unclear whether an image embedding model adept at clustering
images is equally good at retrieving relevant images given a piece of text. We
introduce the Massive Image Embedding Benchmark (MIEB) to evaluate the
performance of image and image-text embedding models across the broadest
spectrum to date. MIEB spans 38 languages across 130 individual tasks, which we
group into 8 high-level categories. We benchmark 50 models across our
benchmark, finding that no single method dominates across all task categories.
We reveal hidden capabilities in advanced vision models such as their accurate
visual representation of texts, and their yet limited capabilities in
interleaved encodings and matching images and texts in the presence of
confounders. We also show that the performance of vision encoders on MIEB
correlates highly with their performance when used in multimodal large language
models. Our code, dataset, and leaderboard are publicly available at
https://github.com/embeddings-benchmark/mteb.
DiffuMural: Restoring Dunhuang Murals with Multi-scale Diffusion
Authors: name: Puyu Han, name: Jiaju Kang, name: Yuhang Pan, name: Erting Pan, name: Zeyu Zhang, name: Qunchao Jin, name: Juntao Jiang, name: Zhichen Liu, name: Luqi Gong
Published: 2025-04-13 10:35:25

Large-scale pre-trained diffusion models have produced excellent results in
the field of conditional image generation. However, restoration of ancient
murals, as an important downstream task in this field, poses significant
challenges to diffusion model-based restoration methods due to its large
defective area and scarce training samples. Conditional restoration tasks are
more concerned with whether the restored part meets the aesthetic standards of
mural restoration in terms of overall style and seam detail, and such metrics
for evaluating heuristic image complements are lacking in current research. We
therefore propose DiffuMural, a combined Multi-scale convergence and
Collaborative Diffusion mechanism with ControlNet and cyclic consistency loss
to optimise the matching between the generated images and the conditional
control. DiffuMural demonstrates outstanding capabilities in mural restoration,
leveraging training data from 23 large-scale Dunhuang murals that exhibit
consistent visual aesthetics. The model excels in restoring intricate details,
achieving a coherent overall appearance, and addressing the unique challenges
posed by incomplete murals lacking factual grounding. Our evaluation framework
incorporates four key metrics to quantitatively assess incomplete murals:
factual accuracy, textural detail, contextual semantics, and holistic visual
coherence. Furthermore, we integrate humanistic value assessments to ensure the
restored murals retain their cultural and artistic significance. Extensive
experiments validate that our method outperforms state-of-the-art (SOTA)
approaches in both qualitative and quantitative metrics.
3D CoCa: Contrastive Learners are 3D Captioners
Authors: name: Ting Huang, name: Zeyu Zhang, name: Yemin Wang, name: Hao Tang
Published: 2025-04-13 11:10:47

3D captioning, which aims to describe the content of 3D scenes in natural
language, remains highly challenging due to the inherent sparsity of point
clouds and weak cross-modal alignment in existing methods. To address these
challenges, we propose 3D CoCa, a novel unified framework that seamlessly
combines contrastive vision-language learning with 3D caption generation in a
single architecture. Our approach leverages a frozen CLIP vision-language
backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to
capture geometric context, and a multi-modal decoder to generate descriptive
captions. Unlike prior two-stage methods that rely on explicit object
proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a
shared feature space, eliminating the need for external detectors or
handcrafted proposals. This joint training paradigm yields stronger spatial
reasoning and richer semantic grounding by aligning 3D and textual
representations. Extensive experiments on the ScanRefer and Nr3D benchmarks
demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by
10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at
https://github.com/AIGeeksGroup/3DCoCa.
MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in
  Multimodal Large Language Models
Authors: name: Pengfei Zhou, name: Fanrui Zhang, name: Xiaopeng Peng, name: Zhaopan Xu, name: Jiaxin Ai, name: Yansheng Qiu, name: Chuanhao Li, name: Zhen Li, name: Ming Li, name: Yukang Feng, name: Jianwen Sun, name: Haoquan Zhang, name: Zizhen Li, name: Xiaofeng Mao, name: Wangbo Zhao, name: Kai Wang, name: Xiaojun Chang, name: Wenqi Shao, name: Yang You, name: Kaipeng Zhang
Published: 2025-04-08 08:06:53

Multimodal reasoning, which integrates language and visual cues into problem
solving and decision making, is a fundamental aspect of human intelligence and
a crucial step toward artificial general intelligence. However, the evaluation
of multimodal reasoning capabilities in Multimodal Large Language Models
(MLLMs) remains inadequate. Most existing reasoning benchmarks are constrained
by limited data size, narrow domain coverage, and unstructured knowledge
distribution. To close these gaps, we introduce MDK12-Bench, a
multi-disciplinary benchmark assessing the reasoning capabilities of MLLMs via
real-world K-12 examinations. Spanning six disciplines (math, physics,
chemistry, biology, geography, and information science), our benchmark
comprises 140K reasoning instances across diverse difficulty levels from
primary school to 12th grade. It features 6,827 instance-level knowledge point
annotations based on a well-organized knowledge structure, detailed answer
explanations, difficulty labels and cross-year partitions, providing a robust
platform for comprehensive evaluation. Additionally, we present a novel dynamic
evaluation framework to mitigate data contamination issues by bootstrapping
question forms, question types, and image styles during evaluation. Extensive
experiment on MDK12-Bench reveals the significant limitation of current MLLMs
in multimodal reasoning. The findings on our benchmark provide insights into
the development of the next-generation models. Our data and codes are available
at https://github.com/LanceZPF/MDK12.