Orionnnn/hub-stats huggingface dataset:
DeepSeek vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and
  Summarization?
Authors: name: Daniil Larionov, name: Sotaro Takeshita, name: Ran Zhang, name: Yanran Chen, name: Christoph Leiter, name: Zhipin Wang, name: Christian Greisinger, name: Steffen Eger
Published: 2025-04-10 20:39:18

Reasoning-enabled large language models (LLMs) have recently demonstrated
impressive performance in complex logical and mathematical tasks, yet their
effectiveness in evaluating natural language generation remains unexplored.
This study systematically compares reasoning-based LLMs (DeepSeek-R1 and OpenAI
o3) with their non-reasoning counterparts across machine translation (MT) and
text summarization (TS) evaluation tasks. We evaluate eight models across three
architectural categories, including state-of-the-art reasoning models, their
distilled variants (ranging from 8B to 70B parameters), and equivalent
conventional, non-reasoning LLMs. Our experiments on WMT23 and SummEval
benchmarks reveal that the benefits of reasoning capabilities are highly model
and task-dependent: while OpenAI o3-mini models show consistent performance
improvements with increased reasoning intensity, DeepSeek-R1 underperforms
compared to its non-reasoning variant, with exception to certain aspects of TS
evaluation. Correlation analysis demonstrates that increased reasoning token
usage positively correlates with evaluation quality in o3-mini models.
Furthermore, our results show that distillation of reasoning capabilities
maintains reasonable performance in medium-sized models (32B) but degrades
substantially in smaller variants (8B). This work provides the first
comprehensive assessment of reasoning LLMs for NLG evaluation and offers
insights into their practical use.
Iterative Self-Training for Code Generation via Reinforced Re-Ranking
Authors: name: Nikita Sorokin, name: Ivan Sedykh, name: Valentin Malykh
Published: 2025-04-13 16:34:17

Generating high-quality code that solves complex programming tasks is
challenging, especially with current decoder-based models that produce highly
stochastic outputs. In code generation, even minor errors can easily break the
entire solution. Leveraging multiple sampled solutions can significantly
improve the overall output quality.
  One effective way to enhance code generation is by pairing a code generation
model with a reranker model, which selects the best solution from the generated
samples. We propose a novel iterative self-training approach for self-training
reranker models using Proximal Policy Optimization (PPO), aimed at improving
both reranking accuracy and the overall code generation process. Unlike
traditional PPO approaches, where the focus is on optimizing a generative model
with a reward model, our approach emphasizes the development of a robust
reward/reranking model. This model improves the quality of generated code
through reranking and addresses problems and errors that the reward model might
overlook during PPO alignment with the reranker. Our method iteratively refines
the training dataset by re-evaluating outputs, identifying high-scoring
negative examples, and incorporating them into the training loop, that boosting
model performance.
  Our evaluation on the MultiPL-E dataset demonstrates that our 13.4B parameter
model outperforms a 33B model in code generation quality while being three
times faster. Moreover, it achieves performance comparable to GPT-4 and
surpasses it in one programming language.
The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via
  Agentic Tree Search
Authors: name: Yutaro Yamada, name: Robert Tjarko Lange, name: Cong Lu, name: Shengran Hu, name: Chris Lu, name: Jakob Foerster, name: Jeff Clune, name: David Ha
Published: 2025-04-10 18:44:41

AI is increasingly playing a pivotal role in transforming how scientific
discoveries are made. We introduce The AI Scientist-v2, an end-to-end agentic
system capable of producing the first entirely AI generated
peer-review-accepted workshop paper. This system iteratively formulates
scientific hypotheses, designs and executes experiments, analyzes and
visualizes data, and autonomously authors scientific manuscripts. Compared to
its predecessor (v1, Lu et al., 2024 arXiv:2408.06292), The AI Scientist-v2
eliminates the reliance on human-authored code templates, generalizes
effectively across diverse machine learning domains, and leverages a novel
progressive agentic tree-search methodology managed by a dedicated experiment
manager agent. Additionally, we enhance the AI reviewer component by
integrating a Vision-Language Model (VLM) feedback loop for iterative
refinement of content and aesthetics of the figures. We evaluated The AI
Scientist-v2 by submitting three fully autonomous manuscripts to a
peer-reviewed ICLR workshop. Notably, one manuscript achieved high enough
scores to exceed the average human acceptance threshold, marking the first
instance of a fully AI-generated paper successfully navigating a peer review.
This accomplishment highlights the growing capability of AI in conducting all
aspects of scientific research. We anticipate that further advancements in
autonomous scientific discovery technologies will profoundly impact human
knowledge generation, enabling unprecedented scalability in research
productivity and significantly accelerating scientific breakthroughs, greatly
benefiting society at large. We have open-sourced the code at
https://github.com/SakanaAI/AI-Scientist-v2 to foster the future development of
this transformative technology. We also discuss the role of AI in science,
including AI safety.
VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search
Authors: name: Yikun Wang, name: Siyin Wang, name: Qinyuan Cheng, name: Zhaoye Fei, name: Liang Ding, name: Qipeng Guo, name: Dacheng Tao, name: Xipeng Qiu
Published: 2025-04-12 08:37:30

Recent advancements in Large Vision-Language Models have showcased remarkable
capabilities. However, they often falter when confronted with complex reasoning
tasks that humans typically address through visual aids and deliberate,
step-by-step thinking. While existing methods have explored text-based slow
thinking or rudimentary visual assistance, they fall short of capturing the
intricate, interleaved nature of human visual-verbal reasoning processes. To
overcome these limitations and inspired by the mechanisms of slow thinking in
human cognition, we introduce VisuoThink, a novel framework that seamlessly
integrates visuospatial and linguistic domains. VisuoThink facilitates
multimodal slow thinking by enabling progressive visual-textual reasoning and
incorporates test-time scaling through look-ahead tree search. Extensive
experiments demonstrate that VisuoThink significantly enhances reasoning
capabilities via inference-time scaling, even without fine-tuning, achieving
state-of-the-art performance in tasks involving geometry and spatial reasoning.
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
Authors: name: Junxiong Wang, name: Wen-Ding Li, name: Daniele Paliotta, name: Daniel Ritter, name: Alexander M. Rush, name: Tri Dao
Published: 2025-04-14 17:38:25

Effective reasoning is crucial to solving complex mathematical problems.
Recent large language models (LLMs) have boosted performance by scaling
test-time computation through long chain-of-thought reasoning. However,
transformer-based models are inherently limited in extending context length due
to their quadratic computational complexity and linear memory requirements. In
this paper, we introduce a novel hybrid linear RNN reasoning model, M1, built
on the Mamba architecture, which allows memory-efficient inference. Our
approach leverages a distillation process from existing reasoning models and is
further enhanced through RL training. Experimental results on the AIME and MATH
benchmarks show that M1 not only outperforms previous linear RNN models but
also matches the performance of state-of-the-art Deepseek R1 distilled
reasoning models at a similar scale. We also compare our generation speed with
a highly performant general purpose inference engine, vLLM, and observe more
than a 3x speedup compared to a same size transformer. With throughput speedup,
we are able to achieve higher accuracy compared to DeepSeek R1 distilled
transformer reasoning models under a fixed generation time budget using
self-consistency voting. Overall, we introduce a hybrid Mamba reasoning model
and provide a more effective approach to scaling test-time generation using
self-consistency or long chain of thought reasoning.
Breaking the Data Barrier -- Building GUI Agents Through Task
  Generalization
Authors: name: Junlei Zhang, name: Zichen Ding, name: Chang Ma, name: Zijie Chen, name: Qiushi Sun, name: Zhenzhong Lan, name: Junxian He
Published: 2025-04-14 11:35:02

Graphical User Interface (GUI) agents offer cross-platform solutions for
automating complex digital tasks, with significant potential to transform
productivity workflows. However, their performance is often constrained by the
scarcity of high-quality trajectory data. To address this limitation, we
propose training Vision Language Models (VLMs) on data-rich,
reasoning-intensive tasks during a dedicated mid-training stage, and then
examine how incorporating these tasks facilitates generalization to GUI
planning scenarios. Specifically, we explore a range of tasks with readily
available instruction-tuning data, including GUI perception, multimodal
reasoning, and textual reasoning. Through extensive experiments across 11
mid-training tasks, we demonstrate that: (1) Task generalization proves highly
effective, yielding substantial improvements across most settings. For
instance, multimodal mathematical reasoning enhances performance on
AndroidWorld by an absolute 6.3%. Remarkably, text-only mathematical data
significantly boosts GUI web agent performance, achieving a 5.6% improvement on
WebArena and 5.4% improvement on AndroidWorld, underscoring notable cross-modal
generalization from text-based to visual domains; (2) Contrary to prior
assumptions, GUI perception data - previously considered closely aligned with
GUI agent tasks and widely utilized for training - has a comparatively limited
impact on final performance; (3) Building on these insights, we identify the
most effective mid-training tasks and curate optimized mixture datasets,
resulting in absolute performance gains of 8.0% on WebArena and 12.2% on
AndroidWorld. Our work provides valuable insights into cross-domain knowledge
transfer for GUI agents and offers a practical approach to addressing data
scarcity challenges in this emerging field. The code, data and models will be
available at https://github.com/hkust-nlp/GUIMid.
How new data permeates LLM knowledge and how to dilute it
Authors: name: Chen Sun, name: Renat Aksitov, name: Andrey Zhmoginov, name: Nolan Andrew Miller, name: Max Vladymyrov, name: Ulrich Rueckert, name: Been Kim, name: Mark Sandler
Published: 2025-04-13 11:25:04

Large language models learn and continually learn through the accumulation of
gradient-based updates, but how individual pieces of new information affect
existing knowledge, leading to both beneficial generalization and problematic
hallucination, remains poorly understood. We demonstrate that when learning new
information, LLMs exhibit a "priming" effect: learning a new fact can cause the
model to inappropriately apply that knowledge in unrelated contexts. To
systematically study this phenomenon, we introduce "Outlandish," a carefully
curated dataset of 1320 diverse text samples designed to probe how new
knowledge permeates through an LLM's existing knowledge base. Using this
dataset, we show that the degree of priming after learning new information can
be predicted by measuring the token probability of key words before learning.
This relationship holds robustly across different model architectures (PALM-2,
Gemma, Llama), sizes, and training stages. Finally, we develop two novel
techniques to modulate how new knowledge affects existing model behavior: (1) a
``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update
pruning method. These approaches reduce undesirable priming effects by 50-95\%
while preserving the model's ability to learn new information. Our findings
provide both empirical insights into how LLMs learn and practical tools for
improving the specificity of knowledge insertion in language models. Further
materials: https://sunchipsster1.github.io/projects/outlandish/
LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety
  in Large Language Models
Authors: name: Minqian Liu, name: Zhiyang Xu, name: Xinyi Zhang, name: Heajun An, name: Sarvech Qadir, name: Qi Zhang, name: Pamela J. Wisniewski, name: Jin-Hee Cho, name: Sang Won Lee, name: Ruoxi Jia, name: Lifu Huang
Published: 2025-04-14 17:20:34

Recent advancements in Large Language Models (LLMs) have enabled them to
approach human-level persuasion capabilities. However, such potential also
raises concerns about the safety risks of LLM-driven persuasion, particularly
their potential for unethical influence through manipulation, deception,
exploitation of vulnerabilities, and many other harmful tactics. In this work,
we present a systematic investigation of LLM persuasion safety through two
critical aspects: (1) whether LLMs appropriately reject unethical persuasion
tasks and avoid unethical strategies during execution, including cases where
the initial persuasion goal appears ethically neutral, and (2) how influencing
factors like personality traits and external pressures affect their behavior.
To this end, we introduce PersuSafety, the first comprehensive framework for
the assessment of persuasion safety which consists of three stages, i.e.,
persuasion scene creation, persuasive conversation simulation, and persuasion
safety assessment. PersuSafety covers 6 diverse unethical persuasion topics and
15 common unethical strategies. Through extensive experiments across 8 widely
used LLMs, we observe significant safety concerns in most LLMs, including
failing to identify harmful persuasion tasks and leveraging various unethical
persuasion strategies. Our study calls for more attention to improve safety
alignment in progressive and goal-driven conversations such as persuasion.
Executable Functional Abstractions: Inferring Generative Programs for
  Advanced Math Problems
Authors: name: Zaid Khan, name: Elias Stengel-Eskin, name: Archiki Prasad, name: Jaemin Cho, name: Mohit Bansal
Published: 2025-04-14 00:06:48

Scientists often infer abstract procedures from specific instances of
problems and use the abstractions to generate new, related instances. For
example, programs encoding the formal rules and properties of a system have
been useful in fields ranging from RL (procedural environments) to physics
(simulation engines). These programs can be seen as functions which execute to
different outputs based on their parameterizations (e.g., gridworld
configuration or initial physical conditions). We introduce the term EFA
(Executable Functional Abstraction) to denote such programs for math problems.
EFA-like constructs have been shown to be useful for math reasoning as problem
generators for stress-testing models. However, prior work has been limited to
abstractions for grade-school math (whose simple rules are easy to encode in
programs), while generating EFAs for advanced math has thus far required human
engineering. We explore the automatic construction of EFAs for advanced math
problems. We operationalize the task of automatically constructing EFAs as a
program synthesis task, and develop EFAGen, which conditions an LLM on a seed
math problem and its step-by-step solution to generate candidate EFA programs
that are faithful to the generalized problem and solution class underlying the
seed problem. Furthermore, we formalize properties any valid EFA must possess
in terms of executable unit tests, and show how the tests can be used as
verifiable rewards to train LLMs to become better writers of EFAs. We
demonstrate that EFAs constructed by EFAGen behave rationally by remaining
faithful to seed problems, produce learnable problem variations, and that
EFAGen can infer EFAs across multiple diverse sources of competition-level math
problems. Finally, we show downstream uses of model-written EFAs e.g. finding
problem variations that are harder or easier for a learner to solve, as well as
data generation.
LLM-SRBench: A New Benchmark for Scientific Equation Discovery with
  Large Language Models
Authors: name: Parshin Shojaee, name: Ngoc-Hieu Nguyen, name: Kazem Meidani, name: Amir Barati Farimani, name: Khoa D Doan, name: Chandan K Reddy
Published: 2025-04-14 17:00:13

Scientific equation discovery is a fundamental task in the history of
scientific progress, enabling the derivation of laws governing natural
phenomena. Recently, Large Language Models (LLMs) have gained interest for this
task due to their potential to leverage embedded scientific knowledge for
hypothesis generation. However, evaluating the true discovery capabilities of
these methods remains challenging, as existing benchmarks often rely on common
equations that are susceptible to memorization by LLMs, leading to inflated
performance metrics that do not reflect discovery. In this paper, we introduce
LLM-SRBench, a comprehensive benchmark with 239 challenging problems across
four scientific domains specifically designed to evaluate LLM-based scientific
equation discovery methods while preventing trivial memorization. Our benchmark
comprises two main categories: LSR-Transform, which transforms common physical
models into less common mathematical representations to test reasoning beyond
memorized forms, and LSR-Synth, which introduces synthetic, discovery-driven
problems requiring data-driven reasoning. Through extensive evaluation of
several state-of-the-art methods, using both open and closed LLMs, we find that
the best-performing system so far achieves only 31.5% symbolic accuracy. These
findings highlight the challenges of scientific equation discovery, positioning
LLM-SRBench as a valuable resource for future research.
S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability
  of Large Reasoning Models
Authors: name: Wenyuan Zhang, name: Shuaiyi Nie, name: Xinghua Zhang, name: Zefeng Zhang, name: Tingwen Liu
Published: 2025-04-14 16:13:23

We introduce S1-Bench, a novel benchmark designed to evaluate Large Reasoning
Models' (LRMs) performance on simple tasks that favor intuitive system 1
thinking rather than deliberative system 2 reasoning. While LRMs have achieved
significant breakthroughs in complex reasoning tasks through explicit chains of
thought, their reliance on deep analytical thinking may limit their system 1
thinking capabilities. Moreover, a lack of benchmark currently exists to
evaluate LRMs' performance in tasks that require such capabilities. To fill
this gap, S1-Bench presents a set of simple, diverse, and naturally clear
questions across multiple domains and languages, specifically designed to
assess LRMs' performance in such tasks. Our comprehensive evaluation of 22 LRMs
reveals significant lower efficiency tendencies, with outputs averaging 15.5
times longer than those of traditional small LLMs. Additionally, LRMs often
identify correct answers early but continue unnecessary deliberation, with some
models even producing numerous errors. These findings highlight the rigid
reasoning patterns of current LRMs and underscore the substantial development
needed to achieve balanced dual-system thinking capabilities that can adapt
appropriately to task complexity.
InternVL3: Exploring Advanced Training and Test-Time Recipes for
  Open-Source Multimodal Models
Authors: name: Jinguo Zhu, name: Weiyun Wang, name: Zhe Chen, name: Zhaoyang Liu, name: Shenglong Ye, name: Lixin Gu, name: Yuchen Duan, name: Hao Tian, name: Weijie Su, name: Jie Shao, name: Zhangwei Gao, name: Erfei Cui, name: Yue Cao, name: Yangzhou Liu, name: Weiye Xu, name: Hao Li, name: Jiahao Wang, name: Han Lv, name: Dengnian Chen, name: Songze Li, name: Yinan He, name: Tan Jiang, name: Jiapeng Luo, name: Yi Wang, name: Conghui He, name: Botian Shi, name: Xingcheng Zhang, name: Wenqi Shao, name: Junjun He, name: Yingtong Xiong, name: Wenwen Qu, name: Peng Sun, name: Penglong Jiao, name: Lijun Wu, name: Kaipeng Zhang, name: Huipeng Deng, name: Jiaye Ge, name: Kai Chen, name: Limin Wang, name: Min Dou, name: Lewei Lu, name: Xizhou Zhu, name: Tong Lu, name: Dahua Lin, name: Yu Qiao, name: Jifeng Dai, name: Wenhai Wang
Published: 2025-04-14 17:59:25

We introduce InternVL3, a significant advancement in the InternVL series
featuring a native multimodal pre-training paradigm. Rather than adapting a
text-only large language model (LLM) into a multimodal large language model
(MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and
linguistic capabilities from both diverse multimodal data and pure-text corpora
during a single pre-training stage. This unified training paradigm effectively
addresses the complexities and alignment challenges commonly encountered in
conventional post-hoc training pipelines for MLLMs. To further improve
performance and scalability, InternVL3 incorporates variable visual position
encoding (V2PE) to support extended multimodal contexts, employs advanced
post-training techniques such as supervised fine-tuning (SFT) and mixed
preference optimization (MPO), and adopts test-time scaling strategies
alongside an optimized training infrastructure. Extensive empirical evaluations
demonstrate that InternVL3 delivers superior performance across a wide range of
multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the
MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its
capabilities remain highly competitive with leading proprietary models,
including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also
maintaining strong pure-language proficiency. In pursuit of open-science
principles, we will publicly release both the training data and model weights
to foster further research and development in next-generation MLLMs.
Mavors: Multi-granularity Video Representation for Multimodal Large
  Language Model
Authors: name: Yang Shi, name: Jiaheng Liu, name: Yushuo Guan, name: Zhenhua Wu, name: Yuanxing Zhang, name: Zihao Wang, name: Weihong Lin, name: Jingyun Hua, name: Zekun Wang, name: Xinlong Chen, name: Bohan Zeng, name: Wentao Zhang, name: Fuzheng Zhang, name: Wenjing Yang, name: Di Zhang
Published: 2025-04-14 10:14:44

Long-context video understanding in multimodal large language models (MLLMs)
faces a critical challenge: balancing computational efficiency with the
retention of fine-grained spatio-temporal patterns. Existing approaches (e.g.,
sparse sampling, dense sampling with low resolution, and token compression)
suffer from significant information loss in temporal dynamics, spatial details,
or subtle interactions, particularly in videos with complex motion or varying
resolutions. To address this, we propose Mavors, a novel framework
that introduces Multi-granularity
video representation for holistic
long-video modeling. Specifically, Mavors directly encodes raw video content
into latent representations through two core components: 1) an Intra-chunk
Vision Encoder (IVE) that preserves high-resolution spatial features via 3D
convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator
(IFA) that establishes temporal coherence across chunks using transformer-based
dependency modeling with chunk-level rotary position encodings. Moreover, the
framework unifies image and video understanding by treating images as
single-frame videos via sub-image decomposition. Experiments across diverse
benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity
and temporal continuity, significantly outperforming existing methods in tasks
requiring fine-grained spatio-temporal reasoning.