Orionnnn/hub-stats huggingface dataset:
Data Efficacy for Language Model Training
Authors: name: Yalun Dai, name: Yangyu Huang, name: Xin Zhang, name: Wenshan Wu, name: Chong Li, name: Wenhui Lu, name: Shijie Cao, name: Li Dong, name: Scarlett Li
Published: 2025-06-26 17:59:07

Data is fundamental to the training of language models (LM). Recent research
has been dedicated to data efficiency, which aims to maximize performance by
selecting a minimal or optimal subset of training data. Techniques such as data
filtering, sampling, and selection play a crucial role in this area. To
complement it, we define Data Efficacy, which focuses on maximizing performance
by optimizing the organization of training data and remains relatively
underexplored. This work introduces a general paradigm, DELT, for considering
data efficacy in LM training, which highlights the significance of training
data organization. DELT comprises three components: Data Scoring, Data
Selection, and Data Ordering. Among these components, we design
Learnability-Quality Scoring (LQS), as a new instance of Data Scoring, which
considers both the learnability and quality of each data sample from the
gradient consistency perspective. We also devise Folding Ordering (FO), as a
novel instance of Data Ordering, which addresses issues such as model
forgetting and data distribution bias. Comprehensive experiments validate the
data efficacy in LM training, which demonstrates the following: Firstly,
various instances of the proposed DELT enhance LM performance to varying
degrees without increasing the data scale and model size. Secondly, among these
instances, the combination of our proposed LQS for data scoring and Folding for
data ordering achieves the most significant improvement. Lastly, data efficacy
can be achieved together with data efficiency by applying data selection.
Therefore, we believe that data efficacy is a promising foundational area in LM
training.
GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable
  Reinforcement Learning
Authors: name: Wenyi Hong, name: Wenmeng Yu, name: Xiaotao Gu, name: Guo Wang, name: Guobing Gan, name: Haomiao Tang, name: Jiale Cheng, name: Ji Qi, name: Junhui Ji, name: Lihang Pan, name: Shuaiqi Duan, name: Weihan Wang, name: Yan Wang, name: Yean Cheng, name: Zehai He, name: Zhe Su, name: Zhen Yang, name: Ziyang Pan, name: Aohan Zeng, name: Baoxu Wang, name: Boyan Shi, name: Changyu Pang, name: Chenhui Zhang, name: Da Yin, name: Fan Yang, name: Guoqing Chen, name: Jiazheng Xu, name: Jiali Chen, name: Jing Chen, name: Jinhao Chen, name: Jinghao Lin, name: Jinjiang Wang, name: Junjie Chen, name: Leqi Lei, name: Leyi Pan, name: Mingzhi Zhang, name: Qinkai Zheng, name: Sheng Yang, name: Shi Zhong, name: Shiyu Huang, name: Shuyuan Zhao, name: Siyan Xue, name: Shangqin Tu, name: Shengbiao Meng, name: Tianshu Zhang, name: Tianwei Luo, name: Tianxiang Hao, name: Tianle Gong, name: Wenkai Li, name: Wei Jia, name: Xin Lyu, name: Xuancheng Huang, name: Yanling Wang, name: Yadong Xue, name: Yanfeng Wang, name: Yifan An, name: Yifan Du, name: Yiming Shi, name: Yiheng Huang, name: Yilin Niu, name: Yuan Wang, name: Yuanchang Yue, name: Yuchen Li, name: Yutao Zhang, name: Yuxuan Zhang, name: Zhanxiao Du, name: Zhenyu Hou, name: Zhao Xue, name: Zhengxiao Du, name: Zihan Wang, name: Peng Zhang, name: Debing Liu, name: Bin Xu, name: Juanzi Li, name: Minlie Huang, name: Yuxiao Dong, name: Jie Tang
Published: 2025-07-01 17:55:04

We present GLM-4.1V-Thinking, a vision-language model (VLM) designed to
advance general-purpose multimodal reasoning. In this report, we share our key
findings in the development of the reasoning-centric training framework. We
first develop a capable vision foundation model with significant potential
through large-scale pre-training, which arguably sets the upper bound for the
final performance. Reinforcement Learning with Curriculum Sampling (RLCS) then
unlocks the full potential of the model, leading to comprehensive capability
enhancement across a diverse range of tasks, including STEM problem solving,
video understanding, content recognition, coding, grounding, GUI-based agents,
and long document understanding, among others. To facilitate research in this
field, we open-source GLM-4.1V-9B-Thinking, which achieves state-of-the-art
performance among models of comparable size. In a comprehensive evaluation
across 28 public benchmarks, our model outperforms Qwen2.5-VL-7B on nearly all
tasks and achieves comparable or even superior performance on 18 benchmarks
relative to the significantly larger Qwen2.5-VL-72B. Notably,
GLM-4.1V-9B-Thinking also demonstrates competitive or superior performance
compared to closed-source models such as GPT-4o on challenging tasks including
long document understanding and STEM reasoning, further underscoring its strong
capabilities. Code, models and more information are released at
https://github.com/THUDM/GLM-4.1V-Thinking.
Radial Attention: O(nlog n) Sparse Attention with Energy Decay for
  Long Video Generation
Authors: name: Xingyang Li, name: Muyang Li, name: Tianle Cai, name: Haocheng Xi, name: Shuo Yang, name: Yujun Lin, name: Lvmin Zhang, name: Songlin Yang, name: Jinbo Hu, name: Kelly Peng, name: Maneesh Agrawala, name: Ion Stoica, name: Kurt Keutzer, name: Song Han
Published: 2025-06-24 17:59:59

Recent advances in diffusion models have enabled high-quality video
generation, but the additional temporal dimension significantly increases
computational costs, making training and inference on long videos prohibitively
expensive. In this paper, we identify a phenomenon we term Spatiotemporal
Energy Decay in video diffusion models: post-softmax attention scores diminish
as spatial and temporal distance between tokens increase, akin to the physical
decay of signal or waves over space and time in nature. Motivated by this, we
propose Radial Attention, a scalable sparse attention mechanism with O(n log
n) complexity that translates energy decay into exponentially decaying compute
density, which is significantly more efficient than standard O(n^2) dense
attention and more expressive than linear attention. Specifically, Radial
Attention employs a simple, static attention mask where each token attends to
spatially nearby tokens, with the attention window size shrinking with temporal
distance. Moreover, it allows pre-trained video diffusion models to extend
their generation length with efficient LoRA-based fine-tuning. Extensive
experiments show that Radial Attention maintains video quality across
Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9times speedup
over the original dense attention. With minimal tuning, it enables video
generation up to 4times longer while reducing training costs by up to
4.4times compared to direct fine-tuning and accelerating inference by up to
3.7times compared to dense attention inference.
Peccavi: Visual Paraphrase Attack Safe and Distortion Free Image
  Watermarking Technique for AI-Generated Images
Authors: name: Shreyas Dixit, name: Ashhar Aziz, name: Shashwat Bajpai, name: Vasu Sharma, name: Aman Chadha, name: Vinija Jain, name: Amitava Das
Published: 2025-06-28 17:34:08

A report by the European Union Law Enforcement Agency predicts that by 2026,
up to 90 percent of online content could be synthetically generated, raising
concerns among policymakers, who cautioned that "Generative AI could act as a
force multiplier for political disinformation. The combined effect of
generative text, images, videos, and audio may surpass the influence of any
single modality." In response, California's Bill AB 3211 mandates the
watermarking of AI-generated images, videos, and audio. However, concerns
remain regarding the vulnerability of invisible watermarking techniques to
tampering and the potential for malicious actors to bypass them entirely.
Generative AI-powered de-watermarking attacks, especially the newly introduced
visual paraphrase attack, have shown an ability to fully remove watermarks,
resulting in a paraphrase of the original image. This paper introduces PECCAVI,
the first visual paraphrase attack-safe and distortion-free image watermarking
technique. In visual paraphrase attacks, an image is altered while preserving
its core semantic regions, termed Non-Melting Points (NMPs). PECCAVI
strategically embeds watermarks within these NMPs and employs multi-channel
frequency domain watermarking. It also incorporates noisy burnishing to counter
reverse-engineering efforts aimed at locating NMPs to disrupt the embedded
watermark, thereby enhancing durability. PECCAVI is model-agnostic. All
relevant resources and codes will be open-sourced.
DiffuCoder: Understanding and Improving Masked Diffusion Models for Code
  Generation
Authors: name: Shansan Gong, name: Ruixiang Zhang, name: Huangjie Zheng, name: Jiatao Gu, name: Navdeep Jaitly, name: Lingpeng Kong, name: Yizhe Zhang
Published: 2025-06-25 17:35:47

Diffusion large language models (dLLMs) are compelling alternatives to
autoregressive (AR) models because their denoising models operate over the
entire sequence. The global planning and iterative refinement features of dLLMs
are particularly useful for code generation. However, current training and
inference mechanisms for dLLMs in coding are still under-explored. To demystify
the decoding behavior of dLLMs and unlock their potential for coding, we
systematically investigate their denoising processes and reinforcement learning
(RL) methods. We train a 7B dLLM, DiffuCoder, on 130B tokens of code.
Using this model as a testbed, we analyze its decoding behavior, revealing how
it differs from that of AR models: (1) dLLMs can decide how causal their
generation should be without relying on semi-AR decoding, and (2) increasing
the sampling temperature diversifies not only token choices but also their
generation order. This diversity creates a rich search space for RL rollouts.
For RL training, to reduce the variance of token log-likelihood estimates and
maintain training efficiency, we propose coupled-GRPO, a novel
sampling scheme that constructs complementary mask noise for completions used
in training. In our experiments, coupled-GRPO significantly improves
DiffuCoder's performance on code generation benchmarks (+4.4\% on EvalPlus) and
reduces reliance on AR causal during decoding. Our work provides deeper insight
into the machinery of dLLM generation and offers an effective, diffusion-native
RL training framework. https://github.com/apple/ml-diffucoder.
SciArena: An Open Evaluation Platform for Foundation Models in
  Scientific Literature Tasks
Authors: name: Yilun Zhao, name: Kaiyan Zhang, name: Tiansheng Hu, name: Sihong Wu, name: Ronan Le Bras, name: Taira Anderson, name: Jonathan Bragg, name: Joseph Chee Chang, name: Jesse Dodge, name: Matt Latzke, name: Yixin Liu, name: Charles McGrady, name: Xiangru Tang, name: Zihang Wang, name: Chen Zhao, name: Hannaneh Hajishirzi, name: Doug Downey, name: Arman Cohan
Published: 2025-07-01 17:51:59

We present SciArena, an open and collaborative platform for evaluating
foundation models on scientific literature tasks. Unlike traditional benchmarks
for scientific literature understanding and synthesis, SciArena engages the
research community directly, following the Chatbot Arena evaluation approach of
community voting on model comparisons. By leveraging collective intelligence,
SciArena offers a community-driven evaluation of model performance on
open-ended scientific tasks that demand literature-grounded, long-form
responses. The platform currently supports 23 open-source and proprietary
foundation models and has collected over 13,000 votes from trusted researchers
across diverse scientific domains. We analyze the data collected so far and
confirm that the submitted questions are diverse, aligned with real-world
literature needs, and that participating researchers demonstrate strong
self-consistency and inter-annotator agreement in their evaluations. We discuss
the results and insights based on the model ranking leaderboard. To further
promote research in building model-based automated evaluation systems for
literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based
on our collected preference data. The benchmark measures the accuracy of models
in judging answer quality by comparing their pairwise assessments with human
votes. Our experiments highlight the benchmark's challenges and emphasize the
need for more reliable automated evaluation methods.
Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive
  Foundations for Artificial General Intelligence and its Societal Impact
Authors: name: Rizwan Qureshi, name: Ranjan Sapkota, name: Abbas Shah, name: Amgad Muneer, name: Anas Zafar, name: Ashmal Vayani, name: Maged Shoman, name: Abdelrahman B. M. Eldaly, name: Kai Zhang, name: Ferhat Sadak, name: Shaina Raza, name: Xinqi Fan, name: Ravid Shwartz-Ziv, name: Hong Yan, name: Vinjia Jain, name: Aman Chadha, name: Manoj Karkee, name: Jia Wu, name: Philip Torr, name: Seyedali Mirjalili
Published: 2025-07-01 16:52:25

Can machines truly think, reason and act in domains like humans? This
enduring question continues to shape the pursuit of Artificial General
Intelligence (AGI). Despite the growing capabilities of models such as GPT-4.5,
DeepSeek, Claude 3.5 Sonnet, Phi-4, and Grok 3, which exhibit multimodal
fluency and partial reasoning, these systems remain fundamentally limited by
their reliance on token-level prediction and lack of grounded agency. This
paper offers a cross-disciplinary synthesis of AGI development, spanning
artificial intelligence, cognitive neuroscience, psychology, generative models,
and agent-based systems. We analyze the architectural and cognitive foundations
of general intelligence, highlighting the role of modular reasoning, persistent
memory, and multi-agent coordination. In particular, we emphasize the rise of
Agentic RAG frameworks that combine retrieval, planning, and dynamic tool use
to enable more adaptive behavior. We discuss generalization strategies,
including information compression, test-time adaptation, and training-free
methods, as critical pathways toward flexible, domain-agnostic intelligence.
Vision-Language Models (VLMs) are reexamined not just as perception modules but
as evolving interfaces for embodied understanding and collaborative task
completion. We also argue that true intelligence arises not from scale alone
but from the integration of memory and reasoning: an orchestration of modular,
interactive, and self-improving components where compression enables adaptive
behavior. Drawing on advances in neurosymbolic systems, reinforcement learning,
and cognitive scaffolding, we explore how recent architectures begin to bridge
the gap between statistical learning and goal-directed cognition. Finally, we
identify key scientific, technical, and ethical challenges on the path to AGI.
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional
  Multimodal Embeddings
Authors: name: Haonan Chen, name: Hong Liu, name: Yuping Luo, name: Liang Wang, name: Nan Yang, name: Furu Wei, name: Zhicheng Dou
Published: 2025-06-29 06:41:00

Multimodal embedding models, built upon causal Vision Language Models (VLMs),
have shown promise in various tasks. However, current approaches face three key
limitations: the use of causal attention in VLM backbones is suboptimal for
embedding tasks; scalability issues due to reliance on high-quality labeled
paired data for contrastive learning; and limited diversity in training
objectives and data. To address these issues, we propose MoCa, a two-stage
framework for transforming pre-trained VLMs into effective bidirectional
multimodal embedding models. The first stage, Modality-aware Continual
Pre-training, introduces a joint reconstruction objective that simultaneously
denoises interleaved text and image inputs, enhancing bidirectional
context-aware reasoning. The second stage, Heterogeneous Contrastive
Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple
image-caption pairs to enhance generalization and alignment. Our method
addresses the stated limitations by introducing bidirectional attention through
continual pre-training, scaling effectively with massive unlabeled datasets via
joint reconstruction objectives, and utilizing diverse multimodal data for
enhanced representation robustness. Experiments demonstrate that MoCa
consistently improves performance across MMEB and ViDoRe-v2 benchmarks,
achieving new state-of-the-art results, and exhibits strong scalability with
both model size and training data on MMEB.
HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context
Authors: name: Qize Yang, name: Shimin Yao, name: Weixuan Chen, name: Shenghao Fu, name: Detao Bai, name: Jiaxing Zhao, name: Boyuan Sun, name: Bowen Yin, name: Xihan Wei, name: Jingren Zhou
Published: 2025-06-26 14:01:03

With the rapid evolution of multimodal large language models, the capacity to
deeply understand and interpret human intentions has emerged as a critical
capability, which demands detailed and thoughtful reasoning. In recent studies,
Reinforcement Learning (RL) has demonstrated potential in enhancing the
reasoning capabilities of Large Language Models (LLMs). Nonetheless, the
challenges associated with adapting RL to multimodal data and formats remain
largely unaddressed. In this paper, we identify two issues in existing
multimodal reasoning models: insufficient global context understanding and
shortcut problems. Insufficient context understanding can happen when a model
misinterprets multimodal context, resulting in incorrect answers. The shortcut
problem occurs when the model overlooks crucial clues in multimodal inputs,
directly addressing the query without considering the multimodal information.
To tackle these issues, we emphasize the necessity for the model to reason with
a clear understanding of the global context within multimodal inputs. This
global context understanding can effectively prevent the model from overlooking
key multimodal cues and ensure a thorough reasoning process. To ensure the
accurate interpretation of multimodal context information, we implement a
context reward judged by a large language model, alongside format and accuracy
rewards. Additionally, to improve complex reasoning capability, we employ the
LLM to assess the logical reward, determining whether the reasoning process
successfully integrates multimodal information with logical methods. We also
introduce a reasoning omni-modal benchmark, IntentBench, aimed at evaluating
models in understanding complex human intentions and emotions. Our proposed
method demonstrates advanced performance across multiple omni-modal benchmarks
compared to other open-source omni-modal models.
Does Math Reasoning Improve General LLM Capabilities? Understanding
  Transferability of LLM Reasoning
Authors: name: Maggie Huan, name: Yuetai Li, name: Tuney Zheng, name: Xiaoyu Xu, name: Seungone Kim, name: Minxin Du, name: Radha Poovendran, name: Graham Neubig, name: Xiang Yue
Published: 2025-07-01 05:23:05

Math reasoning has become the poster child of progress in large language
models (LLMs), with new models rapidly surpassing human-level performance on
benchmarks like MATH and AIME. But as math leaderboards improve week by week,
it is worth asking: do these gains reflect broader problem-solving ability or
just narrow overfitting? To answer this question, we evaluate over 20
open-weight reasoning-tuned models across a broad suite of tasks, including
math, scientific QA, agent planning, coding, and standard
instruction-following. We surprisingly find that most models that succeed in
math fail to transfer their gains to other domains. To rigorously study this
phenomenon, we conduct controlled experiments on Qwen3-14B models using
math-only data but different tuning methods. We find that reinforcement
learning (RL)-tuned models generalize well across domains, while supervised
fine-tuning (SFT)-tuned models often forget general capabilities. Latent-space
representation and token-space distribution shift analyses reveal that SFT
induces substantial representation and output drift, while RL preserves
general-domain structure. Our results suggest a need to rethink standard
post-training recipes, particularly the reliance on SFT-distilled data for
advancing reasoning models.
RExBench: Can coding agents autonomously implement AI research
  extensions?
Authors: name: Nicholas Edwards, name: Yukyung Lee, name: Yujun, name: Mao, name: Yulu Qin, name: Sebastian Schuster, name: Najoung Kim
Published: 2025-06-27 19:41:41

Agents based on Large Language Models (LLMs) have shown promise for
performing sophisticated software engineering tasks autonomously. In addition,
there has been progress towards developing agents that can perform parts of the
research pipeline in machine learning and the natural sciences. We argue that
research extension and its implementation is a critical capability for such
systems, and introduce RExBench to support the evaluation of this capability.
RExBench is a benchmark consisting of 12 realistic research experiment
implementation tasks that aim to investigate research hypotheses that have not
previously been implemented. Each task is set up as an extension to an existing
research paper and codebase, accompanied by domain expert-written instructions.
RExBench is robust to data contamination, and supports an automatic evaluation
infrastructure that executes agent outputs to determine whether the success
criteria are met. We use this benchmark to evaluate nine LLM agents implemented
using three different frameworks: aider, Claude Code, and OpenHands. We find
that all agents evaluated fail to autonomously implement the majority of the
extensions. Although the success rate improves with additional human-written
hints, the best performance under this setting remains below 40%. This
indicates that current agents are still short of being able to handle realistic
research extension tasks without substantial human guidance.
Tower+: Bridging Generality and Translation Specialization in
  Multilingual LLMs
Authors: name: Ricardo Rei, name: Nuno M. Guerreiro, name: José Pombal, name: João Alves, name: Pedro Teixeirinha, name: Amin Farajian, name: André F. T. Martins
Published: 2025-06-20 15:30:06

Fine-tuning pretrained LLMs has been shown to be an effective strategy for
reaching state-of-the-art performance on specific tasks like machine
translation. However, this process of adaptation often implies sacrificing
general-purpose capabilities, such as conversational reasoning and
instruction-following, hampering the utility of the system in real-world
applications that require a mixture of skills. In this paper, we introduce
Tower+, a suite of models designed to deliver strong performance across both
translation and multilingual general-purpose text capabilities. We achieve a
Pareto frontier between translation specialization and multilingual
general-purpose capabilities by introducing a novel training recipe that builds
on Tower (Alves et al., 2024), comprising continued pretraining, supervised
fine-tuning, preference optimization, and reinforcement learning with
verifiable rewards. At each stage of training, we carefully generate and curate
data to strengthen performance on translation as well as general-purpose tasks
involving code generation, mathematics problem solving, and general
instruction-following. We develop models at multiple scales: 2B, 9B, and 72B.
Our smaller models often outperform larger general-purpose open-weight and
proprietary LLMs (e.g., Llama 3.3 70B, GPT-4o). Our largest model delivers
best-in-class translation performance for high-resource languages and top
results in multilingual Arena Hard evaluations and in IF-MT, a benchmark we
introduce for evaluating both translation and instruction-following. Our
findings highlight that it is possible to rival frontier models in general
capabilities, while optimizing for specific business domains, such as
translation and localization.