Orionnnn/hub-stats huggingface dataset:
Cartridges: Lightweight and general-purpose long context representations
  via self-study
Authors: name: Sabri Eyuboglu, name: Ryan Ehrlich, name: Simran Arora, name: Neel Guha, name: Dylan Zinsley, name: Emily Liu, name: Will Tennien, name: Atri Rudra, name: James Zou, name: Azalia Mirhoseini, name: Christopher Re
Published: 2025-06-06 17:48:23

Large language models are often used to answer queries grounded in large text
corpora (e.g. codebases, legal documents, or chat histories) by placing the
entire corpus in the context window and leveraging in-context learning (ICL).
Although current models support contexts of 100K-1M tokens, this setup is
costly to serve because the memory consumption of the KV cache scales with
input length. We explore an alternative: training a smaller KV cache offline on
each corpus. At inference time, we load this trained KV cache, which we call a
Cartridge, and decode a response. Critically, the cost of training a Cartridge
can be amortized across all the queries referencing the same corpus. However,
we find that the naive approach of training the Cartridge with next-token
prediction on the corpus is not competitive with ICL. Instead, we propose
self-study, a training recipe in which we generate synthetic conversations
about the corpus and train the Cartridge with a context-distillation objective.
We find that Cartridges trained with self-study replicate the functionality of
ICL, while being significantly cheaper to serve. On challenging long-context
benchmarks, Cartridges trained with self-study match ICL performance while
using 38.6x less memory and enabling 26.4x higher throughput. Self-study also
extends the model's effective context length (e.g. from 128k to 484k tokens on
MTOB) and surprisingly, leads to Cartridges that can be composed at inference
time without retraining.
Dreamland: Controllable World Creation with Simulator and Generative
  Models
Authors: name: Sicheng Mo, name: Ziyang Leng, name: Leon Liu, name: Weizhen Wang, name: Honglin He, name: Bolei Zhou
Published: 2025-06-09 17:59:52

Large-scale video generative models can synthesize diverse and realistic
visual content for dynamic world creation, but they often lack element-wise
controllability, hindering their use in editing scenes and training embodied AI
agents. We propose Dreamland, a hybrid world generation framework combining the
granular control of a physics-based simulator and the photorealistic content
output of large-scale pretrained generative models. In particular, we design a
layered world abstraction that encodes both pixel-level and object-level
semantics and geometry as an intermediate representation to bridge the
simulator and the generative model. This approach enhances controllability,
minimizes adaptation cost through early alignment with real-world
distributions, and supports off-the-shelf use of existing and future pretrained
generative models. We further construct a D3Sim dataset to facilitate the
training and evaluation of hybrid generation pipelines. Experiments demonstrate
that Dreamland outperforms existing baselines with 50.8% improved image
quality, 17.9% stronger controllability, and has great potential to enhance
embodied agent training. Code and data will be made available.
Saffron-1: Towards an Inference Scaling Paradigm for LLM Safety
  Assurance
Authors: name: Ruizhong Qiu, name: Gaotang Li, name: Tianxin Wei, name: Jingrui He, name: Hanghang Tong
Published: 2025-06-06 18:05:45

Existing safety assurance research has primarily focused on training-phase
alignment to instill safe behaviors into LLMs. However, recent studies have
exposed these methods' susceptibility to diverse jailbreak attacks.
Concurrently, inference scaling has significantly advanced LLM reasoning
capabilities but remains unexplored in the context of safety assurance.
Addressing this gap, our work pioneers inference scaling for robust and
effective LLM safety against emerging threats. We reveal that conventional
inference scaling techniques, despite their success in reasoning tasks, perform
poorly in safety contexts, even falling short of basic approaches like
Best-of-N Sampling. We attribute this inefficiency to a newly identified
challenge, the exploration--efficiency dilemma, arising from the high
computational overhead associated with frequent process reward model (PRM)
evaluations. To overcome this dilemma, we propose SAFFRON, a novel inference
scaling paradigm tailored explicitly for safety assurance. Central to our
approach is the introduction of a multifurcation reward model (MRM) that
significantly reduces the required number of reward model evaluations. To
operationalize this paradigm, we further propose: (i) a partial supervision
training objective for MRM, (ii) a conservative exploration constraint to
prevent out-of-distribution explorations, and (iii) a Trie-based key--value
caching strategy that facilitates cache sharing across sequences during tree
search. Extensive experiments validate the effectiveness of our method.
Additionally, we publicly release our trained multifurcation reward model
(Saffron-1) and the accompanying token-level safety reward dataset (Safety4M)
to accelerate future research in LLM safety. Our code, model, and data are
publicly available at https://github.com/q-rz/saffron , and our project
homepage is at https://q-rz.github.io/p/saffron .
Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers
Authors: name: Zhengyao Lv, name: Tianlin Pan, name: Chenyang Si, name: Zhaoxi Chen, name: Wangmeng Zuo, name: Ziwei Liu, name: Kwan-Yee K. Wong
Published: 2025-06-09 17:54:04

Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress
in text-driven visual generation. However, even state-of-the-art MM-DiT models
like FLUX struggle with achieving precise alignment between text prompts and
generated content. We identify two key issues in the attention mechanism of
MM-DiT, namely 1) the suppression of cross-modal attention due to token
imbalance between visual and textual modalities and 2) the lack of
timestep-aware attention weighting, which hinder the alignment. To address
these issues, we propose Temperature-Adjusted Cross-modal Attention
(TACA), a parameter-efficient method that dynamically rebalances multimodal
interactions through temperature scaling and timestep-dependent adjustment.
When combined with LoRA fine-tuning, TACA significantly enhances text-image
alignment on the T2I-CompBench benchmark with minimal computational overhead.
We tested TACA on state-of-the-art models like FLUX and SD3.5, demonstrating
its ability to improve image-text alignment in terms of object appearance,
attribute binding, and spatial relationships. Our findings highlight the
importance of balancing cross-modal attention in improving semantic fidelity in
text-to-image diffusion models. Our codes are publicly available at
https://github.com/Vchitect/TACA
CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large
  Language Models
Authors: name: Guang Liu, name: Liangdong Wang, name: Jijie Li, name: Yang Yu, name: Yao Xu, name: Jiabei Chen, name: Yu Bai, name: Feng Liao, name: Yonghua Lin
Published: 2025-06-09 06:14:19

We introduce CCI4.0, a large-scale bilingual pre-training dataset engineered
for superior data quality and diverse human-like reasoning trajectory. CCI4.0
occupies roughly 35 TB of disk space and comprises two sub-datasets:
CCI4.0-M2-Base and CCI4.0-M2-CoT. CCI4.0-M2-Base combines a 5.2 TB carefully
curated Chinese web corpus, a 22.5 TB English subset from Nemotron-CC, and
diverse sources from math, wiki, arxiv, and code. Although these data are
mostly sourced from well-processed datasets, the quality standards of various
domains are dynamic and require extensive expert experience and labor to
process. So, we propose a novel pipeline justifying data quality mainly based
on models through two-stage deduplication, multiclassifier quality scoring, and
domain-aware fluency filtering. We extract 4.5 billion pieces of
CoT(Chain-of-Thought) templates, named CCI4.0-M2-CoT. Differing from the
distillation of CoT from larger models, our proposed staged CoT extraction
exemplifies diverse reasoning patterns and significantly decreases the
possibility of hallucination. Empirical evaluations demonstrate that LLMs
pre-trained in CCI4.0 benefit from cleaner, more reliable training signals,
yielding consistent improvements in downstream tasks, especially in math and
code reflection tasks. Our results underscore the critical role of rigorous
data curation and human thinking templates in advancing LLM performance,
shedding some light on automatically processing pretraining corpora.
SpatialLM: Training Large Language Models for Structured Indoor Modeling
Authors: name: Yongsen Mao, name: Junhao Zhong, name: Chuan Fang, name: Jia Zheng, name: Rui Tang, name: Hao Zhu, name: Ping Tan, name: Zihan Zhou
Published: 2025-06-09 07:10:58

SpatialLM is a large language model designed to process 3D point cloud data
and generate structured 3D scene understanding outputs. These outputs include
architectural elements like walls, doors, windows, and oriented object boxes
with their semantic categories. Unlike previous methods which exploit
task-specific network designs, our model adheres to the standard multimodal LLM
architecture and is fine-tuned directly from open-source LLMs.
  To train SpatialLM, we collect a large-scale, high-quality synthetic dataset
consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with
ground-truth 3D annotations, and conduct a careful study on various modeling
and training decisions. On public benchmarks, our model gives state-of-the-art
performance in layout estimation and competitive results in 3D object
detection. With that, we show a feasible path for enhancing the spatial
understanding capabilities of modern LLMs for applications in augmented
reality, embodied robotics, and more.
Pre-trained Large Language Models Learn Hidden Markov Models In-context
Authors: name: Yijia Dai, name: Zhaolin Gao, name: Yahya Satter, name: Sarah Dean, name: Jennifer J. Sun
Published: 2025-06-08 21:49:38

Hidden Markov Models (HMMs) are foundational tools for modeling sequential
data with latent Markovian structure, yet fitting them to real-world data
remains computationally challenging. In this work, we show that pre-trained
large language models (LLMs) can effectively model data generated by HMMs via
in-context learning (ICL)x2013their ability to infer patterns from
examples within a prompt. On a diverse set of synthetic HMMs, LLMs achieve
predictive accuracy approaching the theoretical optimum. We uncover novel
scaling trends influenced by HMM properties, and offer theoretical conjectures
for these empirical observations. We also provide practical guidelines for
scientists on using ICL as a diagnostic tool for complex data. On real-world
animal decision-making tasks, ICL achieves competitive performance with models
designed by human experts. To our knowledge, this is the first demonstration
that ICL can learn and predict HMM-generated sequencesx2013an
advance that deepens our understanding of in-context learning in LLMs and
establishes its potential as a powerful tool for uncovering hidden structure in
complex scientific data.
The Illusion of Thinking: Understanding the Strengths and Limitations of
  Reasoning Models via the Lens of Problem Complexity
Authors: name: Parshin Shojaee, name: Iman Mirzadeh, name: Keivan Alizadeh, name: Maxwell Horton, name: Samy Bengio, name: Mehrdad Farajtabar
Published: 2025-06-07 22:42:29

Recent generations of language models have introduced Large Reasoning Models
(LRMs) that generate detailed thinking processes before providing answers.
While these models demonstrate improved performance on reasoning benchmarks,
their fundamental capabilities, scaling properties, and limitations remain
insufficiently understood. Current evaluations primarily focus on established
math and coding benchmarks, emphasizing final answer accuracy. However, this
evaluation paradigm often suffers from contamination and does not provide
insights into the reasoning traces. In this work, we systematically investigate
these gaps with the help of controllable puzzle environments that allow precise
manipulation of complexity while maintaining consistent logical structures.
This setup enables the analysis of not only final answers but also the internal
reasoning traces, offering insights into how LRMs think. Through extensive
experiments, we show that LRMs face a complete accuracy collapse beyond certain
complexities. Moreover, they exhibit a counterintuitive scaling limit: their
reasoning effort increases with problem complexity up to a point, then declines
despite having remaining token budget. By comparing LRMs with their standard
LLM counterparts under same inference compute, we identify three performance
regimes: (1) low-complexity tasks where standard models outperform LRMs, (2)
medium-complexity tasks where LRMs demonstrates advantage, and (3)
high-complexity tasks where both models face complete collapse. We found that
LRMs have limitations in exact computation: they fail to use explicit
algorithms and reason inconsistently across scales. We also investigate the
reasoning traces in more depth, studying the patterns of explored solutions and
analyzing the models' computational behavior, shedding light on their
strengths, limitations, and raising questions about their reasoning
capabilities.
Well Begun is Half Done: Low-resource Preference Alignment by
  Weak-to-Strong Decoding
Authors: name: Feifan Song, name: Shaohang Wei, name: Wen Luo, name: Yuxuan Fan, name: Tianyu Liu, name: Guoyin Wang, name: Houfeng Wang
Published: 2025-06-09 05:21:22

Large Language Models (LLMs) require alignment with human preferences to
avoid generating offensive, false, or meaningless content. Recently,
low-resource methods for LLM alignment have been popular, while still facing
challenges in obtaining both high-quality and aligned content. Motivated by the
observation that the difficulty of generating aligned responses is concentrated
at the beginning of decoding, we propose a novel framework, Weak-to-Strong
Decoding (WSD), to enhance the alignment ability of base models by the guidance
of a small aligned model. The small model first drafts well-aligned beginnings,
followed by the large base model to continue the rest, controlled by a
well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign,
to fine-tune a small-sized Pilot-3B as the draft model, which effectively
enhances different base models under the WSD framework to outperform all
baseline methods, while avoiding degradation on downstream tasks, termed as the
alignment tax. Extensive experiments are further conducted to examine the
impact of different settings and time efficiency, as well as analyses on the
intrinsic mechanisms of WSD in depth.
OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation
Authors: name: Jingjing Chang, name: Yixiao Fang, name: Peng Xing, name: Shuhan Wu, name: Wei Cheng, name: Rui Wang, name: Xianfang Zeng, name: Gang Yu, name: Hai-Bao Chen
Published: 2025-06-09 17:50:21

Text-to-image (T2I) models have garnered significant attention for generating
high-quality images aligned with text prompts. However, rapid T2I model
advancements reveal limitations in early benchmarks, lacking comprehensive
evaluations, for example, the evaluation on reasoning, text rendering and
style. Notably, recent state-of-the-art models, with their rich knowledge
modeling capabilities, show promising results on the image generation problems
requiring strong reasoning ability, yet existing evaluation systems have not
adequately addressed this frontier. To systematically address these gaps, we
introduce OneIG-Bench, a meticulously designed comprehensive benchmark
framework for fine-grained evaluation of T2I models across multiple dimensions,
including prompt-image alignment, text rendering precision, reasoning-generated
content, stylization, and diversity. By structuring the evaluation, this
benchmark enables in-depth analysis of model performance, helping researchers
and practitioners pinpoint strengths and bottlenecks in the full pipeline of
image generation. Specifically, OneIG-Bench enables flexible evaluation by
allowing users to focus on a particular evaluation subset. Instead of
generating images for the entire set of prompts, users can generate images only
for the prompts associated with the selected dimension and complete the
corresponding evaluation accordingly. Our codebase and dataset are now publicly
available to facilitate reproducible evaluation studies and cross-model
comparisons within the T2I research community.
Robust Preference Optimization via Dynamic Target Margins
Authors: name: Jie Sun, name: Junkang Wu, name: Jiancan Wu, name: Zhibo Zhu, name: Xingyu Lu, name: Jun Zhou, name: Lintao Ma, name: Xiang Wang
Published: 2025-06-04 08:19:37

The alignment of Large Language Models (LLMs) is crucial for ensuring their
safety and reliability in practical applications. Direct Preference
Optimization (DPO) has emerged as an efficient method that directly optimizes
models using preference pairs, significantly reducing resource demands.
However, the effectiveness of DPO heavily depends on the data quality, which is
frequently compromised by noise. In this work, we propose gamma-PO, a
dynamic target margin preference optimization algorithm that adjust reward
margins at the pairwise level. By introducing instance-specific margin
calibration, gamma-PO strategically prioritizes high-confidence pairs (those
demonstrating higher reward margins) while suppressing potential noise from
ambiguous pairs. Moreover, gamma-PO is a plug-and-play method, compatible
with variants of DPO that rely on reward margin between preference pairs.
Across benchmarks such as AlpacaEval2 and Arena-Hard, gamma-PO achieves an
average 4.4\% improvement over other baselines, setting new benchmarks for
state-of-the-art performance. Additionally, gamma-PO requires minimal code
changes and has a negligible impact on training efficiency, making it a robust
solution for enhancing LLMs alignment. Our codes are available at
https://github.com/sunjie279/gammaPO{https://github.com/sunjie279/gammaPO}.
GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular
  Structure Recognition
Authors: name: Jingchao Wang, name: Haote Yang, name: Jiang Wu, name: Yifan He, name: Xingjian Wei, name: Yinfan Wang, name: Chengjin Liu, name: Lingli Ge, name: Lijun Wu, name: Bin Wang, name: Dahua Lin, name: Conghui He
Published: 2025-06-09 08:47:10

Optical Chemical Structure Recognition (OCSR) is crucial for digitizing
chemical knowledge by converting molecular images into machine-readable
formats. While recent vision-language models (VLMs) have shown potential in
this task, their image-captioning approach often struggles with complex
molecular structures and inconsistent annotations. To overcome these
challenges, we introduce GTR-Mol-VLM, a novel framework featuring two key
innovations: (1) the Graph Traversal as Visual Chain of Thought
mechanism that emulates human reasoning by incrementally parsing molecular
graphs through sequential atom-bond predictions, and (2) the data-centric
principle of Faithfully Recognize What You've Seen, which addresses
the mismatch between abbreviated structures in images and their expanded
annotations. To support model development, we constructed GTR-CoT-1.3M, a
large-scale instruction-tuning dataset with meticulously corrected annotations,
and introduced MolRec-Bench, the first benchmark designed for a fine-grained
evaluation of graph-parsing accuracy in OCSR. Comprehensive experiments
demonstrate that GTR-Mol-VLM achieves superior results compared to specialist
models, chemistry-domain VLMs, and commercial general-purpose VLMs. Notably, in
scenarios involving molecular images with functional group abbreviations,
GTR-Mol-VLM outperforms the second-best baseline by approximately 14 percentage
points, both in SMILES-based and graph-based metrics. We hope that this work
will drive OCSR technology to more effectively meet real-world needs, thereby
advancing the fields of cheminformatics and AI for Science. We will release
GTR-CoT at https://github.com/opendatalab/GTR-CoT.
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical
  Understanding and Reasoning
Authors: name: LASA Team, name: Weiwen Xu, name: Hou Pong Chan, name: Long Li, name: Mahani Aljunied, name: Ruifeng Yuan, name: Jianyu Wang, name: Chenghao Xiao, name: Guizhen Chen, name: Chaoqun Liu, name: Zhaodonghui Li, name: Yu Sun, name: Junao Shen, name: Chaojun Wang, name: Jie Tan, name: Deli Zhao, name: Tingyang Xu, name: Hao Zhang, name: Yu Rong
Published: 2025-06-08 08:47:30

Multimodal Large Language Models (MLLMs) have demonstrated impressive
capabilities in understanding common visual elements, largely due to their
large-scale datasets and advanced training strategies. However, their
effectiveness in medical applications remains limited due to the inherent
discrepancies between data and tasks in medical scenarios and those in the
general domain. Concretely, existing medical MLLMs face the following critical
limitations: (1) limited coverage of medical knowledge beyond imaging, (2)
heightened susceptibility to hallucinations due to suboptimal data curation
processes, (3) lack of reasoning capabilities tailored for complex medical
scenarios. To address these challenges, we first propose a comprehensive data
curation procedure that (1) efficiently acquires rich medical knowledge data
not only from medical imaging but also from extensive medical texts and
general-domain data; and (2) synthesizes accurate medical captions, visual
question answering (VQA), and reasoning samples. As a result, we build a
multimodal dataset enriched with extensive medical knowledge. Building on the
curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu
undergoes multi-stage training to embed medical expertise and enhance its
task-solving capabilities progressively. Besides, we preliminarily explore the
potential of applying reinforcement learning with verifiable rewards paradigm
to enhance Lingshu's medical reasoning ability. Additionally, we develop
MedEvalKit, a unified evaluation framework that consolidates leading multimodal
and textual medical benchmarks for standardized, fair, and efficient model
assessment. We evaluate the performance of Lingshu on three fundamental medical
tasks, multimodal QA, text-based QA, and medical report generation. The results
show that Lingshu consistently outperforms the existing open-source multimodal
models on most tasks ...
Reinforcement Pre-Training
Authors: name: Qingxiu Dong, name: Li Dong, name: Yao Tang, name: Tianzhu Ye, name: Yutao Sun, name: Zhifang Sui, name: Furu Wei
Published: 2025-06-09 17:59:53

In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling
paradigm for large language models and reinforcement learning (RL).
Specifically, we reframe next-token prediction as a reasoning task trained
using RL, where it receives verifiable rewards for correctly predicting the
next token for a given context. RPT offers a scalable method to leverage vast
amounts of text data for general-purpose RL, rather than relying on
domain-specific annotated answers. By incentivizing the capability of
next-token reasoning, RPT significantly improves the language modeling accuracy
of predicting the next tokens. Moreover, RPT provides a strong pre-trained
foundation for further reinforcement fine-tuning. The scaling curves show that
increased training compute consistently improves the next-token prediction
accuracy. The results position RPT as an effective and promising scaling
paradigm to advance language model pre-training.