Every Attention Matters: An Efficient Hybrid Architecture for
  Long-Context Reasoning
Authors: name: Ling Team; name: Bin Han; name: Caizhi Tang; name: Chen Liang; name: Donghao Zhang; name: Fan Yuan; name: Feng Zhu; name: Jie Gao; name: Jingyu Hu; name: Longfei Li; name: Meng Li; name: Mingyang Zhang; name: Peijie Jiang; name: Peng Jiao; name: Qian Zhao; name: Qingyuan Yang; name: Wenbo Shen; name: Xinxing Yang; name: Yalin Zhang; name: Yankun Ren; name: Yao Zhao; name: Yibo Cao; name: Yixuan Sun; name: Yue Zhang; name: Yuchen Fang; name: Zibin Lin; name: Zixuan Cheng; name: Jun Zhou
Published: 2025-10-22 07:59:38

Abstract: In this technical report, we present the Ring-linear model series,
specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0.
Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while
Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both
models adopt a hybrid architecture that effectively integrates linear attention
and softmax attention, significantly reducing I/O and computational overhead in
long-context inference scenarios. Compared to a 32 billion parameter dense
model, this series reduces inference cost to 1/10, and compared to the original
Ring series, the cost is also reduced by over 50%. Furthermore, through
systematic exploration of the ratio between different attention mechanisms in
the hybrid architecture, we have identified the currently optimal model
structure. Additionally, by leveraging our self-developed high-performance FP8
operator library-linghe, overall training efficiency has been improved by 50%.
Benefiting from the high alignment between the training and inference engine
operators, the models can undergo long-term, stable, and highly efficient
optimization during the reinforcement learning phase, consistently maintaining
SOTA performance across multiple challenging complex reasoning benchmarks.