Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning
  for LLMs
Authors: name: Ring Team; name: Bin Hu; name: Cai Chen; name: Deng Zhao; name: Ding Liu; name: Dingnan Jin; name: Feng Zhu; name: Hao Dai; name: Hongzhi Luan; name: Jia Guo; name: Jiaming Liu; name: Jiewei Wu; name: Jun Mei; name: Jun Zhou; name: Junbo Zhao; name: Junwu Xiong; name: Kaihong Zhang; name: Kuan Xu; name: Lei Liang; name: Liang Jiang; name: Liangcheng Fu; name: Longfei Zheng; name: Qiang Gao; name: Qing Cui; name: Quan Wan; name: Shaomian Zheng; name: Shuaicheng Li; name: Tongkai Yang; name: Wang Ren; name: Xiaodong Yan; name: Xiaopei Wan; name: Xiaoyun Feng; name: Xin Zhao; name: Xinxing Yang; name: Xinyu Kong; name: Xuemin Yang; name: Yang Li; name: Yingting Wu; name: Yongkang Liu; name: Zhankai Xu; name: Zhenduo Zhang; name: Zhenglei Zhou; name: Zhenyu Huang; name: Zhiqiang Zhang; name: Zihao Wang; name: Zujie Wen
Published: 2025-06-17 17:12:34

Abstract: We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model
optimized via reinforcement learning (RL) to achieve efficient and robust
reasoning capabilities. Built upon the publicly available Ling-lite model, a
16.8 billion parameter model with 2.75 billion activated parameters, our
approach matches the performance of state-of-the-art (SOTA) small-scale
reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench,
GPQA-Diamond) while activating only one-third of the parameters required by
comparable models. To accomplish this, we introduce a joint training pipeline
integrating distillation with RL, revealing undocumented challenges in MoE RL
training. First, we identify optimization instability during RL training, and
we propose Constrained Contextual Computation Policy Optimization(C3PO), a
novel approach that enhances training stability and improves computational
throughput via algorithm-system co-design methodology. Second, we empirically
demonstrate that selecting distillation checkpoints based on entropy loss for
RL training, rather than validation metrics, yields superior
performance-efficiency trade-offs in subsequent RL training. Finally, we
develop a two-stage training paradigm to harmonize multi-domain data
integration, addressing domain conflicts that arise in training with mixed
dataset. We will release the model, dataset, and code.