Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning
  for LLMs
Authors: name: Ring Team, name: Bin Hu, name: Cai Chen, name: Deng Zhao, name: Ding Liu, name: Dingnan Jin, name: Feng Zhu, name: Hao Dai, name: Hongzhi Luan, name: Jia Guo, name: Jiaming Liu, name: Jiewei Wu, name: Jun Mei, name: Jun Zhou, name: Junbo Zhao, name: Junwu Xiong, name: Kaihong Zhang, name: Kuan Xu, name: Lei Liang, name: Liang Jiang, name: Liangcheng Fu, name: Longfei Zheng, name: Qiang Gao, name: Qing Cui, name: Quan Wan, name: Shaomian Zheng, name: Shuaicheng Li, name: Tongkai Yang, name: Wang Ren, name: Xiaodong Yan, name: Xiaopei Wan, name: Xiaoyun Feng, name: Xin Zhao, name: Xinxing Yang, name: Xinyu Kong, name: Xuemin Yang, name: Yang Li, name: Yingting Wu, name: Yongkang Liu, name: Zhankai Xu, name: Zhenduo Zhang, name: Zhenglei Zhou, name: Zhenyu Huang, name: Zhiqiang Zhang, name: Zihao Wang, name: Zujie Wen

We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model
optimized via reinforcement learning (RL) to achieve efficient and robust
reasoning capabilities. Built upon the publicly available Ling-lite model, a
16.8 billion parameter model with 2.75 billion activated parameters, our
approach matches the performance of state-of-the-art (SOTA) small-scale
reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench,
GPQA-Diamond) while activating only one-third of the parameters required by
comparable models. To accomplish this, we introduce a joint training pipeline
integrating distillation with RL, revealing undocumented challenges in MoE RL
training. First, we identify optimization instability during RL training, and
we propose Constrained Contextual Computation Policy Optimization(C3PO), a
novel approach that enhances training stability and improves computational
throughput via algorithm-system co-design methodology. Second, we empirically
demonstrate that selecting distillation checkpoints based on entropy loss for
RL training, rather than validation metrics, yields superior
performance-efficiency trade-offs in subsequent RL training. Finally, we
develop a two-stage training paradigm to harmonize multi-domain data
integration, addressing domain conflicts that arise in training with mixed
dataset. We will release the model, dataset, and code.