FastCuRL: Curriculum Reinforcement Learning with Progressive Context
  Extension for Efficient Training R1-like Reasoning Models
Authors: name: Mingyang Song, name: Mao Zheng, name: Zheng Li, name: Wenjie Yang, name: Xuan Luo, name: Yue Pan, name: Feng Zhang

In this paper, we propose \textsc{FastCuRL}, a simple yet efficient
Curriculum Reinforcement Learning approach with
context window extending strategy to accelerate the reinforcement learning
training efficiency for R1-like reasoning models while enhancing their
performance in tackling complex reasoning tasks with long chain-of-thought
rationales, particularly with a 1.5B parameter language model.
\textsc{FastCuRL} consists of two main procedures: length-aware
training data segmentation and context window extension training. Specifically,
the former first splits the original training data into three different levels
by the input prompt length, and then the latter leverages segmented training
datasets with a progressively increasing context window length to train the
reasoning model. Experimental results demonstrate that
\textsc{FastCuRL}-1.5B-Preview surpasses DeepScaleR-1.5B-Preview
across all five datasets (including MATH 500, AIME 2024, AMC 2023, Minerva
Math, and OlympiadBench) while only utilizing 50\% of training steps.
Furthermore, all training stages for FastCuRL-1.5B-Preview are completed using
just a single node with 8 GPUs.