RLVE: Scaling Up Reinforcement Learning for Language Models with
  Adaptive Verifiable Environments
Authors: name: Zhiyuan Zeng, name: Hamish Ivison, name: Yiping Wang, name: Lifan Yuan, name: Shuyue Stella Li, name: Zhuorui Ye, name: Siting Li, name: Jacqueline He, name: Runlong Zhou, name: Tong Chen, name: Chenyang Zhao, name: Yulia Tsvetkov, name: Simon Shaolei Du, name: Natasha Jaques, name: Hao Peng, name: Pang Wei Koh, name: Hannaneh Hajishirzi

We introduce Reinforcement Learning (RL) with Adaptive Verifiable
Environments (RLVE), an approach using verifiable environments that
procedurally generate problems and provide algorithmically verifiable rewards,
to scale up RL for language models (LMs). RLVE enables each verifiable
environment to dynamically adapt its problem difficulty distribution to the
policy model's capabilities as training progresses. In contrast, static data
distributions often lead to vanishing learning signals when problems are either
too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a
large-scale suite of 400 verifiable environments carefully developed through
manual environment engineering. Using RLVE-Gym, we show that environment
scaling, i.e., expanding the collection of training environments, consistently
improves generalizable reasoning capabilities. RLVE with joint training across
all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement
across six reasoning benchmarks, starting from one of the strongest 1.5B
reasoning LMs. By comparison, continuing this LM's original RL training yields
only a 0.49% average absolute gain despite using over 3x more compute. We
release our code publicly.