Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain
  Perspective
Authors: name: Zhoujun Cheng; name: Shibo Hao; name: Tianyang Liu; name: Fan Zhou; name: Yutao Xie; name: Feng Yao; name: Yuexin Bian; name: Yonghao Zhuang; name: Nilabjo Dey; name: Yuheng Zha; name: Yi Gu; name: Kun Zhou; name: Yuqi Wang; name: Yuan Li; name: Richard Fan; name: Jianshu She; name: Chengqian Gao; name: Abulhair Saparov; name: Haonan Li; name: Taylor W. Killian; name: Mikhail Yurochkin; name: Zhengzhong Liu; name: Eric P. Xing; name: Zhiting Hu
Published: 2025-06-17 20:24:00

Abstract: Reinforcement learning (RL) has emerged as a promising approach to improve
large language model (LLM) reasoning, yet most open efforts focus narrowly on
math and code, limiting our understanding of its broader applicability to
general reasoning. A key challenge lies in the lack of reliable, scalable RL
reward signals across diverse reasoning domains. We introduce Guru, a curated
RL reasoning corpus of 92K verifiable examples spanning six reasoning
domains--Math, Code, Science, Logic, Simulation, and Tabular--each built
through domain-specific reward design, deduplication, and filtering to ensure
reliability and effectiveness for RL training. Based on Guru, we systematically
revisit established findings in RL for LLM reasoning and observe significant
variation across domains. For example, while prior work suggests that RL
primarily elicits existing knowledge from pretrained models, our results reveal
a more nuanced pattern: domains frequently seen during pretraining (Math, Code,
Science) easily benefit from cross-domain RL training, while domains with
limited pretraining exposure (Logic, Simulation, and Tabular) require in-domain
training to achieve meaningful performance gains, suggesting that RL is likely
to facilitate genuine skill acquisition. Finally, we present Guru-7B and
Guru-32B, two models that achieve state-of-the-art performance among open
models RL-trained with publicly available data, outperforming best baselines by
7.9% and 6.7% on our 17-task evaluation suite across six reasoning domains. We
also show that our models effectively improve the Pass@k performance of their
base models, particularly on complex tasks less likely to appear in pretraining
data. We release data, models, training and evaluation code to facilitate
general-purpose reasoning at: https://github.com/LLM360/Reasoning360