Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers
Authors: name: Xingyue Huang; name: Rishabh; name: Gregor Franke; name: Ziyi Yang; name: Jiamu Bai; name: Weijie Bai; name: Jinhe Bi; name: Zifeng Ding; name: Yiqun Duan; name: Chengyu Fan; name: Wendong Fan; name: Xin Gao; name: Ruohao Guo; name: Yuan He; name: Zhuangzhuang He; name: Xianglong Hu; name: Neil Johnson; name: Bowen Li; name: Fangru Lin; name: Siyu Lin; name: Tong Liu; name: Yunpu Ma; name: Hao Shen; name: Hao Sun; name: Beibei Wang; name: Fangyijie Wang; name: Hao Wang; name: Haoran Wang; name: Yang Wang; name: Yifeng Wang; name: Zhaowei Wang; name: Ziyang Wang; name: Yifan Wu; name: Zikai Xiao; name: Chengxing Xie; name: Fan Yang; name: Junxiao Yang; name: Qianshuo Ye; name: Ziyu Ye; name: Guangtao Zeng; name: Yuwen Ebony Zhang; name: Zeyu Zhang; name: Zihao Zhu; name: Bernard Ghanem; name: Philip Torr; name: Guohao Li
Published: 2025-09-03 06:42:40

Abstract: Recent advances in Large Language Models (LLMs) have shown that their
reasoning capabilities can be significantly improved through Reinforcement
Learning with Verifiable Reward (RLVR), particularly in domains like
mathematics and programming, where ground-truth correctness can be
automatically evaluated. However, extending this success to other
reasoning-intensive domains remains challenging due to the scarcity of
high-quality, verifiable datasets and the high cost of human supervision. In
this work, we introduce the Loong Project: an open-source framework for
scalable synthetic data generation and verification across a diverse range of
reasoning-intensive domains. The framework consists of two key components: (1)
LoongBench, a curated seed dataset containing 8,729 human-vetted examples
across 12 domains (e.g., Advanced Mathematics, Chemistry, Logic), each paired
with executable code and rich metadata; and (2) LoongEnv, a modular synthetic
data generation environment that supports multiple prompting strategies to
produce new question-answer-code triples. Together, these components form an
agent-environment loop that enables reinforcement learning, where an LLM-based
agent is rewarded for generating Chain-of-Thought (CoT) solutions that align
with code-executed answers. Empirically, we benchmark LoongBench on a broad
suite of both open-source and proprietary LLMs to evaluate domain coverage and
reveal performance bottlenecks. In addition, we conduct a comprehensive
analysis of synthetic data generated by LoongEnv, examining correctness,
difficulty, and diversity. Code and documentation are available at
https://github.com/camel-ai/loong.