Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
Authors: Zhepeng Cen
Haolin Chen
Shiyu Wang
Zuxin Liu
Zhiwei Liu
Ding Zhao
Caiming Xiong
Huan Wang
Weiran Yao
Venue: ICLR 2026 Poster (2026)
Keywords: LLM data pipeline
Reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) have achieved remarkable success through imitation learning on vast text corpora, but this paradigm creates a training-generation gap and limits robust reasoning. Reinforcement learning (RL) offers a more data-efficient solution capable of bridging this gap, yet its application has been constrained by a critical data bottleneck: existing RL datasets are orders of magnitude smaller and less diverse than web-scale pre-training corpora. To address this, we introduce the \textbf{\texttt{Webscale-RL} pipeline}, a scalable data engine that systematically converts large-scale pre-training documents into millions of diverse, verifiable question-answer pairs for RL. Using this pipeline, we construct the \textbf{\texttt{Webscale-RL} dataset}, containing 1.2 million examples across more than 9 domains. Our experiments show that the model trained on this dataset significantly outperforms continual pretraining and strong data refinement baselines across a suite of benchmarks. Notably, RL training with our dataset proves substantially more efficient, achieving the performance of continual pre-training with up to 100$\times$ fewer tokens. Our work presents a viable path toward scaling RL to pre-training levels, enabling more capable and efficient language models.

Published as a conference paper at ICLR 2026
WEBSCALE-RL: AUTOMATED DATA PIPELINE FOR SCALING RL DATA TO PRETRAINING LEVELS
Zhepeng Cen'2, Haolin Chen', Shiyu Wang'. Zuxin Liu', Zhiwei Liu', Ding Zhao', Caiming Xiong', Huan Wang', Weiran Yao!
Salesforce Al Research, Carnegie Mellon University
ABSTRACT
Large Language Models (LLMs) have achieved remarkable success through imi- tation learning on vast text corpora, but this paradigm creates a training -eniinnttnn gap and limits robust reasoning. Reinforcement learning (RL) offers more data- efficient solution capable of bridging this gap, yet its application has been con- strained by critical data bottleneck: existing RL datasets are orders of mag- nitude smaller and less diverse than web-scale pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a scalable data engine that sys- tematically converts large-scale pre-training documents into millions of diverse, verifiable question-answer pairs for RI. Using this pipeline, we construct the Webscale-RL dataset, containing 1.2 million examples across more than 9 do- mains. Our experiments show that the model trained on this dataset significantly outperforms continual pretraining and strong data refinement baselines across a suite benchmarks. Notably, RL training with our dataset proves substantially more efficient, achieving the performance of continual pre-training with up 100x fewer tokens. Our work presents a viable path toward scaling RL to pre- training levels, enabling more capable and efficient language models.
Code
SalesforceAeesaarch/rretttttnni -pipelineppeeeee
Dataset
Sa Salesforce//ebscalees
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable success, primarily through learning on vast text corpora. However, this predominant paradigm, which includes pretraining with next-token prediction and supervised fine-tuning (SFT), is fundamentally in the form of imitation learning. By training models to mimic static offline datasets, imitation learning creates a "teacher-foreing" de-
pendency that makes models vulnerable to distribution shifts (Ross et al.) 2011 Levine et al. 2020 Foster al. 2024) and leads significant gap between training and generation dynamics (Chnn
et al.) [202cc Bachmann & Nagarajan 2024 Cen et al., 2224).. Consequently, models trained in this way struggle with distribution shift and lack the robust reasoning abilities required for complex problem solving.
Reinforcement learning (RL) offers a powerful alternative overcome these challenges (Shao al., 2024 DeepSeek-AI et al. 2025). By learning from online reward feedback on its own generations, static dataset,
an RL-trained model can explore wider solution space and is not confined bridging the training-inference gap. This online learning process makes RI. a significantly more data-efficient training paradigm. As our empirical results demonstrate, RI. can achieve performance gains comparable to continual pretraining with up two orders of magnitude fewer tokens, provid- ing a compelling motivation for scaling RL to unlock new levels of model capability and efficiency.
Despite the clear advantages of RL, its adoption at scale is severely hampered by critical data bottleneck and most existing practice in RL training mainly limited to reasoning tasks such as math and coding in the post-training stage. Pretraining corpora measured in trillions of tokens, whereas existing RL datasets are orders magnitude smaller (c.g.. <10B tokens for RL vs. >TT tokens for pretraining) and lack the diversity of web-scale data. This scarcity is driven by the high

---

Published as a conference paper at ICLR 2026
Webscale-RL Data Pipeline
Generator
Identifier
Checker
Pretraining
RL
TT tokens <108 tokens Olverse domolia
Pretnciiing Divenso sootable
Figure 1: The scaling on LLM RL is fundamentally bottlenecked by the scarcity of high-quality RL data. While pretraining leverages > IT diverse web tokens, RL datasets remain limited to <10B tokens with limited diversity. We propose Webscale-Ri data pipeline to fundamentally improve the scalability of RL data: we convert the pretraining corpora to verifiable query and ground-truth answer pairs, scaling RL data to pretraining levels while preserving the diversity. The experiments show that RL with Websca Re-RL data is significantly more effective and efficient than continual pretraining and data refinement basclines.
cost of generating high-quality, verifiable question-an answering (QA) pairs, which are essential for effective RL-based reasoning tasks. This immense disparity in data scale and diversity prevents RL from realizing its full potential to enhance the general reasoning capabilities of LLMs.
To address these limitations, we introduce Webscale-RL, scalable data pipeline that systemati- cally converts large-scale pretraining corpora into massive, diverse, and verifiable RI.-redyy datasets. Our pipeline is designed to bridge the data gap between pretraining and reinforcement learning, locking the potential to train LLMs with RL at a scale previously unattainable while preserving the vast diversity of the original pretraining data.
Our main contributions are threefold:
We propose Webscale-RI pipeline, an automated and scalable data engine that converts web- scale pretraining documents into verifiable question-answer pairs for RL. The pipeline incorpo- rates stages for data filtering, domain and persona-driven generation, and quality verification to ensure high-quality output.
We construct Webscale-RI dataset, a large-scale and diverse RL dataset containing 1.2 million verifiable QA pairs spanning over nine domains. Our analysis shows it is significantly more diverse than existing large-scale RL datasets.
We provide empirical evidence demonstrating the effectiveness of approach. The model trained with RL on the Webscale-RL. dataset significantly outperforms continual pretraining the source data and strong data refinement baselines across wide range of benchmarks. Fur- thermore, our results show that RI. with our dataset is substantially more data-efficient, achieving comparable performance continual pretraining with 100x fewer tokens.
Our work demonstrates that by converting massive pretraining corpora into format suitable for RL, we can unlock significant performance and efficiency gains. This provides a path toward scaling reinforcement learning match the scale of pretraining, leading to new generation of more capable and robust language models.
RELATED WORKS
Training Data Development and Synthesis. The development L.LMs hinges on the availabil- ity of vast, high-quality training datasets. However, curating such corpora, especially labeled and
across diverse domains, often prohibitively expensive and time-consuming. This challenge has
spurred significant research into efficient synthetic data generation pipelines (Chen et al.) 2024a Ma
et al. 2025 Fan et al.] 2025). ) Current pre- pre-training corpora are typically compiled from a variety
of public sources, such Wikipedia (Foundation), large-scal web crawls (Computer) 2023 Paster
et al.
2023 Penedo al.) 2024at et al. [2024] Raffel et al. 2019), and code repositories (oozhkov
al. 2024 This is often supplemented with content from digitized books (Stroube 2003) and data
2

---

Published as a conference paper at ICLR 2026
generated by other LLMs (Ben Allal et al. 2024 Huang et al. 2024). To endow LLMs with compre- hensive knowledge and robust downstream capabilities, these corpora are constructed on a massive scale, often containing tens of trillions of tokens, exemplified by the 30-trillion-toker RedPa- jama dataset (Computer) 2023) and the 67.5-rrillion-token Stack-v2 (Lozhkov et al. 2024). More recently, the success of models like ReepSeek-R1-Zen (DeepSeekAA) al. (025) and DeepSeek- RI-Zero (DeepSeek-Al et al., 2025) and Grok-4 XAI|(222), which integrate reinforcement learning (RL) at the pre-training stage, built on and is built on and has intensified the demand for similarly large and high-quality synthetic synthetic RL datasets. There have been multiple lines of work in large-scale data synthesis for LLM training: DeepScaler (Luo et al., (025) curated a small (40K)
RL dataset for mathematical reasoning: OpenR1-Mant (Hugging Face 2025) further scales up the mathematical RI dataset for both SFT and RI. via distillation, resulting 220K dataset; Webln- struct (Yue et al. (024)) Open' Thoughts (Guha et al., 2025) and Nature
(Yuan et al. 2025) expand the distillation path to multiple domains and synthesize over 1M data using teacher models for SFT, respectively; Nemotron (Bercovich et al. 2025) extends the dataset size to 3.9M for both SFT and RL.
Reinforcement Learning in LLMs. Recent advancements in large-scale RL have significantly enhanced the capabilities of LLMs, as demonstrated by models such as OpenAl's o-series (OpenAl
2024a| Jaech et al. 2024 OpenAl 2024b) and DeepSeek- -V3R11 (DeepSeek-A) Al et al.. 2024 2025)
Besides these models, many other works show that L. trained to reason with Chain-of- Mffain-of-Thooee Thought
(CoT) prompting have shown substantial performance gains diverse areas, including mathematical
and scientific reasoning (Xie et al. 2025 Cen et al. [2025] Shao et al. 2024 Luong et al. (024//Cenn
al. 2024b Cui et al. 2025), code generation (Le al. 2022 Wei et al. al.]|| 2025). and tool use (Zhang
et al. 2025a Qian et al. 2025). The optimization of the RL objective in these models is primarily
driven by foundational algorithms like Proximal Policy Optimization (PPO) (Schulman et al., 2017)
and its variant, Group Relative Policy Optimization (GRPO) (Shao et al. 2024). Rooted from post-
training RL, many works further extend RL to a significantly larger scale (Liu et al. 2025c XAI
2025) or carlier stage like pre- pre-training (Zelikman al. [2024, Dong et al. 2025 indicating the effectiveness of prolonged, large-scale RL training.
ct al. 2025
METHODOLOGY
In this section, we first provide brief comparison of the pretraining and RL training paradigms and then present our Webscale-RU data pipeline that systematically converts large-scale pretraining data into RL data while preserving the scale and diversity of web data.
3.1 PRELIMINARIES
Pretraining. In the pretraining stage, a large-scale corpus D preeraining (usually >> tokens) is con- structed by filtering and deduplicating publicly available web data sources (Penedo et al., 2024b Weber et al. 22224] Li et al., 2024).. Given this static dataset, the LLM is trained in a teacher-forcing manner to imitate the next-token distribution of the data by minimizing the negative log- likeliohh:ood::
min Ex D Pretraring
log Po(xs x(<())
where x [ .... IT] is token sequence sampled from the pretraining dataset D Pretraining. This imitation-basee objective enforces the model to learn the given pattern from the demonstration data but does not expose the model to the distribution induced by its own generations, suffering from the
distribution shift issue (Bachmann & Nagarajan 2024 Ross et al. 2011) and leading to a training-
inference gap (Bengio et al. 2015 Levine et al. 2020).
Reinforcement Learning (RL). RL instead optimizes the model as a policy that generates answers online and maximizes expected reward on a query q:
max Pq)QQ [R(q.a)) 8
where Q is the query set and R is a task-specific reward function. The online generation and feed- back loop enable the model to narrow the training-inferent gap. In our setup, we adopt binary reward that returns 1 only when the model's final answer matches the ground-truth answer and otherwise. Consequently, each RL training instance is verifiable question-answer pair.
3

---

Published as a conference paper at ICLR 2026
3.2 WEBSCALE- RL DATA PIPELINE
While RL has shown promise in enhancing LLM capabilities (Jaech et al. 2024 DeepSeek-AL et al,, 2025). its effectiveness is constrained by the limited scale and diversity of existing RL datasets. Therefore, the RL training is typically conducted on a much smaller scale on limited domains in the post-training stage. This discrepancy arises from the high costs associated with human annotation and the challenges in generating verifiable QA pairs at large scale. Furthermore, most cxisting RL datasets focus on specific tasks or domains and thus lack the breadth of topics and styles found in web-scale corpora, limiting their generalization to diverse real-world scenarios. To address the limited volume and diversity of existing RL datasets, we propose a Webscale- data pipeline that converts pretraining documents into RL data at scale while preserving the diversity of web data.
Pretraining
Domain-specific
Few-shot Examples
pataset
QA Generation
Filter Informative Convertible
Prompt Template
Verams
Pretraining Datt
Checkers
Pretraining
info leakage
Answer verify
Identifier
Verifiable pair
RL Dataset
Domain
Obroona
Figure 2: Overview of the Webscale-RL, data pipeline that systematically converts large-scale pretraining data into RI. data while preserving the scale and diversity of web data. The pipeline maintains domain-speeiffe demonstration library for few-shot examples for high quality generation and assigns multiple personas to each document to encourage reflecting different viewpoints. The generated QA pairs are verified for correctness and leakage prevention ensure the reliability of the RL dataset. The prompt templates of four stage are listed in Appendix B.I.I
At high level, Webscal le RR. leverages a generative model to convert narrative pretraining doc- uments into verifiable QA pairs for RI training. cover a wider range of topies and question styles, we first maintain domain-specific demonstration library for few-shot examples to guide the generation process. We further assign multiple personas to each document to encourage reflecting different viewpoints. Figure 2 illustrates the pipeline, which consists of four main stages:
Data Filtering. Our pipeline takes pretraining corpora spanning multiple domains as input instead of focusing data in limited domains (Toshniwal et al. 2024 Liu et al.) 2025a). This stage aims to remove imputs that are unlikely to yield verifiable high figh-quaity questions. We first use heuristics to filter out obviously low-quality documents (< 50 tokens) and then employ LLM for further fine-grained filtering (Gunasekar et al. 2023) Wettig et al. 2024). Different from previous pipelines that strictly filter data from multiple dimensions (e.g., difficulty (Fan al. 2025 Ma et al.] 2025), format (Zhou et al. 2025),, with sophisticated reasoning traces (Yuan et al. 2025), 5) etc.), our filter aims select data for the following stages while maximally preserving the diversity of the original materials. Therefore, the LLM-based filter only identifies and removes (i) oon-ifformativ pages where most contents are boilerplate (e.g. navigation, headers, or footers in website html), and (ii) non-self-contaiie fragments that lack sufficient context to verify answers. This two-stage filtering ensures that the retained documents are both informative and convertible into verifiable RI. data.
Domain Classification and Persona Assignment. After filtering. we then classify each document into a specific domain (e.g., commerce, healthcare, social science, etc.) using a LLM-based clas- sifier. Due to extreme high diversity of the pretraining data, pipeline adopts different few-shot examples for each domain to ensure that the generated questions are contextually appropriate and verifiable, which is absent in existing pipelines (Fan et al.) 2025 Yuan al. 2025 The domain tags are then used to collect relevant few-shot exemplars in the subsequent QA generation step. Ad- ditionally, further enhance the diversity of the generated QA pairs, assign multiple personas
4

---

Published as a conference paper at ICLR 2026
who will be interested in the content to each document (Ge et al. 2024), which defines the style and perspective from which questions will be generated. For example, a document classified under the "healthcare" domain might be assigned personas such as "medical expert," "patient," or "health jour- nalist." This persona assignment encourages reflecting different viewpoints and information needs in question generation given the same document, thereby capturing more information in the source data and enriching the RL dataset's 's diversity.
Verifiable QA Generation. Conditioned on the source document, domain tag, and chosen persona, the LLM-based QA generator produces verifiable question-answer pairs. Specifically, we first sam- ple few-shot examples from the domain-specifee demonstration library, curated pool covering a range of question types and complexities within each domain to ensure that the generated questions are of high quality. We then incorporate all contexts with a prompt template (in Appendix 1.1.1] to guide the LLM-based generator to extract diverse question-answer pairs from the perspective of the assigned persona. For question generation, beyond extracting the questions originally contained in the document Yue et al. 2024), our generator can also raise new questions that are answerable according to the pretraining data. Since the trained model is not allowed to access the source doc- ument during RL, we further instruct the generator provide necessary contexts to ensure that the question is self contaaont.i Meanwhile, we only require a relatively short and verifiable ground-trumt answer (e.g.. number, a name, phrase) grounded by the pretraining materials instead of long explanation or detailed reasoning steps composed by a strong LLM (Yue et al., 2024 Yuan et al,, 2025), which significantly reduces the generation complexity and reliance on the backend LLMs. In other words, our generation is to extract the answer from the document instead of distilling from a powerful LLM. This design choice allows us leverage more cost-effective LLMs for genera- tion while still producing high-quality. verifiable QA pairs suitable for RL training. We provide a conversion example in Appendix B.2.1
Quality Check and Leakage Control. While the most generated question-an question-answer answer pairs are of high quality, some may still contain errors or hallucinations. To ensure the reliability of the RI. dataset, we leverage an L.LM-based verifier to implement multi-stage checking process (Liu et al.) 2024] Prabhakar et al. y025): 1) Correctness verification. Unlike accuracy-based post- post-processing in previous works (Zoou et al. 2025 Ma et al. 2025), our verification assesses the correctness of the answers by checking if the extracted QA data are grounded by the source document, which is much less biased by the backend LLMs and effectively reduces the wrong reward signals during RL training; 2) Leakage prevention ensures that the questions do not reveal answers explicitly (e.g., the ground truth is not trivially embedded in the prompt). The verifier filters any QA pairs that fail to meet these criteria, ensuring that the final dataset truly tests the model's knowledge or reasoning capabilities rather than its ability summarize or retrieve information directly from the prompt.
The prompt templates of four stages are listed in Appendix B.III The examples of pretraining to RL conversion are in B.2 We further apply data decontamination by Im-eval-harness (Gao et al. 2024) to remove overlaps with the evaluation. With this pipeline, we can systematically convert large- scale pretraining data into a massive, diverse, and verifiable RL-ready dataset that closely matches the scale and diversity of the original pretraining corpus. This approach effectively addresses the RL data scarcity issue and enables scaling up RL training of LLMS across a wide range of tasks and domains. More discussions are described in Appendix B.
4 WEBSCALE-RL DATASET
4.1 DATASET CONSTRUCTION
We construct Webscale-RT. dataset by running the data pipeline over subset (~IM data in total)
of the mixture of pretraining corpora including DCLM (Li et al. 2024), Wikipedia (Foundation). MegaMath (Zhou et al.) 2025). Stack-v2 (Lozhkov et al. 2024), etc. The choice of pretraining data here aims to cover diverse domains and mimics previous practice on pretraining (Bakouch et al.) 2025). The data selection is flexible and can be adjusted based on the target model and application.
In RL data conversion, we use GPT--4.1-min for domain classification and final quality check, and GPT-4.1 for data filtering and QA generation. As mentioned in QA generation stage in Sec. 3.2 our pipeline aims to extract answer grounded by the pretraining document instead of distilling from strong L.M,, which reduces the bias and reliance on the backend L.MMs. Therefore, the pipeline
5

---

Published as a conference paper at ICLR 2026
can be extended to other open-source source LLMs such as GPT-OSS (Agarwal al. 2025) and Deepseek series (DeepSeek-Al et al.] 2025). For each qualified document, we assign up to personas to generate diverse QA parrs. The final dataset contains ~1.2 QA pairs covering 9+ domains. Note that the dataset can easily be further scaled up to the pretraining level with our Webscale-RI. pipeline. More details of dataset construction described in Appendix BB.2
4.2 DATASET ANALYSIS
We compare our Webscale-Ree RL dataset with other widely used pretraining datasets (RedPajama-
v2 (Weber et al. 2024). Web-EedEEdu (Penedo et al. (004bb. DCLM-baseline (Li et al. 2024)), SFT
datasets
(Yuan et al. 2025). Nemotron (Bercovich al. 2025 which include
reasoning CoT in the answers, and RL datasets DeepScaler (Luo et al. (202)) OpenR1-Mane (Hug-)
ging Face 2025), OpenThoughts: (Guha et al. (025 which include a ground-trui answer for each question. The detailed comparison is listed in Table
Table 1: The comparison of various datasets with our Websca e dataset. The number of data indicates the number of documents (for pretraining datasets) or the number of QA pairs (for SFT and RL datasets). The scalability indicates the potential of scaling up the dataset size: DeepScaler has low scalability because it is collected from competitions and relies on human annotation. Other post-training datasets generate answers by distillation but they collect queries from limited sources, which limits the further scaling. In contrast, both the questions and answers in the Websca le R-RL dataset are converted from and grounded by the pretraining datasets, which can easily scaled up to pretraining level.
Dataset RedPajama-v2
Type
# data
Domain
Data Source
Scalability
Pretrain
> 100B
Multi-domain
Web crawling
Fin-Eue-Edu
Pretrain
>3B
Multi-domain
Web crawling
DCL.M-baseline
Pretrain
>BB
Multi-domain
Web crawling
DeepScaler
RL
40K
Math Math Math, Code, Science
Competition and other math datasets
Low
OpenRI
SFT/RL
220K
Distilled from DeepSeck-RI
Medium
OpenThoughts3
SFT
1.2M
Distilled from QwQ-32B
Medium
NaturalReasoning
SFT
1.IM
Multi-domain
Converted from pretrain distillation
High
Nemotron
SFT/RL
3.9M
Math, Code, Science
Distilled from multiple models
Medium
Webscale-RI.
RT. 1.2M
Multi-domain
Converted from pretrain
High
The comparison shows that the pretraining corpora are orders of magnitude larger and span broad domains, whereas existing SFT/RI. datasets are significantly smaller and often focus on few areas (notably math and code), which limits coverage of general knowledge and open-ended reasoning found in web-scale text. The Nemotron dataset includes data in other domains such as general QA and safety, which however only constitutes a small portion of the dataset. It is also worth noting that while some datasets have relatively large data volume (e.g., OpenThoughts3, Nemotron), they still encounter the challenge of further scaling due to their limited sources of queries. In contrast, our Websca 1 R- dataset constructed by converting from the pretraining documents, allowing for easy expansion to pretraining scale.
We also obverse that large fraction of the SFT/RL data is distilled from other teacher models. This couples dataset quality and ceiling to teacher capability and availability, In contrast, Webscale-RD RL grounded in source documents; the generator does not need to solve the problems during construc- tion; instead, we extract verifiable QA pairs from existing texts, reducing the dependence on strong teachers. Furthermore, because both questions and answers are derived from pretraining documents and verified against the source, Webscal 1 l--RI can scale naturally with the size of available cor- pora (i.e., the pretraining scale) while maintaining diversity, unlike human-labeled or fully distilled datasets whose growth is bottlenecked by annotation query generation.
We list the domain distribution of our dataset in Fig. 3 left. Webscale RL spans 9+ domains in- herited from pretraining sources, substantially more diverse than most public post-trainingg datasets. While we observe that the selMerel domains (Math, Science, Code) constitute a significant por- tion of the dataset, it is also worth noting that the underrepresented domains in existing RL datasets, such as Lifestyle (> 8.6%), Commerce (> 3.3%), etc., are well covered in Webscale- RLL which essential for general-purpose assistants.

---

Published as a conference paper at ICLR 2026
Websrale-RL
Nath
Social Science
Natural Science
Technology
10.00 5%
Other
6.38%
Lifestyle
Healthcare
Education
6.75%
Coding
9.32%
8.05%
Commerce
8.57%
Figure 3: Left: The domain distribution of Webscale-Ri, dataset. Right: question embedding of Webscale-Ree and Nemotron data. We randomly sample 5K questions from each dataset and visualize the embedding (by
The comparison
reduced to 2D using UMAP.
To further illustrate the diversity of Webscale- RL dataset, compare it with Nemotron, large- scale SFT/RL dataset mainly covering math, code, and science. Since we focus on question diver- sity, first randomly sample 5K questions from each dataset and encode them using the Qwen3 Embedding model (Zhang et al. 2025b). We then reduce the embedding dimension to using UMAP (Melnnes al.) (218)) for visualization. The results shown in Fig. right. Although both datasets cover multiple domains, Nemotron data points are mainly clustered in several regions, indicating focus specific topics. In contrast, the Websca cesssclleRR..... data points converted from a larger variety of documents and are generated from diverse perspectives by different personas, resulting in a distribution that more uniform and more scattered, indicating broader coverage of topics and knowledge areas. The diversity along with the large scale of Webscale-HP RL can help models learn wide range of knowledge and reasoning skills, enhancing their versatility and perfor- mance across various tasks.
EXPERIMENTS
In this section, we conduct experiments evaluate the effectiveness of the Webscale--B dataset generated by our proposed pipeline. Our experiments aim to address two main questions: (1) Can RL data generated by pipeline enhance model performance across various benchmarks? (2)
Does RL training scale more effectively and efficiently than standard teacher-forcing training?
5.1 EXPERIMENT SETUP
Baselines. To answer these questions, finetune Qwen2.5-3B model (Yang et al. 2024a) using GRPO (Shao et al. 2024) on the Websca le-RL, dataset and compare it with continual pretraining on the corresponding base dataset, i.e., the original pretraining data prior to RL conversion. We fur- ther compare our method with several advanced data refinement techniques: (1) QuRating(Wettip al. 2024), which selects high-quality data via LLM ranking and filtering; (2) ProX al.)
2024 which uses programmatic cleaning to enhance data quality; and (3) Generative Data Refine- ment (GDR) (Jiang et al. 2025). which originally uses LLM to improve the safety of the corpus (e.g., remove personally identifiable information, toxic content). In our experiment, we use it to improve the quality of the pretraining dataset. For these baselines, we refine the pretraining data using each method and perform continual pretraining the resulting datasets.
Notably, we observe that RL training substantially improves the model's instruction- wowpotiion-ooooiinnggg abil- ities, while the continual pretrained models may fail to start answering in the evaluation, especially for questions with zero-shot examples, potentially introducing an evaluation bias. To mitigate this and enable a fair comparison, construct SFT dataset comprising 10K high-quality examples. Specifically, we first generate QA pairs via our Webscale- RL pipeline and then use GPT-4.1 to distill relatively short reasoning CoT for each question given the ground-truth answer.

---

Published as a conference paper at ICLR 2026
Training. For the continual pretraining and data refining baselines, start from the base model and continue to pretrain on 1M corpus, which represents a superset of the source data for the Webscale-RR dataset. We then follow with SFT training with a smaller learning rate using the 10K high-quality examples. For RL training, we first apply SFT with the same SFT dataset for warm-up. We then sample 150K data points from the Webscale c-RR. dataset and run standard GRPO training. More details of the SFT dataset and training are described in Appendix B.3
Benchmarks. We evaluate the models on a diverse set of benchmarks assess their gen- eral capabilities and domain-speciffic performance, including general tasks (MMLU-pro (Wang al. 2024), Big-Bench (Srivastava al., 2023)), math STEM tasks (GSM8K (Cobbe et al. 2021). MATH500 (Hendrycks et al. [222)) GPQA-diamond (Rein et al., (024h) and coding tasks (MBPP (Austin et al., 2021) and EvalPlus (Liu et al. 2023)). For EvalPius, we report the average score of HumanEval (Chen et al. 2021), MBPP, HumanEval+ and MBPP+. In evaluation, we use the same pipeline and configurations for all models. Specifically, we use zero-shot evaluation for Big-Bench, GPQA-diamond and MATH500, We use 5-shot for MMLU-pro and 8-shot for GSM8K evaluation following the default setting Im-eval-harnese (Gao et al. 2024). More details evalu- ation described in Appendix B.3
5.2 MAIN RESULTS
Sabll[[summarizz the comparisons Webscale-RL. with other baselines. Our method outperforms all baselines across most benchmarks, including continual pretraining and advanced data refinement pipelines. We observe an average improvement of 3.4 over the strongest baseline (GDR), Notably, Webscale-RL RL even narrows the performance gap to the much larger Qwen2.5-77 model from 10.6 pts to 6.1 pts on average. This indicates that converting web-scale corpora into verifiable QA and optimizing with RL yields stronger downstream gains than further imitation on even refined text.
Particularly, the improvements are most pronounced on general knowledge and reasoning tasks (MMLU-pro, Big-Bench, GPQA-diamond), which significantly benefit from the diversity and breadth of the Webscale serve a large jump on MATH500 from 47.6 to 58.0 after RL training with Webscale-RL, which is close to the 7B model. This aligns with prior findings that RL can better incentivize math rea- soning (Shao et al., 2024, Yang et al. 2024b) compared to simply imitating refined documents or QA demonstrations. The gain GSM8K is relatively smaller, likely due to the saturation effect as the base model already achieves strong performance. Meanwhile, the performance improvement coding tasks is relatively smaller, likely reflecting the lower proportion of coding data in the pretraining corpus. Notably, the 3B model finetuned with Webscale-RL substantially narrows the performance gap to the 7B base model on the macro average. suggesting a practical path stronger small models via RL scaling.
dataset inherited from pretraining sources. On math tasks, we ob-
Table 2: Comparison results our Webscale-RL with baselines on various benchmarks. To mitigate evaluation bias, continual pretraining and data refinement baselines are followed by SFT training to enhance instruction following. While all finetuning are based on the Qwen2.5-3B model, we also compare with the 7B base model. Blue bold indicates the best result among 3B baselines; green bold shows where we match or exceed the 7B model.
Method
MMLU-pro
BigBench
GPQA-D
MATH500
GSM8K
MBPP
EvalPlus
Avg
Qwen2.5-3B
37.8
41.2
20.8
47.6
74.2
54.6
57.3
47.6
Qwen2.5-7B
58.7
29.6
60.8
84.4
63.4
62.2
Cont. Pretrain
39.9
45.1
18.6
44.0
77.4
55.2
57.8
48.3
QuRating
39.7
44.9
19.4
44.6
76.8
54.8
57.6
48.3
ProX
40.0
46.0
19.5
44.4
77.3
54.2
57.5
48.4
GDR
39.9
46.0
20.8
44.4
77.4
55.0
57.6
48.7
Webscale-RL.
48.3
23.2
58.0
78.5
55.0
57.8
52.1
Despite using a small SFT set to reduce evaluation bias toward instruction-followiie RL still main- tains clear advantages over SFF-augmented continual pretraining baselines. This suggests that the
8

---

Published as a conference paper at ICLR 2026
gains from Webscale-RL. RL are not solely due to improved instruction adherence but stem from the reward-driven online learning signal. Overall, these results demonstrate that our Webscale-RL data pipeline effectively scales up RL data by converting from pretraining corpus and enables sig- nificant capability improvements across diverse tasks and domains.
5.3 PERFORMANCE COMPARISON OF SCALING TRAINING
While RL shows remarkable advantages over teacher- reacher-foreing training in terms of final performance, further investigate the scaling efficiency of RL training compared standard pretraining with respect to the amount of training tokens. To this end, we compare the performance of RL. training and continual pretraining the Webscale -RL dataset and the original pretraining corpus, respectively. Notably, we observe that the length of QA pairs in the Webscales original pretraining corpus while their source data are the same. Therefore, for a fair comparison on token efficiency of the original data, we compute the token number of RL training by the origi- nal pretraining corpus used to generate the Websca l ee-RI, RL, dataset instead of the Webscal 1 e-RL dataset itself. For example, if we generate two 300-token QA pairs from a 4000-token pretraining text, then we count the RL training token number 4000 instead of 600 when training these two QA pairs. Note that for continual pretraining with different training data volume, we also apply the same SFT training as follow-up using the 10K high-quality examples to mitigate the evaluation bias.
different training scales by varying the amount of data sampled from
dataset differs from the length of document the
40-
Fratrainng Owant2:3 buae
Cort. Netrokiing
Gward 5 base
Cward 39 base
Avarage Nerfommanca on benchmares
Performence MILL PTO
Perforr Performanca on 9-Sanch
07-
14.5%
4/-
41-
12'
10'
Training Tokens
Taining Tokens
Training Tokens
Figure 4: Scaling comparison between Webscale-RL training and continual pretraining with the original pretraining corpora. We report the performances on MMLU-pro (left), Big-Bench (middle) and average all benchmarks (right). The token number for RL training is calculated based on the original pretraining corpus used to generate the Webscale-22 dataset. The each data point in continual pretraining baselines followed by SFT training using the same 10K high-quality examples. The RL training on Websca le e-RL consistently outperforms continual pretraining at different training scales and exhibits better scaling efficiency.
Since the pretraining corpus mainly consists of general web text, we focus on evaluating the models on general tasks (MML.U-pro and Big- (eg-Bench to better reflect the impact of training scale. We also report the average performance across all benchmarks to provide holistic view.
Figure iillustrates the performance comparison between RL training with Mebscale-RL dataset and continual pretraining with pretraining corpora at different training scales. We observe that RI. training consistently outperforms continual pretraining across all three metrics (MML.U-pro, Big- Bench, and average performance) at various training scales. With the same amount of training tokens (100 millions,, RL training achieves 4.4% improvement over Qwen2.5-3B base model in average while continual pretraining exhibits similar performance to the base model.
Notably, RL training achieves comparable or better performance with significantly fewer training tokens. For instance, MMLU-pro, RL training with approximately 10M tokens attains similar performance to continual pretraining with IB tokens, indicating over 100x improvement in data efficiency. Furthermore, RI. training exhibits steeper upward trend the training scale increases,

---

Published as a conference paper at ICLR 2026
which is also true for other benchmarks, demonstrating that RL training not only leads to higher final performance, but scales more effectively and efficiently than standard seucher-forcing approaches.
CONCLUSION
In this paper, we introduced the Webscale-RI. pipeline, an end-to-end data engine that converts web-scale pretraining corpora into verifiable, RL-reedy data while preserving diversity. With this pipeline, we constructed the Webscale-RL. dataset, which is orders of magnitude larger and more diverse than existing RL datasets. Empirically, training LLM with RL on Webscale-RL improves performance across a diverse suite of benchmarks and delivers better data efficiency than continual pretraining comparable token budgets, especially on general knowledge and open-ended reasoning (MMLU-pro, Bench),, with consistent improvements in math and STEM areas.
While our results demonstrate the promise of scaling RL data to pretraining levels, several limita- tions and future directions remain. The current Webscale-RL RI. dataset lacks coverage of high-quality data in certain domains such as coding, which leads to smaller gains on coding benchmarks. There- fore, one future direction is to rebalance the domain distribution of the pretraining sources according to the target applications (e.g., to integrate repository-seee bility). Meanwhile, the current RL training employs generative reward model that provides binary feedback based on match with the ground truth. While this reward exhibits high performance and stability for RL training, it introduces substantial extra inference cost, becoming one bottleneck for scaling up. Future work can explore more efficient reward models to further scale up RI. training to larger models and datasets.
code data to enhance the coding capa-
REPRODUCIBILITY STATEMENT
Our Webscale-RL. data pipeline is built upon publicly available datasets and publicly available LLMs for generation and verification. The detailed data sources, prompts, and implementation details are described in Appendix B.1 and Appendix B.2 For the continual pretraining and RL finetuning, we use the standard pretraining and RL algorithms (GRPO), and the hyperparameters are detailed in Appendix B.3
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai. Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-1208 1il-oss-100b- & gpt-oss-20b model card. arXiv preprint arXiv:2508. 10925, 2025.
Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereassoinng: Advancing data distil- lation for competitive coding. arXiv preprint arXiv:2504.01943. 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 100.07200..772.. 2021.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. arXiv preprint arXiv:2403.06963, 2024.
Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel PatÎño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek. Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, Xuan-Son Nguyen, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmoILM3: smol, multilingual, long-context reasoner. https:
//hugg fg ace. ca co/bloo ol smol 1m3 2025.
Loubna Ben Allal, Anton Lozhkov. Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, 2024. URL cosmopedial
co co/datasets http:////..tttteeaaaaaaaaaaaeeea BuggingFaceTB:
10

---

Published as a conference paper at ICLR 2026
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Akhiad Bereovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El- Vaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemottonn: Efficient reasoning models. arXiv preprint 1rXiv:2505.00943 2025.
Zhepeng Cen, Yao Liu, Siliang Zeng, Pratik Chaudhari, Huzefa Rangwala, George Karypis, and Rasool Fakoor. Bridging the training-inference gap in Ilms by leveraging self-generated tokens. arXiv preprint arXiv:2410. 1004410..46555 14655, 2024.
Zhepeng Cen, Yihang Yao, William Han, Zuxin Liu, and Ding Zhao. Behavior injection: Preparing
language models for reinforcement learning. arXiv preprint arXiv:2505.18917. 2025.
Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I Abdin. On the diversity of synthetic data and its impact on training large language models. arXiv preprint arXiv:2410. 15226, 2024a.
Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arXiv:2411.04282, 2024b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024c.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10.16821 2021.
Together Computer. Redpajama: open dataset for training large language models, 2023. URL htt.://bw...coooooogeeeooooooere
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455, 2025.
DeepSeck-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang. Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan. Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan. T. Wang, Tao Yun. Tian Pei, Tianyu Sun. W. L. Xiao, and Wangding Zeng. Deepseek- v3 technical report. CoRR, abs/2412. 2024. doi: 1.48550/ARXIV.2412.1999 ntttps://dos
URL
wtw.g//..... 48550
2412. 19437
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu,

---

Published as a conference paper at ICLR 2026
Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongje Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501. 2025. doi: 10. 12948
48550/ 25941.24.....
Qingxiu Dong. Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, and Furu Wei. Rein- forcement pre-training. CoRR, abs/2506.08000 2025. doi: 1048550ARXXV:2506.0000 URL
https: ora.i..orgg/0.... wtt..o/gg.......//....
08007
Run-Ze Fan, Zengzhi Wang, and Pengfei Liu. Megascience: Pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812 2025.
Dylan J Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon in imitation learning. Advances in Neural Information Processing Systems, 37:120602- 120666, 2024.
Wikimedia Foundation. Wikimedia downloads. URL https::////mggeee www.mmpawwwwiimmiia..mmmmm uit.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. (ttp:tts:: wwtp///endddoooooooooooo
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXhv:2406.20094 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymiss Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reason- ing models. arXiv preprint 00t::22206044788, 2025.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprini arXiv:2306. 11644, 2023.
Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighte- val: A lightweight framework for Ilm evaluation, 2023. URL htttps://iithut com/
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0387. 2021.
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang. J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi. Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models. 2024. URL
Hugging Face, Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: t/t.thub.co./////o wtt.hu...oo//uuggiiiii.. wgw.huggiiggggggoooooo open--i
12

---

Published as a conference paper at ICLR 2026
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, An- drey Mishchenko, Andy Applehaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lu- garesi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang. Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler. Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Ed- mund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' O'Connell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. Openai ol system card. CoRR, abs/2412.16720, 2024. doi: 1048550/ARXI.2212..6777 URL sttpps://oi... org/10 1111:/00.855555 48550/ wttp:. 2412 16720
Minqi Jiang, JoAGo GM AraAsjo, Will Ellsworth, Sian Gooding, and Edward Grefenstette. Gener- ative data refinement: Just ask for better data. arXiv preprint arXiv:2509. 00000,,,,33,.... 2025.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314 213288, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-Im: search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 47004420044282 2024.
Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Evander Yang, Kun Li, Haoyuan Wu, Jiajia Wu, Zihao Zheng, Chenchen Zhang, Kun Shi, Kyrierl Deng, Qi Yi, Ruibin Xiong. Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang Xu, Zheng Fang, Shuai Li, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bo Chao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Wayyt Wang. Bo Zhou, and Di Wang. Reinforcement learning pre-training data, 2025. URL utLhttps: /ttps:://aaxivii http/aixii....//// org/ abs 2et...//.......9.
Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew C Yao. Augmenting math word problems via iterative question composing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24605-24613, 2025a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In seirth- Conference on Neural Information Processing Systems, 2023. URL httpp: forum 7id=1 qvx 0Cu7
openreview.eo
Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, et al. Scaling up rl: Unlocking di- verse reasoning in ilms via prolonged training, arXiv preprint arXiv:2507.12507 2025b.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong- Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint wrxiv:2505.24880 2025c.
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse dunction-callingg datasets. Advances in Neural Information Processing Systems, 37:54463-544 -44482, 2024.
13

---

Published as a conference paper at ICLR 2026
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poiriert Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tanyangg Liu, Max Tian, De- nis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang. Qian Liu, Dmitry Abulkhanov, In- draneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui. Tri Dao. Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Cha- pados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Star- coder 2 and the stack v2: The next generation, 2024.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol ol-preveew with a 1.5b model by scaling rl, 2025. Notion Blog.
Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuning, arXiv preprint arXiv:2401.085: 00996,, 3, 2024.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing IIm reasoning across all domains. arXiv preprint arXiv:2505 2025.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: /802.0346, 2018.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art math- ematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891, 2025.
OpenAL Learning to reason with LLMs. Learning-ton to- reason wwth 11ms/| 2024a. [Online].
OpenAl. 03 and 04 mini system 2024b.
card.
https://openai..com/nnee http:://ognaal.commneee
o3 04 wysteni-sssttetrrananae 03-04- 18-o4-mini-syae 1a-o--mmiin-ssstt----aaa--
URL
[Online].
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. dataset of high-quality mathematical web text, 2023.
Openwebmath: An open
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The ehiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. URL https:/ http://///peeeeeeeeeeeeeeeeeeeee wet.:/iwwwweweeeteeeeeeeeeeeeeee www.nnttt//ooiooooooooo In6SCkn2QaG
Guilherme Penedo, Hynek Kydliček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale, Advances in Neural Information Processing Systems, 37:30811 1--0089, 2024b.
Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al. Apigen-mt: Agentic pipeline for multi- turn data generation via simulated agent-human interplay. arXiv preprint arXiv:250.00360,, 2025.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tüis Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
14

---

Published as a conference paper at ICLR 2026
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proot q&a bench- mark. In First Conference on Language Modeling, 2024.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707.06347, 2017. URL http::/arxiv..//
1707. 06347
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rIhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. Transactions machine learning research, 2023.
Bryan Stroube. Literary freedom: Project gutenberg, XRDS: Crossroads, The ACM Magazine for Students, 10(1):3-3, 2003.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-22: Accelerating ai for math with massive open-source instruction data. arXiv preprint 24X.002410.001660 2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266 9522900099990..11 2024.
Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, al. Redpajama: open dataset for training large language models. Advances in neural information processing systems, 37:1 16462- 116492, 2024.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Ilm reasoning via rein- forcement learning on open software evolution. arXiv preprint arXiv:2502. / 18449, 2025.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint axX1v22402.09739, 2024.
XAI. Grok 4. http://///..aa////////rrooo/ 2025. [Online].
Tian Xic, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Jocy Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 15X 15 15, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5 5-math technical report: Toward mathematical expert model via self-improvemennt arXiv preprint arXiv:2409. /2/22 2024b.
15

---

Published as a conference paper at ICLR 2026
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho, Dong Wang, Yuandong Tian, Jason Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint arXiv:250..13124 2025.
Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 37:90629-90660, 2024.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves think before speaking. arXiv preprint arXiv:2403.09629, 2024.
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemotron-ressancchtoooo--t Tool-using language models with rein- forced reasoning. arXiv preprint wwXiv:2505.00024 2025a.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05/70 2025b.
Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu. Programming every example: Lifting pre-training data quality like experts at scale. arXiv preprint 1rX1v:2409.17115. 2024.
Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He. Zhengzhong Liu, and Eric P Xing. Megamath: Pushing the limits of open math corpora. arXiv preprint 02880 2025.
16

---

Published as a conference paper at ICLR 2026
A USAGE OF LLMS
In paper writing, the LLMs are mainly used for proofreading and polishing the language, including grammar, spelling, and clarity. The main content, ideas, experiments and following presentations (e.g., results and visualizations) are done by the authors. The LLMs assist to draft the results analysis and conclusion sections based on the experimental results provided by the authors. The authors care- fully checked the content and made necessary modifications to ensure the accuracy and correctness of the statements.
B DETAILS OF DATASET CONSTRUCTION AND TRAINING
B. WEBSCALE-RL DATA PIPELINE DETAILS
Our data pipeline employs GPT-4.1 --mini for domain classification and quality checking, while uti- lizing GPT-4. for QA generation to ensure higher quality outputs. In second stage, we assign up different personas to each document and generate tailored QA pairs for each persona respec- tively.
B.1.1 PROMPT TEMPLATES
Our pipeline consists of four main stages, each with carefully designed prompts ensure high- quality data generation:
Stage 1: Data Filtering
Role:
Data Analyst
Objective: Identify whether the material meets quality criteria for QA generation
Prompt:
You are a helpful data analyst. You will be given a material which can come from very diverse sources and may not be identify whether the material qualified for the following criteria:
In this stage, your task is to
The material is informative and self-contained for the user
It's possible to extract question and corresponding answer from the material
The content has sufficient depth and clarity
Based above instructions, identify whether the material is qualified or not.
(Raw Document)
Stage 2: Domain Classification & Persona Assignment
Role:
Data Analyst
Objective:
Classify domain and identify target personas
Prompt:
are helpful data analyst. You will given a material which can come from very diverse sources and may not be well-structured. In this stage, your task is to identify the domain and persona of the material.
Here are the instructions for the domain and persona:
The domain is the main topic the material. You should choose from the following domains: {All Domains)
The persona the intended audience the material. the material in- tended for multiple personas, you should list several personas that will be interested in the material
Based the above instructions, identify the domain and persona of the material. (Raw Document)
17

---

Published as a conference paper at ICLR 2026
Stage 3: QA Generation
Role:
Domain Expert (Porsona-speciie
Objective:
Generate high-quality question-answer pairs from material
Prompt:
You will be given material which can come from very diverse sources and may not be well well-structuured In this stage, your task is generate question and answer pair from the material.
Here the instructions for the question and answer generation:
You will act as given persona. You should generate question and answer pair from your perspective
Both the question and answer should be totally from the material. Do not
generate information that is not in the material
You should generate such a question that its corresponding answer is rela- rively short and can be easily and clearly verified
Ensure the question is natural and reflects genuine curiosity from the target persona
[Few-shot Examples}
Based above instructions and examples, generate a question and answer pair from the material.
RRa Document
[Persona]
Stage
Quality Check
Role: Data Labeler
Objective: Verify QA pair correctness and detect information Icakage
Prompt:
You data labeler. You will be given material and question and pair generated from material. Your task is check whether the question and answer pair is correct according to the material and whether there info leakage from question to answer.
Here are the instructions for checking:
For the answer correctness, you should check whether the answer is correct according to the original material
The information leakage indicates that the question explicitly provides infor- mation about the answer and then the answer can directly obtained from the question
Ensure the question requires gemuine understanding of the source material
Few- Examples)
Based on the above instructions, check the QA pair extructed from the original material in terms of the correctness and info leakage.
[Raw Document)
(QA Pair)
B.2 WEBSCALE-RL DATASET COMPOSITION
We curate dataset from diverse pretraining сorpora to ensure comprehensive domain coverage while emphasizing reasoning capabilities. The selected sources include DCLM (Li et al. 2024),
Wikipedia (Foundation), MegaMath (Zhou et al., 2025). Stack-v2 (Lozhkov al. 2024),, with ad-
ditional data from OpenMathheasonnng (Moshkov et al. 2025) and OpenCodeReassningg (Ahmad
al. 2025) following SmolL.M3 (Bakouch al. 2025
protocols.
18

---

Published as a conference paper at ICLR 2026
Table 3: Source distribution of the Webscale-RL. dataset (~1.2MM total QA pairs)
Source Dataset
# of Converted QA Pairs
DCLM
~500
Wikipedia
~350K
MegaMath
100K
OpenMatthReasoning
~00K
Stack-v2
~50K
OpenCodeReeasoning
~50K
B.2.1 DATA CONVERSION EXAMPLE
The following example demonstrates our persona-driven conversion process:
Original Wikipedia Document: Alterna Bank
CS Alterna Bank O, operating as Alterna Bank O, is a Canadian direct bank and a wholly owned subsidiary of the Ontario-based credit union Alterna Savings. The bank offers chequing and high- interest savings accounts and morigages.
Operating primarily
direct bank since 2017, most customers access accounts using the bank's website, telephone service, mobile apps. Unlike most other direct banks, some accounts can also accessed through branches. There are two Alterna Bank locations Gatineau, QC, and Alterna Savings branches also administer deposits and loans on its behalf...
The bank originated the Civil Service Loan Corporation, founded 29 October 1992 and operating hecame CS Altema Bank after receiving letters patent of continuation 2 October 2000 as federally regulated institution under the Bank Act...
CS Loan Corporation.
Alterna Bank is a member of Canada Deposit Insurance Corporation (CDIC)...
Converted QA Pair: Financial Analyst Persona
Question: In examining the regulatory protection for depositors, is Alterna Bank member of the Canada Deposit Insurance Corporation (CDIC)? Answer: Yes, Altema Bank member Canada Deposit Insurance Corporation (CDIC).
Converted QA Pair: Commerce Student Persona
Question: Canadian direct banking, what notable about the way Alterna Bank allows its customers to access their accounts compared to most other direct banks? Answer: Some Alterna Bank accounts can he accessed through branches, unlike most other direct banks.
B.3
TRAINING IMPLEMENTATION DETAILS
B.3.1 BASELINE IMPLEMENTATION
Generative Refinement Baseline: Following (Jiang et al. 2025). we adapt their safety-focused
approach to quality improvement. GPT-4.1 processes each document by: (1) assessing content quality similar our filtering stage, returning original text if adequate; (2) refining documents by removing non-in an-iifoormativ sections or discarding low-quality content entirely.
Our 10K SFT dataset enhances cntrraction-olllwwinn capabilities post- continual pretraining and provides RL training warmup. We sample 10K queries from a held-out Webscale-RL RL subset with no training overlap. Since original answers concise, GPT-4.1 generates detailed Chain-of Chain-of-Thought explanations based on ground truth, reducing hallucination compared to full model distillation.
SFT Dataset Construction:
19

---

Published as a conference paper at ICLR 2026
B.3.2
TRAINING HYPERPARAMETERS
Table 4: Training configuration and hyperparameters
Training Stage
Hyperparameter
Value
Batch Size
256
Continual Pretraining
Learning Rate
x 10-6
Max Input Length
4096
Batch Size
128
Supervised Fine-tuning
Learning Rate
5 10-6
Max Input Length
4096
Batch Size
256
Learning Rate
10-6
Reinforcement Learning
Samples per Query
16
Max Rollout Length Algorithm
2560
GRPO
Shao al, 2024
All experiments use AdamW optimizer with VeRL (Sheng et al. 2025 as the training backend. RL training employs binary rewards, where L.M judges whether generated answers match ground truth responses.
B.3.3
EVALUATION FRAMEWORK
Table 5: Evaluation benchmarks and configurations
Benchmark
Framework
Shots
Domain Focus
MMLU-Pro
LM-Eval
5
Multi-domain Knowledge
BigBench
LM-Eval
0
Reasoning & Language
GPQA-D
LightEval
0
Scientific Reasoning
MATH500
LightEval
Mathematical Problem Solving
GSM8K
LM-Eval
8
Grade School Math
MBPP
EvalPlus
Python Programming
EvalPlus
EvalPlus
0
Code Generation & Testing
We employ LM-eval-harness (Gao et al. 2024). LightEval (Habib et al., (023). and EvalPlus Liu al. 2023) with default settings for prompt templates, metries, and decoding parameters. MMLU- Pro and GSM8K use few-shot evaluation (5 and 8 shots respectively) following standard protocols, while other benchmarks use zero-shot evaluation.
20


Reviews:
[{"review_id": "96VEO2TnEJ", "reviewer": "Reviewer_TSDn", "summary": "This paper studies the scaling of reinforcement learning dataset. They propose to utilize LLM to convert pretraining dataset into verifiable QA pairs. The empirical evaluation shows that this is much more efficient than naive continual pretraining, providing a potential way to scaling rl and the possibility to change the pretraining/rl paradigm.", "questions": "1. Could the authors expand the descriptions of Sec 3.2 to specify more details of each pipeline stage?\n2. Do the authors have any ablation to show the diversity of the generated rl data and how it correlates with the final performance (Are there any redundancy in the generation thus the performance seems to be saturating or are there  room for further improvements).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper demonstrates the viability of converting pretraining data into reinforcement data. This is a rather novel research direction compared to ones that simply do filtering or rephrasing of pretraining data.", "weaknesses": "1. There seems to be a limitation of diversity for generating verifiable answers. The correctness also might depend on the powerfulness of the LLM used for conversion.\n2. The description of the proposed method seems to be a bit overly succinct. Even though some parts of the pipelines might share similarities with the paper cited. For completeness, it is better to expand the description in Sec 3.2 and clearly discuss the difference with the prior works. The section for verifiable generation seems to be especially important for further discussions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofm7pHA9fU", "reviewer": "Reviewer_giqw", "summary": "This study presents an automated pipeline for scaling reinforcement-learning (RL) data generation to pretraining-level diversity and scale, and it  identifies a fundamental bottleneck: RL training data (<10 B tokens) are vastly smaller and narrower than web-scale pretraining corpora (>1 T tokens). To bridge this gap, the authors propose a scalable RL-data engine that converts pretraining documents into verifiable question–answer pairs; and provide a webscale-RL dataset, containing ≈ 1.2 M verified QA pairs across 9 + domains (math, code, science, lifestyle, commerce, etc.), derived directly from pretraining corpora and easily extendable to trillions of tokens;  Empirical evidence that RL training on Webscale-RL yields large gains over continual pretraining and data-refinement baselines.", "questions": "- How robust are results if GPT-4.1 is replaced by a smaller verifier? Does verification bias the dataset toward specific model families?\n\n\n- Have you tested pipeline scalability beyond 1.2 M pairs, e.g., to 100 M, regarding quality degradation, cost, or RL performance saturation?\n\n\n- Since persona assignment drives diversity, do personas from one domain (e.g., healthcare) transfer to unseen domains?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novel scalable data pipeline: first systematic approach to transform general pretraining corpora into verifiable RL data; elegant combination of domain and persona-based generation with automated quality control.\n\n- Massive coverage & diversity: dataset spans 9 + domains, addressing the narrow-domain limitation of prior RL datasets (mostly math / code).\n\n- Practical contribution: reusable, reproducible pipeline that could standardize RL-data construction for open-source models.", "weaknesses": "- Limited transparency of generation details: prompt templates, filtering thresholds, and judge-LLM configurations are summarized only in appendices; reproducibility may depend on access to proprietary models (GPT-4.1).\n\n\n- Evaluation bias: The SFT “warm-up” and differing token budgets between RL and continual pretraining make it difficult to ensure the strict fairness of comparison.\n\n\n- Domain imbalance: coding and certain STEM domains remain underrepresented, reflected in smaller gains on code benchmarks.\n\n\n- The cost and environmental impact of multi-stage LLM generation and verification are not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v9xDRvcfmf", "reviewer": "Reviewer_QNn9", "summary": "The paper introduces Webscale-RL pipeline, a scalable data engine that systematically converts large-scale pre-training documents into millions of diverse, verifiable question-answer pairs for RL. Using this pipeline, the paper constructs the Webscale-RL dataset, containing 1.2 million examples across more than 9 domains.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The construction pipeline of this paper is solid and the proposed method is sound.\n2. The manuscript is also well-written and easy to follow.", "weaknesses": "1. The motivation and methodology of this paper are quite similar to NaturalReasoning (Yuan et al., 2025), which significantly limits the novelty of this work.\n\n2. The entire data generation and annotation pipeline heavily relies on GPT-4.1 and GPT-4.1-mini. This raises concerns about potential biases introduced by these models, and whether the associated costs are acceptable, especially when compared to the scale and cost-efficiency of large-scale pretraining corpora.\n\n3. The experiments and analysis in this paper are quite limited — almost no in-depth analysis is provided. At the very least, a comparative experiment with NaturalReasoning should be included. Moreover, using RL as a post-training method and comparing it against continued pretraining may not be a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rF8fTJWmQY", "reviewer": "Reviewer_cyfs", "summary": "The paper introduces Webscale-RL, an automated and scalable data pipeline designed to bridge the data bottleneck in reinforcement learning (RL) for large language models (LLMs). The core idea is to systematically convert large-scale pretraining corpora into verifiable question–answer (QA) pairs, enabling RL training at web scale. Using this pipeline, the authors construct the Webscale-RL dataset, consisting of 1.2 million QA pairs across nine or more domains. Empirical results demonstrate that models trained with RL on this dataset outperform continual pretraining and advanced data refinement baselines across a wide range of benchmarks (MMLU-Pro, BigBench, MATH500, etc.), achieving comparable or superior performance with up to 100× fewer tokens. Overall, the work aims to make RL training as scalable as pretraining, thus improving both capability and efficiency of LLMs.", "questions": "none", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper addresses one of the central challenges in applying RL to LLMs — the lack of large, diverse, and verifiable datasets. The proposed Webscale-RL pipeline is a substantial step toward scaling RL data generation to the same order of magnitude as pretraining corpora, a problem of clear importance for the community.\n\n2. The data engine is well structured, with clearly defined stages — data filtering, domain classification with persona assignment, QA generation, and multi-stage verification. This modularity enhances reproducibility and potential adaptability to new domains.\n\n3. The proposed framework could substantially reduce human labeling costs and enable scalable RL training for future LLM development. The paper provides sufficient implementation detail for reproducibility.\n\n4. Experiments are extensive and convincing. RL training with Webscale-RL yields consistent gains across general reasoning, math, and STEM benchmarks. The results on data efficiency (100× fewer tokens to reach equivalent performance) are particularly impressive and suggest genuine benefits beyond data refinement or imitation-based methods.", "weaknesses": "1. Although multi-domain, the dataset underrepresents certain crucial areas (e.g., coding, tool use). This imbalance is reflected in weaker improvements on programming benchmarks (e.g., MBPP), limiting generalizability across task types.\n\n2. The pipeline relies heavily on GPT-4.1 models for data filtering, classification, generation, and verification. This dependency may restrict reproducibility for groups without access to such models and raises cost and sustainability concerns.\n\n3. Experiments are performed mainly on a 3B-parameter model (Qwen2.5-3B). While the improvements are substantial, it remains unclear how well the approach scales to larger or smaller models, or to different RL algorithms. Real-world downstream evaluations (dialogue, reasoning under uncertainty, tool use) are also absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)