Synthetic Data Generation and Multi-Step Reinforcement Learning for Reasoning and Tool Use
Authors: Anna Goldie
Azalia Mirhoseini
Hao Zhou
Irene Cai
Christopher D Manning
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Reinforcement Learning
Multi-Step Reasoning
Tool Use
Synthetic Data
TL;DR: We propose a synthetic data generation and RL methodology for multi-step reasoning and tool use.

Abstract: Reinforcement learning has been shown to improve the performance of large language models. However, traditional approaches like RLHF or RLAIF treat the problem as single-step. As focus is shifting towards solving more complex reasoning and agentic tasks, language models must take multiple steps of text generation, reasoning and environment interaction before generating a solution. We propose a synthetic data generation and RL methodology targeting multi-step optimization scenarios. This approach, called Step-Wise Reinforcement Learning (SWiRL), iteratively generates multi-step reasoning and tool use data, and then learns from that data. It employs a simple step-wise decomposition that breaks each multi-step trajectory into multiple sub-trajectories corresponding to each action by the original model. It then applies synthetic data filtering and RL optimization on these sub-trajectories. We evaluated SWiRL on a number of multi-step tool use, question answering, and mathematical reasoning tasks. Our experiments show that SWiRL outperforms baseline approaches by 21.5\%, 12.3\%, 14.8\%, 11.1\%, and 15.3\% in relative accuracy on GSM8k, HotPotQA, CofCA, MuSiQue, and BeerQA, respectively. Excitingly, the approach exhibits generalization across tasks: for example, training only on HotPotQA (text question-answering) improves zero-shot performance on GSM8k (a math dataset) by 16.9\%.

Published as a conference paper at COLM 2025
Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use
Anna Goldie Azalia Mirhoseini
Hao Zhou Irene Cai Google DeepMind Mountain View, CA, USA {haozhouml, /eeeaaaaegeeeigggggl.cooomm
Department of Computer Science
Stanford University {agoldie, azalia]êcs wtt.sstttaaordoooo edu
Christopher D. Manning
Department of Computer Science Stanford University @ccs @cs. stanford. edu
[manning
Abstract
Reinforcement learning has been shown to improve the performance of large language models. However, traditional approaches like RLHF or RLAIF treat the problem as single-step. As focus shifts toward more com- plex reasoning and agentic tasks, language models must take multiple steps of text generation, reasoning and environment interaction before generating solution. We propose synthetic data generation and RL methodology targeting multi-step optimization scenarios. This approach, called Step-Wise Reinforcement Learning (SWiRL), iteratively generates multi-step reasoning and tool use data, and then learns from that data. It employs a simple step-wise decomposition that breaks each multi-step trajectory into multiple sub-trajectories corresponding to each action by the original model. It then applies synthetic data filtering and RI. opti- mization on these sub-trajectories. We evaluated SWiRL on a number of multi-step tool use, question answering, and mathematical reasoning tasks. Our experiments show that SWiRL outperforms baseline approaches by 21.5%, 12.3%, 14.8%, 11.1%, and 15.3% in relative accuracy on GSM8K, HotPotQA, CofCA, MuSiQue, and BeerQA, respectively. Excitingly, the approach exhibits generalization across tasks: for example, training only on HotPotQA (text question-answa -nswerrngg) improves zero-shot performance on GSM8K (a math dataset) by a relative 16.9%.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities Natural Language Processing (Gemini Team al., 2024; Anthropic, 2024; OpenAl al., 2024). However, they often struggle to answer complex queries that require reasoning and tool use across multiple steps (Wu al., 2024), such as multi-hop question-answering, mathematical problem-solvingg coding, and other agentic tasks, (Yang et al., 2018; Trivedi et al., 2022; Wu et al., 2024; Cobbe et al., 2021; Jimenez et al., 2024; Ehrlich et al., 2025; Li al., 2022).
Traditional reinforcement learning (RL) approaches, such as RL From Human Feedback (RLHF) (Christiano et al., 2023), RL. from AT Feedback (RLAIF) (Bai et al., 2022), and RL from Execution Feedback (RLEF) (Gehring et al., 2025), have focused on single-step optimization, leaving the challenge of multi-step tasks largely unaddressed. Many real-world problems require a sequence of interrelated actions; for example, when answering challenging ques- tion, a model must determine not just what information to seek, but when to stop searching and synthesize its findings. Multi-step reasoning creates compounding challenge, as incorrect intermediate steps often lead to incorrect final results, making it critical to maintain accuracy across the entire chain of actions or learn to effectively recover from such errors.

---

Published as a conference paper at COLM 2025
To address this challenge, we present Step-Wiss Reinforcement Learning (SWiRL), an offline multi-step optimization technique. We consider a setting where the model has access to tool, such as search engine or calculator, and can run sequence of tool use calls as needed to answer the question. Our goal is to teach the model how to decompose complex problems into a sequence of more manageable subtasks, when to call the tool, how to formulate call to the tool, when to use the results of these queries to answer the question, and how to effectively synthesize its findings. In particular, we propose two stage approach, in which we first generate multi-step synthetic data and then learn from these data using step-wise reinforcement learning method. This approach has the key practical advantage that we can quickly generate large volumes of multi-step training data via parallel calls to avoid throttling the training process with slow tool use execution. In addition, this offline process enables greater reproducibility due to having fixed dataset.
To generate multi-step synthetic training data, we provide an open-source LLM (Gemma (Gemma Team et al., 2024b)) with access to relevant tool (e.g,, search engine or calculator). We iteratively prompt the model to generate multi-step trajectories; at each step, the model is free to generate chain of thought, and may either call tool or produce a final answer, which we refer to as the model's action. If the model generates a tool use call, its query is automatically extracted from the overall response and executed in the environment, and the result is presented to the model in the next step. The trajectory ends when the model generates an answer to the original question, which it indicates using special markers. We convert each trajectory with k actions into k subtrajectories, containing the context from the beginning of the trajectory up to that action. We then use step-wise reinforcement learning approach to optimize over this dataset, employing a generative reward model that evaluates each action in the context of its subtrajectory.
This granular approach enables us to apply direct feedback after each step of the trajectory, and to do so in a manner that is contextually aware. Unlike prior RI. finetuning approaches used in frontier open-source models like DeepSeek-R1 (DeepSeek-AI and others, 2025) and Llama-3 (Grattafiori et al., 2024), we do not solely optimize for final performance, and use no golden labels; however, by optimizing for the reasonableness of each step given prior steps, SWiRL does in fact improve final performance.
In addition to evaluating SWiRL on challenging multi-hop question-answerinne and mathe- matical problem-solving tasks, we also study the generalization properties of this methodol- ogy. This is of key interest because there is an explosion of agentic applications for language models, and methods that generalize across datasets and tasks will be easier, cheaper and faster to adapt to new environments. We also measure the effectiveness of different syn- thetic data filtering strategies, study SWiRL's ability to generalize across datasets and tasks, measure the impact of model size and dataset size, and explore the mechanism driving these performance improvements.
Our contributions are as follows:
We propose Step-Wise Reinforcement Learning (SWiRL), an approach to synthetic data generation and offline RL that advances multi-step reasoning and tool use.
We demonstrate generalization across datasets. For example, training SWiRL on HotPotQA not only improves performance on the dataset itself, but also yields superior performance on other multi-hop cresstion-nnweringg
datasets, e.g., 21.5% on GSM8K (Cobbe et al., 2021), 15.3% on BeerQA (Qi et al., 2021b), 11.1% on MuSiQue (Trivedi et al., 2022) and 14.8% on CofCA (Wu et al., 2024).
We also show transfer across disparate tasks, namely mathematical reasoning to question-answering and vice versa. Training only on multi-hop HotPotQA question- answering improves performance on GSM8K (Cobbe et al., 2021) (a math dataset) by 16.9% and training on GSM8K improves performance on HotPotQA (multi-hop question-answering) by 9.2%.
We analyze the impact of synthetic data filtering strategies in a multi-step reasoning and tool use setting, and demonstrate that models learn best from datasets which have been filtered step to ensure high-quality reasoning traces, but which are not filtered by outcome (correct final answer).
2

---

Published as a conference paper at COLM 2025
We explore the impact of training dataset size and model size SWiRL, observing that significant gains can be achieved even with just 1000 trajectories and that smaller models (Gemma-2-2b and 9b) can benefit from in-domain SWiRL, but do not display the same generalization as their larger counterpart, Gemma-2-27b.
We demonstrate that SWiRL effectively improves the average process reward, even when evaluated on out-of- cuttof-diitribution tasks, suggesting that the downstream per- formance gains are driven by improved multi-step reasoning.
2 Methodology
Our methodology, Step-Wise Reinforcement Learning (SWiRL), consists of two stages. In the first stage, we generate and filter synthetic data. In the second stage, we step-wise reinforcement learning approach to optimize generative base model on the synthetic trajectories. SWiRL does not require golden labels human annotations, and instead relies entirely on model-based judgments for data generation, filtering, and RL optimization. The overall flow of our methodology is depicted in Figure (Stage 1) and Figure 2 (Stage 2).
2.1 Multi-Step Data Collection
SWiRL Stage Synthetic Data Generation
Action
Action N
Promp:
Env resporse Step
Env resporse Step 2
(Final Answer)
Stop
Action 1's process label, given prior context (prompt)
Action 2's process label, given prior context (prompt and Step 1)
Action final answor)'s process label, given prior context (prompt and Steps to N-1)
steps judged ressonable by an LLM (all process positive)?
Final enswer correct given golden answer?
Process-fitered data
Outcome filtered data
Figure 1: In SWiRL Stage 1, we generate and filter multi-step synthetic trajectories. At each step, the model is free to generate a chain of thought, call tool such as search engine or calculator, and/or / produce an answer to original question. Process-filtered data corresponds to trajectories in which every step is judged to be reasonable by model judge (Gemini 1.5 Pro Thinking). Outcome-filteed filteeedded data corresponds to trajectories with final answer that matches the golden label.
In Stage (see Figure 1), we generate synthetic trajectories consisting of multiple steps of reasoning and tool use, which we use as training data for the step-wise RL methodology described in the next section. To compile a large-scale collection of synthetic trajectories, we augment a language model with tool (e.g., à search engine calculator), and iteratively prompt the model to generate multi-step trajectories. At each step, the model is asked to choose whether call tool or produce a final answer, and is always free generate chains of thought (which it typically does). If the model generates tool use call, it is parsed from the overall response, executed in the environment, and the result is presented to the model in the next step. See Appendix A for the prompt, which contains question, explicit instructions regarding multi-step tool utilization,
the results of prior tool use calls.
For each multi-step synthetic trajectory, we define the following annotations. The trajectory itself is denoted by 7 = (s, ",,,,,,,, following state s; contains the entire context so far, containing state 5;-1, action di=1, and the environment (tool call) response to a; Each action a; is the model response, given state s,. The last action, ag, is the model's answer to the original prompt.
"ty....,,,,, Sk, ax). The first state $₁ is the original prompt. Each
3

---

Published as a conference paper at COLM 2025
In this work, compiled dataset of 50,000 synthetic trajectories seeded by 10,000 multi- step questions from the HotPotQA training set (Yang et al., 2018) (i.e,, 5 trajectories per question), and mathematical reasoning dataset of 37,500 synthetic trajectories seeded by the 7,500 questions in the GSM8K training set (Cobbe et al., 2021). Note that, for HotPotQA, filtered out "Easy" questions, which can typically be answered with single search query. To prevent synthetic trajectories from being excessively long, we set a maximum step count of 5 for HotPotQQA questions, and 10 for GSM8K questions (which typically require 2-8 steps to solve).
Having compiled these datasets, we consider four different filtering strategies and measure their impact on performance (Figure 1): (1) No filtering; (2) Process filtering, where we retain trajectories in which each step was deemed reasonable given all previous steps. Concretely, model (Gemini 1.5 Pro Thinking, in our case) is prompted to render binary judgment as to whether action aj reasonable given the context sj. See Appendix A for our prompt. No golden labels are used; (3) Outcome filtering, where we select trajectories based solely whether the final response, Ak, matches the golden answer; and (4) Process and outcome filtering, in which take the intersection of both filtering approaches and retain only trajectories that exhibit both step-wise soundness and correct final outcomes.
Recent approaches to synthetic data distillation, such as Deep-Seek R1 (DeepSeek-Al and others, 2025), have demonstrated that synthetic data filtered for correct outcomes can lead to good performance with single-step RL and supervised finetuning (SFT). In this work, we sought to explore whether this pattern would hold in multi-step, tool use setting, and to explore the impact of both outcome and process filters. Like these prior work, we observed that filtering multi-step trajectories for correctness was effective for SFT, and in fact critical for good performance. However, we found that SWiRL, unlike SFT, can learn even from trajectories that end in incorrect final answers. In fact, we achieve our best results by including process-fillen filleeed data, regardless of the correctness of the outcome.
Step-Wise Reinforcement Learning Methodology
SWiRL Stage 2: Step-Wise RL Optimization
Action
figure older. search age Hughos. <search_ query>age of Glenn hughes</search_gge querrr"
Prompt
Rewward
Neward model
Hughes Ross Lynch?"
Step
generatus score for ial wetionn given prizr slaps.
Action
"Next, should find out what Ross Lynch's age is. <search, query>Ross Lynch age</search_ooe query=
Reward
Prompt "Who Clann Hughts
Env
Respanee
generates scom lest actionn
Neward
Step
Step
"Given the results of my previous searches. have enough information to answer the question. <answer>GGenn Hughes<lanswer>>
Action 1
Action
Action (Response)
Reward
Promot
Env
Eni Response
Reward made generalas soone For last action, givan prior stepe. (no access to golden answers.
"Whe Gienn Hughos Ross Lynch?
Response
Step
Step
Step
Figure 2: In SWiRL Stage 2, we perform step-wise RL to train on the synthetic multi-step trajectories from Stage 1. Each step contains an action, which corresponds to tool call or the final response. The model is free to generate chains of thought during each step. The environment responses are captured in the prior steps of the synthetic trajectories, which were generated offline. Granular feedback is provided by generative reward model, which used perform RL optimization directly on each action, given the prior context.
As shown in Figure 2, we propose a RL approach capable of learning effectively from the synthetic multi-step trajectories generated in Stage 1. At each step, base model is optimized to predict either the next intermediate step or the final response based on preceding context. At each step i, the model has access to the full contextual history, including the original prompt, all previous model-generated steps and any applicable environment response corresponding to those steps.
4

---

Published as a conference paper at COLM 2025
Thus, our objective function is the expected sum of stepwise rewards:
((0) = Eg-T, (>) [[(as)]
Here, ne is the base model parametrized by 0, which is finetuned via SWiRL (Note that we also use Ty to generate synthetic data.) T denotes the set of all states in the synthetic multi-step trajectories, i.e. each incremental state within each trajectory T. The reward signal (alss is derived from a generative reward model, specifically Gemini 1.5 Pro in our experiments, which assesses the quality of the generated response given the context No golden labels are used.
We optimize the above expected reward using the same policy gradient algorithm used in Gemma 2 for optimizing the human feedback reward (Gemma Team et al., 2024a;b). Our granular, step-by-step finetuning paradigm enables the model to learn both local decision- making (next-step prediction) and global trajectory optimization (final response generation) while being guided by immediate feedback on the soundness of each prediction.
Step-Wise Inference-time Evaluation
As shown in Figure 3, at inference time, we iteratively prompt the model to either call a tool or produce a final answer. the model generates a search query (indicated by <search quear> < </search query> tags), we parse out that query, embed it with Gecko model, perform a nearest neighbor lookup in the corresponding vector database, and inject the retrieved article into the model's context window. If the model generates calculator tool call (indicated by <math_exp expression, execute with a SymPy interpreter, and inject the calculated results into the context window. This process terminates when the model either produces an answer (signaled by producing <answer> </answers tags) or reaches the maximum number of queries (5 for question-anssering datasets, and 10 for mathematical reasoning datasets). See Appendix E for example trajectories.
< </math.exp>>> tags), we parse out the mathematical
SWIRL Multi-Step Inference
Action
Action N (Final Answer)
Prompt
Env response
Ertv response Slop
Step
Slep
USER: Please hals anaree Kalowing cueetoni just uas calculater, plesss perenaeuu
USER: Planse help arower Kill ques ion judt fiee aatermaton adi??
USER: Plaase arewar Kelowing quertion
USER: Please helo ni anover the yorth fhrr Yuch Sorarrabor sok77
USERC Please help snowill wortss How mich sooh walermeton scid
waternatoo add?
vobovet
wandor add 20% been 0.33 30 the origi price]
MODEL:
MODEL:
VODEL:
NOUEL:
chocks ligs
aiternokons
Fill poouds reulte in tha reliwving
USER: DUer39 20
USER:
0129
MODEL:
That vocir that anternatin (orighae (oed Hour Sturiulloorrd Doiacc latei
VODEL:
NODEL:
Deeee you hows arough Intrrmation tamaienn www.e//wtwttitttteeaeeeeeeae begs
USER: 2.20000
USER:
The quecil AuA Bul vcedgi bouget 50 sutermations dinn Hlw muchh uea each vellarme bit
USBR) 60+20 9-90
MODEL: the vordoo
NORRI (-1003)
vown snl 130 (00 icalloulal (00) Stwicalcuaaoor
USer:: 100 20
NODEL: 1am ready INIRANT
Figure SWiRL Multi-Step Inference. At inference time, we iteratively prompt the model to call available tools as many times as necessary (up to limit) before answering the original question. Here, prompts are truncated for clarity and to adhere to space limitations, but full trajectories are available in Appendix E.
3 Related Work
Reinforcement Learning for LLM Finetuning, One prominent approach, Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Christiano al., 2023), con-
5

---

Published as a conference paper at COLM 2025
sists of training a reward model on human preference labels at the response level, followed by RL optimization using Proximal Policy Optimization (PPO) (Schulman et al., 2017). Build- ing upon this framework, Reinforcement Learning with AI Feedback (RLAIF) (Bai et al., 2022) has emerged as a scalable alternative that leverages Al models to generate feedback based on predefined principles or constitutions, reducing the need for costly human annota- tions. RL from Execution Feedback (RLEF) (Gehring et al., 2025) uses environment feedback, such as pass rate on coding test cases, to calculate the reward, which it then optimized via PPO. Besides PPO, other RL optimizations, such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) and its successors (e.g., Azar et al. (2023); Ethayarajh et al. (2024); Meng et al. (2024); Lanchantin et al. (2025)) as well as GRPO (Shao et al., 2024) have also proven to be effective for finetuning LLMs to maximize a target reward. A limitation of the above approaches is that they focus on single-teep optimization with the reward being calculated only at the end of the episode, leading to suboptimal performance for multi-step optimization (Liu et al., 2024; Wang et al., 2024). In SWiRL, we focus on scenarios where mul- tiple steps of reasoning and tool calls are necessary prior to generating a response. Unlike the above methods, SWiRL enables the model to receive feedback on its granular stepwise actions which leads to better multi-step reasoning and tool use across longer horizons.
Multi-step Optimization with RL. Recent work including DQO (Liu et al., 2024) and OREO (Wang et al., 2024) propose offline reinforcement learning to improve multi-step reasoning for LLMs. However, neither focuses on enhancing a model's ability to use tools or interact with an external environment. Additionally, unlike our approach, which optimizes at the (reasoning) step level, DQO relies on token-level actions, which as shown in (Wang al., 2024), are generally less effective than step-level actions. Moreover, OREO requires training a separate value network and policy, and relies on iterative co-optimization of both models. The process of maintaining, training, and serving these two models can be prohibitively expensive, particularly for larger models. PRIME (Cui et al., 2025) proposes an online approach to improve multi-step reasoning, but does not enable tool use offline training. Tulu-3 (Lambert et al., 2025) uses verifiable rewards to train a language model to do better at math, but requires access to golden labels.
Reasoning Improvement with Synthetic Data. Several approaches have been proposed for generating synthetic reasoning data. These methods either rely on golden labels to filter the data or use a combination of golden labels and process or outcome reward models (Zelikman et al., 2022; Singh et al., 2024). For example, STaR (Zelikman et al., 2022) generates chain- of-thoughts (CoT) for reasoning questions, filters for those that result in correct answers, and performs Supervised Fine-Tuning (SFT) on those reasoning traces. The paper also proposes an augmentation technique called "rationalization" in which for each question the model answered incorrectly, the model is provided with the correct answer and prompted to generate a CoT that leads to that answer. Rejection finetuning (RFT) (Yuan et al., 2023) is another method that relies on collecting reasoning traces from the model and using those with correct outcomes for SFT. ReST (Gulcehre et al., 2023) demonstrates strong performance on machine translation by iteratively generating data and then finetuning on that data using either a supervised or reinforcement learning objective. ReSTEM (Singh et al., 2024) is an extension of ReST which outperforms training human data alone for math and coding evaluations, but which plateaus after a few iterations, presumably due to overfitting. Our method also uses a model-based approach to generate multi-step trajectories. However, we show that using a model to label the steps within each reasoning trajectory leads to higher out-of-domain generalization than using only the trajectories which contain correct final answers, meaning that we do not require golden labels. In addition, we enable the model to use tools iteratively to perform multi-hop question answering and mathematical reasoning.
Process vs. Outcome Based Optimization. There have been a number of attempts to compare the effectiveness of process and outcome-based approaches in the domain of math and reasoning (Lightman et al., 2023; Uesato et al., 2022; Snell et al., 2024). For example, (Lightman et al., 2023) showed that (Outcome Reward Models) ORMs are more effective than (Process Reward Models) PRMs at the task of ranking samples from a fixed generator model, whereas Uesato et al. (2022) demonstrated that outcome supervision yields comparable accuracy to process supervision at lower cost, but that the reasoning traces from the resulting model exhibit lower fidelity. Both rely on expensive human annotations and golden labels,
6

---

Published as a conference paper at COLM 2025
and do not explore the impact of PRMs and ORMs in reinforcement learning optimization, or the differential effect of data filtering on supervised vs. RL optimization objectives.
4 Experiments
Datasets Metrics
HotpotQA PMf
CofCA (Avg) PM+
MuSiQue PMf
Proprietary LLMs
GPT-4
74.8
51.9
63.9
GPTT-3.5
62.8
40.7
53.1
Gemini 1.0 Pro
63.5
33,3
46,9
Bing Chat
72.1
41.6
52.3
Ol-preview
76.9
58.5
67.9
Open Source LLMs
Llama 2-7b Mistral-7b
38.5
28.9
34.2
34.9
25.6
29.2
Qwen 2-7b
39.3
30.7
33.5
Base Gemma 2-27b
58.6
31.7
35,4
SWiRL
Gemma 2-27b (Ours)
67.8
39.3
43.6
Table 1: Comparison of Accuracy (PMt: Partial Match) across Multiple Datasets: HotpotQA, CofCA (Average of 2-hop, 3-hop, and 4-hop), and MuSiQue. Baseline results were drawn from Wu et al. (2024). The Gemma-2 models, both SWiRL and the base model, were not given access to the context documents, but were allowed to sequentially query vector database. The SWiRL model was trained on HotPotQA using process- process-filtered filtered data, and for consistency with baseline results, evaluated on GPT-4o with the same prompts as Wu al. (2024) on 300 randomly subsampled questions. See Appendix F for example ids.
4.1 Evaluation Datasets
To evaluate performance multi-step search tool use, we selected five challenging multi- hop question-answering and mathematical reasoning datasets:
HotPotQA (Yang et al., 2018) is comprised of multi-hop questions from a variety of domains. Human annotators constructed the questions to be answerable only by combining information from multiple paragraphs of Wikipedia.
MuSiQue (Trivedi et al., 2022) is a multi-hop question-answering dataset con- structed by chaining together multiple single-hop questions.
CofCA (Wu et al., 2024) is multi-hop dataset constructed to be answerable only by querying counterfactual version of Wikipedia. It contains 2- to 4-hop questions.
BeerQA (Qi et al., 2021a) is an extension of HotPotQA cesigned to include an even greater number of hops than the original dataset.
GSMSK (Cobbe et al., 2021) is dataset composed of grade school math word problems, which typically take 2-8 steps to solve.
For question-answerinn datasets, we set up vector database containing all articles from each data split using Gecko-1B 1B with 768 768-dimmesional embeddings (English) (Lee et al., 2024).
For the experiments in Table 1, we follow the same procedure as Wu et al. (2024), evaluating performance on 300 randomly subsampled examples from the target dataset, using the same language model as judge (GITT40) and the same prompt. For every other experiment in this paper, we used Gemma-2-27b as our judge, as this was more cost effective, with the exception of GSM8K for which we used Gemini 1.5 Pro as it exhibited noticeably better numeric evaluation. Model-based evaluation is emerging as a scalable and less brittle alternative to exact match and F1 metrics (Zheng et al., 2023; Gu et al., 2025), but does
7

---

Published as a conference paper at COLM 2025
introduce source of stochasticity into the evaluation. See Appendix D for manual inspection and enror analysis three different model judges.
As described in Section 2.3, for each question, iteratively prompt the model to either call a tool or produce a final answer, and limit the maximum number of queries to for question-answering datasets, and 10 for mathematical reasoning datasets.
4.2 Results and Discussion
Impact Data Filtering, Inference Method: Multi-Step Tool Use
1,2
SwiRL-Gemme-2-277. Teiining Date: Outcome Filtered
Training WWIRL-Gemme-22271 Training Data: Not Filtered
SWIRL-Geemm--22 training Data:
0.8
Accuracy
0.6
0.2
0.0
HotPotQA
MuSiQue
CofCA
BeerQA
Figure 4: Impact of Data Filtering Model Performance. Synthetic data for training is derived from HotPotQA, SWiRL learns to perform multi-hop question answering even when trained on unfiltered synthetic data. SWiRL's best performance comes from training on process-only filtered data, where the data is selected based on the soundness of each step within its reasoning traces, but which includes both correct and incorrect responses.
Impact of Data Filtering on Model Performance: We evaluated the influence of various filtering mechanisms on downstream task accuracy, as shown in Figure 4. Concretely, we consider 4 different types of filtering: no filtering, outcome-based filtering that ensures correct final answers, process-based filtering that ensure that each step is correct as judged by model, and both process and outcome-based filtering.
In all experiments, we fix the number of trajectories used for finetuning (with the exception of our ablation study on the impact scaling dataset size), and we provided all models with access to an appropriate tool. Notably, process-only filtering consistently yields the highest accuracy, suggesting that focusing on the procedural aspects of data refinement more important than the correctness of training trajectory. While both unfiltered and filtered data demonstrated an improvement over the baseline model, filtering for correctness usually harms performance; with the exception of MuSiQue, nutcome-filteres and process-filtered data is less effective than unfiltered data. We hypothesize that this is because SWiRL actually benefits from having access to both positive and negative examples. These results underscore the relative unimportance of outcome-based filtering, which requires golden labels. They also demonstrate that our process RL method can effectively learn from even trajectories with incorrect final answers.
outcome
To measure generalization across training tasks, we evaluated the mathematical reasoning capabilities of a model trained on multi-hop question- answering with search tool use (HotPotQA). Specifically, we evaluated the performance of this model on GSMSK, mathematical reasoning task, providing the model with SymPy interpreter to use as a calculator. This experiment was run on a different random subsample of 300 examples. As shown in Table 2, applying SWiRL on out-of-distributio data and tasks still improves performance.
Generalization Across Disparate Tasks:
Effect of Tool Use:
As discussed in Section 2.3, at inference time, multi-step eval as shown Figure 3 and we iteratively prompt the model to make tool calls necessary to answer the question. As shown in Figure 5, both base and SWiRL models
the proposed
8

---

Published as a conference paper at COLM 2025
GSM8K (math)
HotPoiQA
CofCA (qa)
BeerQA (qa)
MuSiQue
(qa)
(qa)
Base Model
0.65
0.65
0.54
0.59
0.45
SWIRL on GSM8K (math)
0.79
0.71
0.56
0.68
0.49
SWIRL on HotPotQA (qa)
0.76
0.73
0.62
0.68
0.50
Table 2: SWiRL Generalization Performance, Finetuning synthetic traces from HotPotQA or GSM8K improves performance on both in-distribution and out-of-distributin tasks. Interestingly, training different domain and tool (e.g. math and calculator) improves performance on cuestion-answering with a search engine and vice versa, suggesting the effectiveness of SWiRL in improving general multi-step reasoning and tool use capability.
Porformance Base SWIRL Models With and Without Tool Use
Gemma-2-2//b inference Method: No Use
GWIRL-Gemma-22222 Inference Method: No Tool Use
Gemmaa2-22/b inference Method: Multi-Step tool Use
SWRL Gemma 27b, Inference Method: Multi Use
Accuracy
HotPerQA
MuSique
CotCA
BeerGA
GS
Figure 5: Performance of SWiRL With and Without Multi-Step Tool Use. SWiRL's multi- step tool use inference improves the performance of both the base model and the SWiRL- finetuned model, but benefits the latter substantially more. Even without access to a tool, the SWiRL model outperforms, suggesting that SWiRL training improves the model's ability to break down complex problems into multiple manageable subtasks.
improve with SWiRL's multi-step tool use inference, but SWiRL-training offers even further improvements. Notably, the SWiRL model exhibits substantial improvements, even without access to a tool, suggesting that SWiRL training improves the model's ability to break down complex problems into multiple manageable subtasks.
Our experiments on scaling the fine-tuning dataset size reveal clear trend: SWiRL has the ability to leverage larger datasets, even when using only process-filtered data, as shown in Figure 6. As the fine-tuning dataset size increases, consistent enhancement model performance is observed across our target multi-step reasoning tasks. While a limited dataset of 100 data points appears insufficient for the model to effectively generalize, a significant improvement is evident with 1,000 data points, showing solid gains across all datasets. Furthermore, scaling up to 10,000 data points continues to yield further performance enhancements, confirming the efficacy our method in capitalizing on larger datasets for improved reasoning capabilities.
Impact of Scaling Finetuning Dataset and Model Size:
We also varied model size, observing that smaller models (2b and 9b) may benefit from in-domain SWiRL, but do not display the same generalization as their larger counterpart, Gemma 27b. See results in Appendix B.
Effect Mean Process Label Accuracy: In the previous subsections, evaluated the effect of SWiRL on downstream task accuracy. Here, we take a deeper look to understand how SWiRL achieves these performance improvements. In Table show the average process

---

Published as a conference paper at COLM 2025
Training Data: Flitered
HotPotOA
MusiQue
CafCA
BeerQA
OSMAAK
0.75
Accuracy
SWIRL-Genma 2 27b
Gemmin-2-221
10"
10"
10
10"
10a
10a
12
10a
10
10'
10'
10
10
Training Dataset Size
Training Dataset Size
Training Datoset Size
Training Dataset Size
Traniing Dotaset Size
Figure 6: Performance as Function Synthetic Dataset Size. Synthetic training data is derived from HotPotQA, and accuracy is evaluated by Gemma 2 27b. As scale the dataset size, we observe consistent improvements in model performance, With only 1000 data points, the model robustly improves both on in- and out- of distribution datasets.
label accuracy for the baseline model vs. a SWiRL finetuned model on 500 trajectories (seeded by 100 questions) for both HotPoiQA and GSM8K. To calculate the score per step, we use the same model and prompt as we used for process filtering, as described in Section We take a macro-average of the process label scores within and then across trajectories. We observe that both for in-distribution and out-of-distribution tasks, the SWiRL model generates trajectories with higher average process labels, suggesting that the higher final accuracies are driven by better multi-step reasoning.
HotPotQA (in distribution)
GSM8K
(out of distribution)
Base (Mean Process Label) SWiRL on HotPotQA (Mean Process Label)
82.5%
87.5%
91.0%
91.6%
Table 3: Impact of SWiRL on Process Correctness. After our multi-step RL optimization, we observe that the average correctness of each step improves over the base model on both in- and out- of distribution tasks.
Conclusion
In this work, we propose a synthetic data generation and offline reinforcement learning approach to multi-step reasoning and tool use. This approach outperforms baselines by average 15% across challenging multi-hop question-answering and mathematical reasoning tasks. We explore the effect of different data filtering strategies in multi-step, tool use setting, and find that our RL approach is effective even on unfiltered data, but performs best on process-filtered data. Unlike supervised finetuning, our RL approach can learn from trajectories with incorrect final answers and actually benefits from the presence of mixture of both correct and incorrect final answers. SWiRL demonstrates strong generalization properties, improving performance mathematical reasoning (GSM8K) by 16.9% when trained on multi-hop duestion-answent (HotPotQA) and 9.2% vice versa.
References
Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024.
URL
http:://ww......tttooococcoooooe http://baaabiicccccccaaaaa..c... Model_Card__llade. pdf.
8857627/
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. General Theoretical Paradigm to Understand Learning from Human Preferences, 2023. URL https://arxxvv.rge/ abgg/////////ag/ 12036.
12036.
10

---

Published as a conference paper at COLM 2025
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-oodds, Ben Mann, Darío Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional Al: Harmlessness from AI Feedback, 2022. URL https://arxiv.ort 2////bb//2222 08073.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL htttt:://rxxx org/ abs/1706. 03741.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoe V. Le, Sergey Levine, and Yi Ma. SFT Memorizes, RL Generalizes: A Compara- tive Study of Foundation Model Post-training, 2025. URL http:::://aaxvvoorg/ab////////// 17161.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers Solve Math Word Problems, 2021. URL https://arxiv https://arxi..or//bb//2/2/ orgg/bs///// 14168.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process Reinforcement through Implicit Rewards, 2025. URL https://arxiv.org wttp://ww..gg////////222/// 01456.
DeepSeek-AI and others. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025. doi: (0.48550/axXiv.55011 12948. URL
12948.
Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini. CodeMonkeys: Scaling Test-Time Compute for Software Engineering, 2025.
URL https://arxxvvorr//baaa. 24422 14723.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model Alignment as Prospect Theoretic Optimization, 2024. URL https://arxiv.org/ abs/2402. 01 306..
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning, 2025. URL / howll.orrg//bbbe http:///.roo//////1/11/ 2410 0.02999.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganíni, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chu- layuth Asaw aroengggn,,,,,, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo
11

---

Published as a conference paper at COLM 2025
Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, Tom Hennigan, George Roelots, van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puig- domenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Whadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross Mcllroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong Choe, Alex Tomala, Chalence Safranek-Shrader, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićević, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, lain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa Anne Hendricks, Corey Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skenry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Mag- gioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mi- haela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeyncep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Ilic, Taylor Johnn Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz Davoodi, Albert Weston, Lakshman Yagati,
12

---

Published as a conference paper at COLM 2025
Arun Ahuja, Isabel Gao, Golan Pundak, Susan Zhang, Michael Azzam, Khe Chai Sim, Sergi SheeleesJamee Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Hassas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevey, Ramona Comanescu, Richard Ives, Sherry Yang, Cosmo Du, Bo Li, Zizhao Zhang, Mariko linuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Sa- putro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savi- nov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Korrn-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, Anton Briukhov, Neil Houlsby, Mia Glaese, Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Amnol,, Solomon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hel- lendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang, Craig Swanson, Biao Zhang, Benjamin Lee, I'aul Kishan Rubenstein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi bessemmi,essiee Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Felix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iwanicki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, Mukarram Tariq, Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Fgyed, Francois Galilee, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Fara- bet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V, Gabriel Carvajal, Josef Broder, Tulia Comsa, Alena Repina, William Wong, Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lu- cia Loher, Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, lonel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand lyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Centtamrri--Frmanddee Wojciech Fica,
Simon
Chemmawat.Jasmne
13

---

Published as a conference paper at COLM 2025
Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Eslami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, lan Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong, Boyu Wang, Sujeevan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann-Rice, Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Nejc Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Pic- cinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora, Tsendsuren Munkhdalai, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yannis Assael, Thomas Brovelli, Prateek Jain, Mihajlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Latorre-Chimoto, Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Choquette-Chon Choo, James Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velic, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Bartek Perz, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplanis, Xiang- Hai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon, Adrian Goedeckemeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri, Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de Cesare, Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Áharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana, Austin Matthews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis
14

---

Published as a conference paper at COLM 2025
Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Liu, Madhavi Sewak, Bryce Petrini, DongHyum Choi, Ivan Philips, Ziyue Wang, loana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MNM Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjiao Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Freder- ick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz Kepa, François-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejczuk, Mark Geller, Niccolo Dal Santo, Valentin Anklin, Majd Al Merey, Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuoglu, Jeff Dean, and Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL http:///arxiv..or/bbbsstte
05530.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, lan Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel I. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open Models Based on Gemini Research and Technology, 2024a. URL 08295.
https://arxizs http:///axxvv.orab///ss//
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Pi- otr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean- Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christo- pher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Plucisskk
Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Göreer, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo
15

---

Published as a conference paper at COLM 2025
Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh De- vanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kup- pala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Iom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Gi- ang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving Open Language Models at a Practical Size, 2024b. URL https://arxiv.org// 240. 001 18.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert,
16

---

Published as a conference paper at COLM 2025
Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, lean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philipp Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, lamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniy Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito,o, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama 3 Herd of Models, 2024. URL https: //arxiv..vvv///////// org hbtp:/bs/2200 240b.///440... 21783.
Hakan Inan,
17

---

Published as a conference paper at COLM 2025
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey llm-as-a-judge, 2025. URL https: //arxiv.ore warxiv.org/absst htsp///s///111/ wwt.//rg//////l//ao//
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced Self-Training (ReST) for Language Modeling, 2023. URL
08998.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2024. URL httt::://aaxivvorree.. 22 06770.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing Frontiers in Open Language Model Post- bost-Training, 2025. URL https: //axxiv. http//axvvvogggg/a// orgg/bs///11111 15124.
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse Preference Optimization, 2025. URL https://arxiv.
org/abs/2501. 18101.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gus- tavo Hernandez. Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Iftekhar Naim. Gecko: Ver- satile Text Embeddings Distilled from Large Language Models, 2024.
URL https:
20327..
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-Level Code Generation with AlphaCode. arXiv preprint ar xxXie:2203.07814, 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's Verify Step by Step, 2023. URL https://arxiv..rom wwtprriwww..ggab////eeee 20050.
Guanlin Liu, Kaixuan Ji, Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, and Lin Yan. Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization, 2024. URL https://arxiv..or httg//bb/2411 2410. 009302.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization with Reference-Free Reward, 2024. URL /tL..ttts://rrvvv..gr/bb////
14734.
OpenAL, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Sramadett-Shapirge Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty
18

---

Published as a conference paper at COLM 2025
Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Contijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Eukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeey Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Iseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report, 2024. URL
28777.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.or//abbst
021555.
Peng Qi, Haejun Lee, Oghenetegiri "TG" Sido, and Christopher D. Manning, Answering Open-Domain Questions of Varying Reasoning Steps from Text, 2021a. URL https: hatrgg/bb/22000 12527.
Peng Qi, Haejun Lee, Oghenetegiri "TG" Sido, and Christopher D. Manning. Answering Open-Domain Questions of Varying Reasoning Steps from Text, 2021b. URL https: //arxiv. wtt.xiv.org/bbb/20000 12527.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model Secretly a Reward Model. ar Xiv preprint ar Xii::2305.18290, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, 2017. URL http:///axxxv..r/bbbbaa/ 166417707.06
19

---

Published as a conference paper at COLM 2025
Amrith Sethur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold, 2024. URL
14532.
Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute Trends Across Thrre Eras of Machine Learning. In 2022 International Joint Conference on Neural Networks (JJCNN), PP. 1-8. IEEE, July 2022. doi: 10.1109/ 9jamm55064420222.8999114 URL http://dx. doi org/10. 1119/11CNN5566G 2022. 9891914.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. URL https://arxiis org/abs/ 2402. 03300.
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond Human Data: Scaling Self-Training for Problem- Solving with Language Models, 2024. URL /
arggabs//312...m 0312.885..
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model parameters, 2024. URL https:// arxiv.org/abs//22 03314.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop Questions via Single-hop Question Composition, 2022. URL https://arxiv. org/abs/2108. 00573.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. URL https: httt::://rrii..or/aaa/////111111 wrgg
Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline Reinforcement Learning for LLM Multi-Step Reasoning, 2024. URL https://
16145.
Jian Wu, Linyi Yang, Zhen Wang, Manabu Okumura, and Yue Zhang. CofCA: A Step-
Wise Counterfactual Multi-hop QA benchmark, 2024. URL 11924.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, 2018. BRL.https//ariii.org//bbss
09600.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models, 2023. URL https: / /twpiv/wxivvvv///v////////a
https////aaxxvvooo///////// wtt.or///////...ao//////// 1182.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping Reason- ing With Reasoning, 2022. URL https::/arxxv.oor//asssss
14465.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxivy wwwp
http///bb//2..... 2tt.//3336000000 05685.
20

---

Published as a conference paper at COLM 2025
A
Prompts for Synthetic Data Generation, Filtering, and Evaluation
In this work, we use the following prompts for data generation, filtering, and evaluation.
Prompt Type
Prompt Text
Prompt Multi-Step Synthetic Data Generation for Question- Answering with Search Tool Use
<start_of <start_.ofturn>usser _turn>user
Please help me answer the following question in just a few words. If you think it would help to do search, please generate search query enclosed by <search <search_query>> query>> QUERY </sear ch.query> tags.
Some questions may require multiple searches in order to answer, so will allow you to make up to { sequential queries before answering the question.
Please do not repeat queries you have already issued, as this is waste time.
will provide search results in the following format: QUERY RESULT.
Once you have enough information, generate an answer enclosed by <answer ANSWER<<ansswer tags. Please either issue search query or answer the question, but not both.
The question is: 0
<end _frru
Prompt Type
Prompt Text
Prompt for Multi-Step Synthetic Data Generation for Mathemati- cal Reasoning with Calculator Tool
<start_of oof turn>user
Please help me answer the following question in just a few words. If you think it would help to use a calculator, please generate mathematical query enclosed by <math exp">> MATH EXP </math exp>> tags.
Use
Some questions may benefit from using a calculator multiple times in order to answer, so I will allow you to make up to {} sequential queries before answering the question.
Please do not repeat queries you have already issued, as this is waste of time. I will provide results in the following format: QUERY RESULT.
Once you have enough information, generate an answer enclosed by <answer>> ANSWER< tags.
Please either issue search query or answer the question, but not both.
The question is: 0
<end of turn
21

---

Published as a conference paper at COLM 2025
Prompt Type
Prompt Text
Prompt for Filtes--iltering Multi-Step Search Tool Use Trajec- tories
<start_of .etart.of.turn>-usrr .uurn>user
My boss asked me answer the following question with the help of a search engine: {
This means that I might need to decompose the question
into sequence of searches before being able to answer the question.
am trying to learn how to do this more effectively, so please provide feedback on my last message. Please take look at our conversation so far: {}
When evaluating message, please only consider the last message and do not penalize or reward me for previous messages.
When evaluating an answer, please consider only whether the answer follows from the search results, and not whether you believe the answer be correct. there not enough information from the search results to answer the question, you should rate any answer as "BAD". Pay close attention as it may initially seem like the answer is present when it is not.
When evaluating search query, please consider whether is likely to help me answer the original question. Explain your reasoning and then answer with either "GOOD" or "BAD". < <end_of.turn
Prompt Type
Prompt Text
Prompt for Evaluation Outcome- Filtering on Multi-Step Trajectories with Search Tool Use
<start of turn>>uee
nned you to help me grade the answer to the following question: "()".
The answer key says: (,, and my answer (}. Am I correct?
Please explain your reasoning and then answer "YES" or "NO".
Do not your knowledge to the decide, but simply check whether gave answer the answer key. <end_of_ turn?>
Prompt Type
Prompt Text
Prompt for Evaluation Outcome- Filtering on Multi-Step Trajectories with Calculator Tool Use
<start of turn>>user
Ined you to help me grade the answer the following question: "()".
The answer key says: (), and my answer is O- Am
correct?
Please explain your reasoning and then answer "YES" or "NO".
There are multiple ways to write the same answer. For example, "10", "10.00", "$10", and "$10.00" are all equivalent.
<end of turn
22

---

Published as a conference paper at COLM 2025
B Impact of Model Size on Effectiveness of SWiRL
The trend is that models are growing in parameter count over time (Sevilla et al., 2022), so measuring the impact of model size on the effectiveness of method can provide insight into its longevity and future impact. It is also interesting to see whether larger models are able to learn more general patterns from the training process, and therefore exhibit greater transfer learning across datasets and even domains (e.g. math vs. wuestion-nnserinng As shown in Figure 7, SWiRL demonstrates a clear performance boost over the baseline Gemma 2-27b model, showcasing consistent improvements across both in-domain (HotPottQ)) and out- of-domain datasets (MuSiQue, COFCA, and BeerQA), while the 2b and Gemma models also exhibit enhanced performance in-domain data, their generalization performance out-of-domain data is less consistent. This suggests that the effectiveness of SWiRL grows with increased model size, which is consistent with the observation that methods such RLHF (Ouyang et al., 2022) and RLAIF (Bai et al., 2022) are more effective for larger models.
Difference (SWIRL Score Bazz Model Score)
Performance Gains SWiRL over Base Model as Furvction
0.15
0.3
00.05
Evail Dataset
+0.00
CAICA
275
Model
Figure 7: SWiRL Performance vs. Model Size. Synthetic data for training is derived from HotPotQA. Step-Wise RL finetuning robustly improves performance over baseline for the 27b model across both in-domain (HotPotQA) and out-of-domain datasets (MuSiQue, CofCA, and BeerQA). However, while the in-domain improvements hold for smaller models, the out-of-domain performance is mixed, suggesting that the relative effectiveness of SWiRL is higher for larger models.
23

---

Published as a conference paper at COLM 2025
C Comparison of Supervised Finetuning and SWiRL
Figure 8 compares the performance of Supervised Fine- Fine-Tuning (SFT) and SWiRL on down- stream tasks. The results show that SFT leads worse overall performance when compared to SWiRL across all data filtering strategies. We observe that SFT performs better if we apply data that both process and outcome-filtered rather only process-filtered However, interestingly, SWiRL learns best from data that is only process-filtered. We attribute this to SFT's tendency to memorize, rather than generalize (Chu al., 2025; Setlur al., 2024), which can hinder the model's performance on new, unseen scenarios. In contrast, SWiRL has the ability to improve model performance by targeting per-step reward maximization. SWiRL enables the model to develop deeper understanding of the necessary steps of query generation and retrieval, which leads to enhanced planning and generalization.
Performance of Supervised Finctuning SWIRL inference Method: Multi-Step Tool Use
SI 911--emme-22 2-10mm training Data: Process and Outcome Hitered
SFT-Gemme 2-277b, Training Date: Process filtered
SWILLGGmmas Ge2ma-2-2/b. training Ueta: Process and Outcome Hitered
0.8
Training
Fittered
Accuracy
0.4-
0.0
MuSiQue
BearQA
Figure 8: Comparison of SFT and SWiRL. Synthetic data for training is derived from HotPotQA, and derive accuracy, Gemma 2 27b evaluates whether the model's answer matches the golden answer. SWiRL greatly benefits from process-only filtered traces, and unlike SFT, capable of learning from traces with both correct and incorrect outcomes.
24

---

Published as a conference paper at COLM 2025
D Error Analysis of Three LLM Judges
Table 4: Error Rates for Gemma-2-277 Judgments on HotPotQA (N=100)
Metric
Rate (%)
False Positive Rate (FPR)
4
False Negative Rate (FNR)
1
Table 5: Manual Analysis of LLM Math Grading Accuracy (N=100)
Model
FPR
FNR
Notes
Gemma-2-277
15
Overly permissive ("nice"); all errors involved units.
GPT-4o
0
10 Overly harsh; all errors in- volved units.
Gemini 1.5 Pro
0
Accurate, slightly permissive; all errors involved units.
To evaluate the suitability of language models serve as evaluators (i.e., check the correct- ness of a model answer, given golden answer), we manually checked the correctness of 100 model judgments from Gemma-2-27b on HotPotQA questions, As shown in Table 4, we found that the error was relatively low (4% false positives and 1% false negatives), justifying the use of this low cost open-source model as our LLM judge.
However, noticed that Gemma-2-27b made more ties, so we decided to run separate analysis for GSM8K, manually evaluating 100 model judgments each for three language models (Gemma-2-27b, GPT-4o, and Gemini 1.5 Pro). Interestingly, we found that Gemma-2-277 tended to be overly "permissive" in its grading, but had zero false negatives, whereas GPT-4o had relatively high false negative rate but false positives. We also observed that relative results were consistent across model judges; if GPT-44 gave a higher accuracy score a particular model, Gemma-2-27b did well, even the absolute scores differed. To reduce noise, we chose to use Gemini 1.5 Pro as the LLM judge for GSM8K, in spite of its higher cost.
when it came to numeric quanti-
25

---

Published as a conference paper at COLM 2025
E
Example Trajectories
For clarity, we provide example trajectories from question-answering and mathematical reasoning datasets, which showcase both search and calculator tool use.
Trajectory Type
Trajectory Text
Example Multi-Hop Question- Answering Trajectory with Search Tool Use Generated by SWiRL model on HotPotQA
<start.of toofff.tooffueeee
Please help answer the following question in just a few words. you think it would help to do a search, please generate search query enclosed by <ssaarch query? QUERY </search ch query> tags.
Some questions may require multiple searches in order to answer, will allow you to make up to sequential queries before answering the question.
Please do not repeat queries you have already issued, as this is waste of time.
will provide search results in the following format: QUERY RESULT.
Once you have enough information, generate an answer enclosed by <answer ANweerr>AAWWRRR-AA //an/e/e/////wr>> tags. Please either issue search query answer the question, but not both.
The question is: What company published both The
Scorch Trials and The Death Cure?
<end_of turn>>
<start tootrtoofoooo turn>>model
<search.query>>hh scorch trials publisher
< </search.query>
< <end_of of_turn.>
<start.or tf .furnn>eueerr
the scorch trials publisher -? The Scorch Trials 2010 young adult post-apocalyptic dystopian science fiction novel written by American author James Dashner and the second book, fourth chronologically, in "The Maze Runner" series. The novel was published on September 18, 2010 by Delacorte Press. preceded by "The Maze Runner", and followed by "The Death Cure". A was released on September 18, 2015 by 20th Century Fox. Lend_of_turn."
<start_of .ff.tttt>>mddell
<search.quee Teearrchquurr>>The Death cure publisher
< </search_query>
<end.of -ondn> turn>>
<start.of sitart.of.turn>-sse turnn>>uerr
The Death cure publisher-2 - The Death Cure 2011
young adult dystopian science fiction novel written by American writer James Dashner and the third book, fifth chronologically, in the "Maze Runner" series. It was published October 11, 2011 by Delacorte Press and preceded by "The Maze Runner" and "The Scorch Trials" and followed by the series prequels, "The Kill Order and The Fever Code." <end_of -urn>>> <start.of -tf_tttntommeell enswerrDelacorte Pressj/answer, answery wtt.offfttuuteoss...
26

---

Published as a conference paper at COLM 2025
Trajectory Type
Trajectory Text
Example Multi-Step Math Reason- ing Trajectory with Calculator Tool Use Generated by SWiRL model on GSM8K
<start.of -ooftturruuser
Please help me answer the following question in just a few words. If you think it would help to use a calculator, please generate mathematical query enclosed by <math_exp>> QUERY </math_exi _epp>>>ee>>____ tags.
Some questions may benefit from using calculator multiple times in order to answer, so will allow you make up to 10 sequential calls before answering the question.
Please do not repeat queries you have already issued, as this is waste of time.
will provide search results in the following format: QUERY RESULT.
Once you have enough information, generate an answer enclosed by <answer>> <answer>ANSSWERRR ANSWER<//aswee tags. Please either issue math query or answer the question, but not both.
The question is: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? <end_of _ndrr>
<start of turn>>model
<math exp><48 / 2 </math exp>>
<end of turn>> <start.of.turn>uu
48 / 24.0 <end_of furrrf
<start_of .ft.otfffftttteeelll
<math_exp>>45 24</math.exp>>
of
<start.or of_turn>>usr
48 + 24 72.0 <fnd.off.urm turn?>
<start of turn>>model
aasserr772, answerz <end.of.tur>><<es>>>
27

---

Published as a conference paper at COLM 2025
F
Example Ids
For reproducibility, we provide the ids of the examples used for evaluation.
HotPotQA Example Ids
5a82a0l::5 De5eea
Sable7
5u8 2344,
54552299 1377
288,
44277144552299200 2366,
55 Maa,
7e 391
SabaSb2155
1511112-511999900222222 127,
5 1599559999099 599506663333
Sudfbr5155 22d90 29,
Saddf64 244527222
5 15313333335599 15542999388e 42 4200
5a7mr776 55477775549944224 (816
54b45744554429 751 1-44748,
5a77660255 (2995181181300
5u89 158 1299 ca90)
5a7a1d84 (29007332400
Sac tasdadaa.fff..
50/23215442997 M9dc
5acH4055542299 2eoaddd
55548155544299 1799 162ctt9,
Sabc39 Sabc399 129770
5a8a7cb255 1299 30
Soc1178e55
5a
5tt.ae555229999 51299999990000 70099/9800
Sub 1e59555 2297222622200 1205500
3d75d 70eaee 1.,
5ab83588554 Mi00ee
5558 (29994737777
Sae 35436339999941
15
Se 99
Sudesinb/554 200
3ud15735554299955
798,
Sa8901 1 1096553366 25b,
55 1wwe//a
el.
5.79176655
Sadc101755 1299 1299 1385688200
508da553555 1 1299 Lae Addfe.
SaddHae255 2292200
(51226266552222222626 Idee 1o03a.
Sadia 55429942 25dd
5edbH5555429944
6633666545994444 c4 lefl,
Sa7b665/55 129931 da 12a866
5 SS 26605662666555
Saffb00/55 (9600//7752995000aad 10
5ae522 dd 16250
5u77-5-17555-
Sab 185756 60dd/ lbc5,
5u891 0899182556-2220 129111299555585 Zaf.
5695516915422977 2062
Sa de lnd 33.
118730885542991177777 7777777777710 7811770
5u8668:115-1199211 220
Sabe99a5555 (atpaa555519999995 6229 1809.
5ab2688a855 5ab2ehaaS5 200
5ae5dec7SS 080788
Sabe 8354 2995a 2644
5u8459-6551299 301999946
5587099953499000 299000
Sab 55 2299/ 43d 22e2be
Sa8-4dd-5551294 229.5/555599979777
5a87 55871e 229 29900
Saff-d33755 12977 dd3.
5ab277 299 999
5abceu8155 1299 1 199.
5889756 155 1255555199551553 1133995553313600
SudldHHa55429900 (2
26 de
300- 2209224331 dall)
5 229339
Sae 4a4461133599996666666
5a76de0355 42995 3b/7 Bd.
Sabc/af155 Sutb
188518855--223 12990a I dd31 File
5a732ffb655-1 (atobbbb5--29999943322
Sael 222245549999422eeeee
5a72d5 1555-1999 Pig 20056
5a72 27 5x722x4555:29977 151
Sa7a9ca 55429941 66392663
3d778e3d555 129922
Sa5chia255
5abac9ed55 12996cc5e
Sae Bee7188899999216600000 148228
5a8td29d55 2991 Adda000
Hea 581 2995da 100
Sad 1299 2
(obodd115542996eee 1663,
Sa8lla 155 5396777
Sabo 224.222552222222
Sac29dde55 222.dddcc52222233333 37% 20ide.
Sa Sa7222ee55 12997 1e9de9227.
5ac221 5555 (2991889
Sa7P9cc255 42955 Rdddeee
Sabe Qaa 55 1299 76d 08022
Sue 329e 15-12991 1al
55582 2293 e5
155 12996 be2222
5.76200655 (ww.0e065512992210l 22
1a/918ec55 12991 4891 19ff,
SabSaafe 3833532999981131133 671 Debbaac
5aba94663342944bb 9953,
Sal 2ef22255--2225555
deddla,
2ab2 12995 15 offf.
Sab3d 5 22d% 2óec5
5ac2588255 5 $299 3605 1988..
5355SS 1299 feec
9 55 22900e6 if,
SoSa 22925229992222552222 2e
52 429e5 31
558c564 22922408922222 288
Salteff 24222224
5ab5500945449963777 88c37888882222
Sactif (5581299 129962296 7018,
Saft8d1tt33-19997777 bab.b/bbbbbbbbbbb//bb 12bb,
5a81 leelle (e5.998882200 12900
Sah 5a51129955525900 12500al leee
Sac26 250 4299211 32b3
Sac51904 (8 6494
fa8221lfc5554 2424 14, 2ebe5,
Sabce 36 55 29 Ui 2922.
5adbds725542200 23844294 80)
Sa7Sdd025 5 4299 1091 76e5aa,
Sa82 4055 edade2.
5a80901055 12296-0026..55.
Sadida 2291256622 d5e.
55759 12996976611166 laee.
557b566 155 $2990 124888757
Sa8a/b (tt///////0200/0
Sbddi/ SabSeab 4364429922222 34 sadddd,
Sae aa 499 e5/66 ett
5477169955449992 291
Sen d5s.
Sabe6 be584 429054 d.2.a
52/2333555 (50055552290000 M429
Sa 5ac262 5-555522200 2995 68
Sat7 Saf7act aardd 1 1299
di colon,
Sact 5555 1299 11 1i 18.
Sa8 18c2) 155 Ca 131 dal...
Sa89979 555 1299032 c77
942999 Ibd 18, Sae7 84555 1298 at 55566,
527299
5a7d ddds 54 www. 76966,
Sac1 15222 34296441
Sac2 555855 299 2dd Ioc2,
5a8835 554 224409964a5000
Su (23.e5555999667/aaeeeeee
5a77an f6 3 6e 199 dia,
fh 2 29977 25 1966,
Sar 3e 4299 Be4 1841,,
Sah oh 21 dld 15462.
5abdi5 2665 42942442222 Haflb,,,, Sae 15 5544999244 Sadect99s5 2 af
Sa 955 29 eifiii
Sa 98429 Wabeeff
Faechfa $225 2 4299 4 22ce
4AdE5 54 2299 www.ae22aa Be
50c0229 1296629999999998 af
Sa 2294 cead.
Sabbict005544 54905552999 1 4333a
Sal aal--eaadd55550 ba 15579/990
2111995442990 42990 1227 dis--i---s 4556,6 Sa/c1c59955
Safch8 be 3342996 9119
Sa8e/2205 54 222000 d 408 2b13, 33,
Sad5aa 10 (01555429912200 dd0e
Sadd582005 222055552244 le die dl2
Sa 55555999999999988 as bob9fta.
Sa7fee 0.55 49948877 a 7685,,
5a7b 2177bb222555 1299 31 da dacaaa.
Babeal Mateaa//434429974 20w422977/77744449 146,
Sabbee 7ebbee
5a8d4 42991 Bedaodd fl,
Bael afle 188429944bb dab, Sa/ / 2999995442999 7 clee Ibd.
Sa /a7//444798 4755 2994 de de66fff
5a87 245.5455455555555555.
Sab 2900 33 2311,
Sabe 22993 Ralle
199769919999999920 1999429924199477744
Sa te 2990 19
Sac a 204205 99 2449 fas8dd of 4294 34 3333
Sabac3e 428442996c Se44oe)
Sad Ea 24429992eeb ad
f133849 Sacl 1 P2 155 aoddd dooooee
Sal 75
Sacl0227655 494442294444 25c42ac
Sa88 12999994777777733
Sac32x27554222p 4299277997-2221193 7300600
Sale Sa8e 429
5a761 13 1977 oe
Raoeh 3 954-99922999929 2ne 6392.
fabs 31 c242777
Sactle 564 2054 492299999222999999
Sa7 Zedihh 2 29992 23 9bc Ida,
Sa Sa77 4200 da 1877
Sa ebe 42990499999946bbe c/weee.
So84 0S 48949996888888 488
Sala 255
& 5.23,
Sac fibee S.595555444555 25462226627777/// laal.
Saba 1 29
Fae 100 209477922
Sabell de 25 ded 1t177, ,
Sainbdu 3t 20
Sabed 254.51955599999911
5a742421 1 425991 4891 faoe,
5a80 29 2e00 151,
Sab41 462888429962344666669
Sade37s055 555 4 299 246c
Socl 4 d w....55420000 4296644 3ibelab,
Sab 2abf223d685 4299 Tnd df
Sa/elld 425 229
Sar 26
Sa Sao4hhdeh55 34440
Sood Ph 55--995-999988
Bac231 22122227742299892 429964499961113 31he2e3,
3230 42992e 3233c 368,
55776804455 32440226dd
Sabassdd 4129449991199 9300f1811
Sofe I 552
567 13555555-255 13 99477
5a805 Be 13542999997
Sa8dih Th 2775429944 4edd011
Sa9d440 2254000--99222994444499 3dcehh,
Tie fol 0554499944
5a8d 2420000222444440405040
Sab 3669 45 4255 499666
Saddille of 42990
Suil BaMkc380355442955666--6
Subfier 28842095475777
Sue444u wtw./755522999ddddd 1243144
Sande 6-77 2977777999999466666 e9bcn
40.
Sa8b00021 59429990
5a766 65542999444444 254 19a,
Sab52207 2002222222522222 42996 Ba
Safa/3dd wad.dd/4344996666666400000
S29063e9 24244429888820000
Sa7Ba4 Sc855 29933 da dadal22aaa 12544a
SaBeSbbes' 14 4299
5075573334299777559
255-2229999999 Delle,
67724666633479944
Sal Sal771562554220 37e0l,,
Sa7eldda
Sac Buc305778429922de
5ab72 222721922 192219c3,
106554299224442222
5ub66559654299511 92nd 272,
Sac 25x2296655242921 1112544112212244444m B125daccf)
Sah 31 2AA.R8544999299999999
Sabl (754555229944100 34116525443,
Sac wtw.he9699999d6733....
5ah99250 2ww.9022224222244aeeeeee
5w82 5w82100955 220.555660222666e22222
1100946666688880 168866a, 5582222777772222208444446 54173995552277777777777
28

---

Published as a conference paper at COLM 2025
CofCA Example Ids
Su86éfeee5 12911777116677
507db2/755 12900
SuSeella 25900 6052bal
Sue5255055 Icd.
5 Se44 3tx93c
Subobbdlss-42990 deall 4+d5,
54880722555 55366
Sab381115 15 554299699999 16b,
Sac2a9 1255 1292218
5ac 5455 4 2292282129668
2295 laaall.
Sabaa25155 255:29900 30
5ac3991 c55 129965772290999
5373332555 22923996
5 129908) 99865599,
Sedd117e 2542290
5479 (297))898299
Sab58 15422992
secoboffsS--190aadleee.... cl.
Soe (2ff655-229225522--488
5 16555129022eec////
Saeibidos 00000099599080000990000 (2555299922222 2922.62399
5uc77a31.55449955 55429965044555557
S 555429924deeeb6 1677 198,
Swe6-cab
5-5554200 1299 966e7
5a50 42992o Land)
(292180299b222
Sac7b03e55 12990222222 3eff
Saed (22
558 555oefco (255ecelsseeeee
5ue6 (bb
5.72000
5a8eac 555521999999997777777
5ab9255te554225 11 ca422/f
Sa
Sa5 (199117718110
Su888afa55 9297755c19M400
Sabeed7755 1 1299 SfeS 11 dal
da
85dd288855 22922eeeeccBbb
Sebff 00055--2992220 1990022118552
Sa5bab 1655 2992 c207
52b 62
5abba27555 1299660224
5ab 92
Sabati 25542990555222 930f
45b5549999299 30000111
Su533id 12999) P220ddd
Sac 15 e2845e
Sac3dd2255 12995 Iacld.
5575009555
Sue60381 087755
SahSdd (alosla2555122//5555055 12c573
Sab lb 155 1299699 7a616b. 16b,
5a77bd5455 1299558881111 291,
5a7524ca55 d252
54 29944bb Se5//// d70.
Saellfa8b
5x7336 229 19
5088560853 dfc
Sadce28855 (590050227555
Sac491eb X 94429666hb
5a7 4 429944777774477
5a/2b 42991 19a 200
5 28393/2859433 299 118366 2e6.
Sac219d/5$ F2b 37Ec,
Sa8a8477 011777555549995564488
Sabd catss 291 302266aaab
Sacil aacd77558ss 429969 d899ce00
Saße 11855 12990888
5a79e0 155-299 (99 9et0077
Sael 55 1299 afe
5a800 www1:5555555/555//3b
Sa877& 423777663224299444444444
Sade1 30 3055542299 4299 18-466 ib,
Sac219d/ (ttp///////2222/2 4225422 37Cc,
992555 429039 1541362d
Su/2bic25 4 23962396611111 12,
Sabc3 199776d 48 Maaa.
5a877de355 1589
Sab7 255 12991 1002 Sacf
5 12755 655 1 118118
Sa 5225022555 559
Sadc318<554222
Sa7bed 77775554299222 4bSeet
4558 34a884411 1,
Sadef28c38129966 Sade1 33 22294
Sa/e 5555555555 4299 1959 1996.
Sadbe 27511//99777777777888
5a83 118316880 (48001142266464448
Sa 2828042999 6if 4c736x.
Sa 3 3-594499 188830d119
Sabbbde1 3554294 44c7///
Sadd/ d00s 22590ddb2277esl.
5ah344a 425495 36
Sa8a 3 42999221 fd.
Sab3dad Slccc,.
Sa (59 dell60,
5 1555 299
do 1255555266dd55555555
Sac 439391 31 66484aac
Sac31 42999999971 di 248,200,
fax 49920
Jab-5 228-66227779227777777777
Sabdisuf1 450445560022246666
Sa/2b26655 1 19220e55f
Sabbs55 2999 d4ad 1190
5a/E9d 225.d555554422222 2299 2229cc
Sacll6 1 15384429960000 1810e.
S 5542995 tob
des 1299 2ww.eeeebbcoooooo
5ab6e856655 12997 dlc..
Ba7a 489854299907 3224cc.
Sactl6 abl 299999999999999 Mfac.
Sac Sada 22599200 de eelf,
Sab 4855-1299922. Zaa 34a2ff,
5h77h 3 4298942992944 54aa0))
5 1745 299 Th 49th5
Fed 2990 22222
5v904, 72504 1h51 18,
Sadd99 22 42997 24 35bbd88,
Sade 4299 dh.
Sadida 4299 (2d4
Sab 39701 244292239944448
Salbb (42959995545095555550
4925,
Sa7laSac5542944 dididdedd
5584322255 429432226344299668862222 22000
Sadd 6554229975444 7ib
5a80d 3066642998864 421
5a7 259275995 19220
5a81 iftt aid 24c442999660bbbeec
Sac775 666 2422 226656
Sa7 2994 18 9effbe?
Sao4k 14 da
sabbbdotss422e dhal 44d5,
Sa
2290 S2d5/ba/2
Sa7bbade5 4 2299 Atlaaaa,,
Sa8385 4499 2008
5ab 514 9542991 162470
447344220000 2000 1243,77
2227 1554 4554299 2429288 2239522,
Saleee 2 999.4469999994111 6144774
Jaboeess65 4299/5552509999664 S0ade
Sa7799 4299969 9iee ldd8,
ha77bd5at 54 2000 idics 291,
5a Bedd 5o79 44891 19a6,
Sae 15 153542999111 2006
1 0n5 220.0ddd /t/////////// /h////////
Sah fadd S42399U I d223 fed.
Sab5 2522200555522222555222
Sac 16886 429944550772 4.
5a7eedos 354299999999999 (76699
Sab (1 522237
Bac edd1ed244222dddoo
5ab979da 9342999 434233,
Sab34 db 20 99
222
Sa88loar 520
Sae 411.440555922922228838
39701 56429923 3u 5454,
5add1992 (eddd19926554299227 755444777---/7
d4ad 1232.
5a88b 199442299 (597777
Sad 30t 2142290066 IT
fa885 344 22942992244233 31 2ad
Sadi did 229 29999
Sac Solle 342299666 4344477.
5ab55205 34259 34229 Maaalla,
5a74f ff 62996/ befaadd,
5ab-05433555
Bae 31 5849997 423955..
5ab67 454799947777
Sael 3652 400 De 7nd 166,
5a7e 499 6Eth,
Sade 22289542997 aeddded
5a7607 af 877334999 1991 7 766e la,
S488660 to 7h foo Iiff1
3ac17 1ccc17/41554400 25542294444 Seb 220,
Sa/lta 3as.5 dedddee
Saln 3345555555555 24455-229988 299666 2624
ddccddddeff chd 5- 4299 219 22222229
5al5354355544 429 1419999
Su9044 72554 9993 (K51
5a87% 945
Sueth 10
39077 710,
Sabddde225 247744
Bacd 2255442222255 ae Se Belda,
Bae1 aa67 7554 29907 ffddadddd
3ob91 1 42991 bood
756655665544 22266665222977242422292
Sue31 61 299299286 423055,
Sabtbb-4555422 75:22977777764444
S066377 557795544777755 (9.577
5480999225 27669988
5a74 47 4888542299990 dodd
Sae 39n1 I e5542999 777 20009
TabblcT T7 2000 7
Sae 915225 42994 2e
Sael 77 4299641 a he
5o843222 4 488 te Hod,
5a7738d et
Sa 7770666557299999 0299544497666633 (90e,,
5w77 38 7792 77
S 255422000 1980 509
5w82 Hn 554 2090 d233 ISc.
Sw90e2hh 1544499951151511 di,
Sw7 4445549991 4891 19a6,
5u84544 8554299 2e 23he.
SueSt 21444544429994444 2 add.
Sac 1688663442994
14,
Sabebr7 38 4 1299 9e 15ac1799
5a77a ft700 4299 Zade 186.
Sab 997011 1. 2342292333995 233954
Sac21 Nw.219915542999111 37hc,
778657
13,
54546141555 255544991158881
548799 (309777
Su 1894 261 edd
Sa7aild4 279999977 3024 13,
5 (1
Sue5365d554 2294266666666
Su7Jebbda 2901 0200
Sabifbb 200.555555997766559550 7ofdd 660.
5 55566633555522770o3355 200
5ab-9121555428 299 432
Swfdffbet Seeeefffee555442000 906
Sa8a 28992 B679.
5oc31 w1.. 203,
Sae Ellil 244.642222222224444444444
Sadd 22006 F (0227700
5ac64 42299922 162c
Sade 3.4 20
Sad 2255 2299 Su22640
Sae 2517555654
5.8u847755 477777559966669999999 9519
Suc377a 2994444999444494449449999
Sae 31 ank 4844449900 429928 42395ff
Sadi3ebdd oudd33edd5559999 5079959977999709000 81d8.
Suc77a31 34255599999555
Sac2ad 255 290657
5ab552 22d7
Sabd (6855 22244499922222229
Sae1 3516555529225555333355
ed,
5a7738d 333300011
5a7a52744 0227455549966 259 2dld4,
Su 120.555
Sae5 1 (2885522996 Ib2a.
5x7336u005449991
Sukcdec555442999
14d221
163
Sar2aba115442966 15429691155599644444444132
Salth 242273335200000422222200
Sae Bucla 24w.5599994/77/1ecoo
9c.
Sae5 129.255222222
5ubbc
Sabx
2atb///675544299.bbb0 (995449999/000009 96a1)
55429954 166da
29

---

Published as a conference paper at COLM 2025
MuSiQue Example Ids
1opp..76129944553
Shop_764155.
3hup1_444518. 136629
2hop 1hop_355084 36340.
2hop.62755.33118//
2600_760900.1110
Shop1 3hop1 57,522 shop__266790 276122
2hop. 62031.
153p_.13881..6900
Shopl. 213191 782813
2hop
2741, 2hop 3880 908, 2hop 317335 36735, 2hop 14393 14393387777
87372.
dbap1
149.__20082..41111130 45335 2hop 14tp__144444.000
2/22
2hop 2577 7211, Shopt 12999/251111 517267 451901, 2hop. 46917.77762
2hmp.22002_56000 Moop2_8351 2212222222222
86231 21190..8499,,1 19700, 3hopl. 228008 35132995303.
2hop_661861
Shup1, 66228333366666 0.2283350772 351187 2hop__ 48222
3hopl 858308 1001 84004
2hop--565717 17336..
22op_119389
5365, 2hop_
2hop 2hop- 20857 20779,
2hop__770_91. 2hop_2375619
80178
211361
10sp_101100114000
2hup...66545.8882)
51890_
131886, thop1 28235.71795
84660 15312,
2hop.. 101S4) 92821,
2hop..445544412000
2hop_46764
2hop..24221..111
2hop. 126102.
2hop 199777755 541620,
Shapl 18619 2612 581618, 2hop 87287 8396, dhopl 411538 805015 175503 32631, 2hop 658198 72962, 2hop 12307 122207.
Zhop 308.. 558w9, Shop1 8373 870224 shop2 38725344877 83643, 2hop 879 Shop1 161000 6395099990000 644660
Shap._562231. 52667, 2hop...241199 1944, Shop1..82102 82122222222 74047 755423, Sh1p_5 NA15, 25791 257981 29442299722111 611044.
2hop 370765 1901 2hop 16352 1488, Zhop 81057 13292, 2hop 681911 303901, 2hop
69926, 2hop 858097 121880,
4hap2. 221.7224444444001
431977
thop_49288 2088,
shopt 405751 4520.63877 49736,
2hop 1 128610 126060
Shop1. 35551 7883888999999999
2htp_.31130 36335 2hop_ 43766 2hop_..1664e 090423
2hop 22533211122
2hop 856157 495, 2hop 129231 330515 2hop 15671 12367. 161p1.166916 81298 53/11,
2hop 48999 83539, "
2hop_ 2036,
396362 131877, 2hop_ 136618,22266,
2hop 2hop-199836 198399966666 185893,
2hop 9000__9005500000000 57555,
shapl 31942 4860 15069 Zhop 35105 16087. 2hop 128804 351187, 2hop 153004 86887
2hop 715355 565667. (
Zhop 40484 135138, Shop 52622 67783, Shop 713501 58946, 2hop 30776 99199, zhop 5430 5348, Shop2 29467 1 132027 73894 (
22 2wt.22222222281222222 23818 48069, 2hop 367077 8017 2hop_ 343473 5324, zhop. 848923 16214, shop1 16001 26072 227.222271111 16167%
2hop 250015 61214, Shop1 101311 83380 61459, 2hop 1835 822987. Shop1 836616 291186 1303, 2hop 531924 1894,
Zhop 2h18_.131831 84128, 2hop 328708 90697, 2hop 79691 82816, 2hop 8083s 2001, 2hop 1 196285 61424, 2hop I 180864 42336, (
Shop1,
3hop1, (400199944555.11 159613, Shopl .,525
25525,22225 52355 55099, Shopl, 58622, Thep 250913 58115, 2hop 131095 85298, 2hop 149377 8600, Shop2 625639 25582 21116, Shop2 30023 63995 53125,
6989, 2hop 58284
2hop LA0._9889999000
58487 88978, 2hop.. 116433 351162 zhop s26213 62031, 2hop 80036 909, 2hop- 62996 29042 zhop 236731 229413, (
2hop 15169 87001, 2hop_143790 143791 75878, zhop 658198 50536, 2hop 70m 15755, shop 131105 48117, 2hop 145162 438686,
Zhop 2077 68517, 2hop 65149 46180, 2hop 251426 88653, Shoot 233083 403313 61770 2hop 28291 709757.
thep. 391909 3430, shop1 266733 291186 2094, Thop 205655 180377 zhop 2414.344141 70246, shop1 383692 434040 59381,
26673.
2hop 240975 736888 2hop
368521, shop1,
590099 76243, 2hop 109244 62766, shop1 16401 4520 65397 52251,
zhop 140991 256194, 2hop 104757 74309, Zhop 194976 55966, 2hop 361127 140822,
T 108774 104782 14771, (
dhopa, 393666 12866662001100 (20110 (116116,,227771
Thep 54753 729624, zhop 693650 61232, shopi 89787 49253 632017 4hop1 104663 221189 833580 61459, Zhop 664573 MASA,
2241__224128__ 224128888884
shop1 pooop__89880 849336, 301867 127418,
2hop...4408 2408855411111 5416300
Shopl 702271 823374 26254, Zhop 129892 62851, Shop1 6991 25 30400 23332, Zhop 222162 386543, Zhop 446009 1
2hop_ 781411111 788888888888 Shop1_ 20613 20196 10585, 2hop_ 809948, 162428, Shap1 438612 581261 369731, Zhop 524082 1
shap1 388966 s08864 145463. 2hop 582169 370460, 2hop 225632 52135, 2hop 302691 81463,
22133,
2hop 196889 52356.
thep 81363 42667,
Shop1 599990 544161 92922, Zhop 504710 515189,
zhop 149939 11443,
2hop 521853
zhop 20133 8503,,
Shop_ 145110 4hop1 15118 31258 43153 32993,
861627 2hop 149891 44369, 2hop__ 376266 37939, Shop2_ Zhop 129782 51 517267.
10879,37094,111 161139, (
Shop2 199915 8509 1970,
Shop1 522518 132413 16066,
Shopt 252998 715836 26008
4hop1 201937 144938 83779 44678,
2hop 131318 4244 47463,
2hop 358406 68172,
3153 3356 11988 24628, Thop 106465 54200 Shop 997761 4N718, 4hop1_ 632232 166964 68S 6891, 2hop__121872 208662. 3
Zhop 73501 31113, Zhop 378511 191233, Shopl, 8504 96305 2507, Shop1 755950 99270R 78102,
Zhop 811421, 377891,
63999, 391767 53125, 2hop... 31380 shop1 1 188678 48408, 3778, Shopl shopl__.331 L,7312 231622306222222 68600 2h0p__.012122 21002,
shop1_10/25 693397 74345,
2hop. 445228 778811,
shap1 603090 s18753. 783943 26110,
shop_17718 646648,
shopt 801682 192919 16121, zhop 243908 50443, shop2 89818 157714 4107, 2hop 167546 26427, Zhop 128772 745471,
2hop 62586888887770 20779,
2hop_661616,
20 89066, 2hop_ $68188,19194
Shopl_
153357 411195 A682. (
Thop Sto4_ 9899,,
2hop 647590 134798
Shop2 30786 804098 24137, 2hop 146227 42308, 2hop 152881 620955.
Zhop 11693 42892,
2hop 753498 7606,
Zhop 2745 2741,
Shopl 573317 533132 1660,
2hop 229374 333904,
shop1_ 370220 31187
Shopt choop__11220,pp01 211230 83884,
zhop 2hop__10044.0001 18117
shopt 25122
534118 41330, 1
Zhop 69149 58376,
2hop 182310 365529
Shopl.. 136209 84467 59676
2hop_ 454055 86874
Zhop 604878 40786, (
Zhop 2hop_300769_ 1196119796_11111,,
#hop_s84002_1190
2hop_ 198557
L. 506157
44678,
2hop_ 468848, 44537.
2hop_2n7777 126101,
shop2 53235, 18485 57802 57802,311666,
2hop2_
2hop_ 148822
Shop1 37692 84288 133741.
Shopl. 6721199 196807
760519, Shop2. 131210 661360
54023, shop_ 8531 24846, Shop? 77866 64137
Zhop 730762 8600,
2hop_330033__4v00
2hop_ 131112_33511,
shop1_ 157334 81660,
2hop_ 677777
677577
2hop_ 271999077770
2hop_66755 682__6675__80000, Shopl..
503050 76293, Shop2 162189 61 1045, 73761, Zhop
2hop_45622 45625 21952
2hop_.146200 146207 33651,
2749_
Zhop 57499 78714, Zhop 3736
52135, shop! 501828 348668
Shop1
35178 686699, 106423
Shop__103209_23144
3hop1_27985,6 J618).
Zhop_ 157740
2hop_ 106,_10615,,0044
40 40174,
shopl_39844161546 16532,
2hop.. 581._5885212121
755188,
2hup 80508, 40874,
2hop 2hop_68t2 684137 58115,
2hop
Shop1__
3hop1__29555_0000
113255300077460000 246600 2hop__144944_66 6-900,
2hop_ 1200___6677_80000 482901, 4hops
4hop3__9198 9198 (809 19770, 2hop_ 2 J4008,
2hop_71269._1 36735,
1oop__00228_666 64699
6047888488,11867
2hop 2nop_s1s72_00sa1
Shopp
Shop1_664688,00
2hop_11576661865
30


Reviews:
[{"review_id": "tGXOxOqt0Z", "reviewer": "Reviewer_WJVa", "summary": "* This paper proposes to apply multi-step reinforcement learning to improve LLMs' reasoning and tool use. While multi-step RL has already been explored in LLMs, the contribution of this paper should be considered as the introduction of a teacher LLM as a process reward model (PRM) to provide reward in the tool use area.\n\n* The writing and clarity of this paper can be further improved. Some key contributions and novelties in the paper are not easy to capture.\n\n* The experiment results are generally good, demonstrating the effectiveness of the proposed method in Gemma/ Gemini. One concern should be the limited number of teacher/ student models used in the experiment, which calls into question the generalization and robustness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper explores the effect of multi-step RL in tool use, which is an important area that should benefit from RL training intuitively.\n\n* The experiment results are good. The authors demonstrate that their method improves the base model's performance significantly and outperforms vanilla SFT by a considerable margin. Additionally, some insights they provide about the selection of rejection sampling and reward methods can also benefit future works in that area.", "reasons_to_reject": "* This work appears somewhat incremental and engineering-oriented. The authors integrate several existing techniques—multi-step reinforcement learning, data synthesis with rejection sampling, and the use of LLMs as process reward models—and apply them to the task of tool use.\n\n* The experimental evaluation is limited to a single student-teacher model pair, which weakens the generalizability of the conclusions and findings.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "44PJO2Es9m", "reviewer": "Reviewer_4922", "summary": "The authors propose a simple and effective RL method that enables LLMs to make decisions during CoT reasoning and tool use, aiming to enhance the multi-step reasoning abilities of LLM agents. Specifically, the synthetic rollouts are generated from a base LLM interacting with tools. Then the rollout receives an outcome reward based on the final result and a process reward generated by a stronger LLM. These rewards are then used to update the model. Experiments demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method effectively demonstrates the usefulness of PRM in LLM agents’ tool use. The idea of learning from imperfect but informative trajectories is also insightful for future work.\n\n2. The method is evaluated on multiple tasks and shows good OOD generalization.\n\n3. The performance improvement over the base model and SFT is substantial.", "reasons_to_reject": "1. A potential limitation is that the process reward relies entirely on a single strong model (Gemini 1.5 Pro Thinking) for scoring. Although this model is strong enough to support the current experiments, it narrows the conclusion. In broader task settings, particularly those involving harder problems or less generalizable reward models, the accuracy of process evaluation might not be reliable, potentially compromising RL optimization. As a result, the stability of the SWiRL method still largely depends on the quality and selection of the reward model, and the conclusion may not be stable for other tasks or reward models.\n\n2. In addition, the current tool selection is still relatively over-simplistic, which may make the reward estimation easier. For example, if multiple search tools were involved, evaluating the process reward would become more challenging as it needs to check which searching tools are appropriate for this certainty question.", "questions_to_authors": "1. Regarding the generalization across tasks, has the author analyzed the reason for this OOD improvement?", "extra_scores": {}}, {"review_id": "zgAoqqgcrv", "reviewer": "Reviewer_SiLC", "summary": "### First, I want to mention that I am not familar with both RL and Tool use in LLMs. Sorry about that. If I make some errors, just let me know. Thank you for AC and authors.\n\nThe paper introduces Step-Wise Reinforcement Learning (SWiRL), a novel approach designed to enhance large language models in performing complex, multi-step reasoning tasks that involve tool usage.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The decomposition of trajectories into sub-trajectories for step-wise optimization is a novel approach that allows for granular learning and feedback.\n\n- By relying on synthetic data and model-based evaluations, SWiRL reduces the need for costly human annotations, making it a scalable solution.", "reasons_to_reject": "- The paper does not provide a comprehensive comparison between SWiRL and other contemporary approaches to multi-step reasoning and tool use in language models. Including such comparisons would offer clearer insights into SWiRL's relative advantages and limitations.\n\n- SWiRL relies heavily on synthetic data generated by language models and filtered through model-based evaluations. This approach may introduce biases or inaccuracies inherent in the models themselves, potentially leading to suboptimal learning outcomes. The absence of human oversight in data generation and filtering raises concerns about the reliability of the training data.\n\n- I guess I miss something, but I didn't find the model used in generated training data. What model did you use, and is there the results to show whether this method can be used in self-improve?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5moegecQ06", "reviewer": "Reviewer_SNTP", "summary": "This paper presents SWiRL, a multi-step optimization technique. SWiRL consists of two stages. The first stage involves generating multi-step synthetic data. The second stage is the learning stage: learning from this data in a step-wise reinforcement learning style.\n\nThe first stage is the generation of multi-step synthetic trajectories (s1,a1,…,sk,ak): a sequence of states and actions where the first state is the prompt to a language model, subsequent states build context from the previous state, action, and environment response—actions are the model's responses to the current state—and the last action ak is declared as the (final) answer to the original prompt. The model has the option at each step to either invoke a tool (e.g. a calculator) or give a final response, and it usually produces chains of thought. When the model generates a tool use call, the system parses it, executes it in the environment, and presents the result to the model in the next step. In training, based on the preceding context, the base model is optimized at each step to predict the next step or the final response.\n\nThe paper presents four filtering approaches: none, outcome-based, process-based (model-judged steps), and combined process and outcome-based. There are several multi-hop question-answering and mathematical reasoning datasets used in experiments.\nWhile some recent approaches filter synthetic data based on correct final answers, the authors argue that SWiRL benefits from training on the data that is not filtered for the correctness of final answers. This boosts the accuracy of LLMs on average by 15 percent. Moreover, the authors suggest that filtering for correctness usually impairs performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has an interesting topic that should definitely appeal to computational linguistics: boosting the performance of a language model on reasoning tasks without using human annotated labeled data but synthetic one.\nThe approach is intriguing because it's different from how synthetic data has been used in recent methods (the authors argue that filtering for correct final answers harms the performance).\nIt is a well written paper.", "reasons_to_reject": "While I believe that this paper should not be rejected, to me it seems somewhat necessary to have a more detailed analysis of the obtained results. Maybe trying it out on a few more datasets could help the authors to understand why the unfiltered data seems to work better than the filtered one.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents SWiRL (Step-Wise Reinforcement Learning), a method for improving multi-step reasoning and tool use in large language models through synthetic data generation and reinforcement learning optimization. The paper demonstrates clear technical contributions in the step-wise breakdown of multi-step trajectories into single-step RL optimizations that enables granular learning and better reward attribution, while reducing reliance on human annotations through synthetic data generation. The empirical results show substantial improvements across multiple benchmarks, including GSM8k and HotPotQA, with notable cross-task generalization. This confirms the potential of the method for reasoning tasks. While the tool environments are relatively simple, they are sufficient to demonstrate that the approach is workable.
Decision: Accept