Behavior Injection: Preparing Language Models for Reinforcement Learning
Authors: Zhepeng Cen
Yihang Yao
William Han
Zuxin Liu
Ding Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Large Language Model
Data augmantation
Primary area: deep_learning
TL;DR: We analyze what kind of LLMs have large improvement in RL finetuning and propose behavior injection augmentation to prepare the LLMs for RL.

Abstract: Reinforcement learning (RL) has emerged as a powerful post-training technique to incentivize the reasoning ability of large language models (LLMs). However, LLMs can respond very inconsistently to RL finetuning: some show substantial performance gains, while others plateau or even degrade. To understand this divergence, we analyze the per-step influence of the RL objective and identify two key conditions for effective post-training: (1) RL-informative rollout accuracy, and (2) strong data co-influence, which quantifies how much the training data affects performance on other samples. Guided by these insights, we propose behavior injection, a task-agnostic data augmentation scheme applied prior to RL. Behavior injection enriches the supervised finetuning (SFT) data by seeding exploratory and exploitative behaviors, effectively making the model more RL-ready. We evaluate our method across two reasoning benchmarks with multiple base models. The results demonstrate that our theoretically motivated augmentation can significantly increase the performance gain from RL over the pre-RL model.

Behavior Injection: Preparing Language Models for Reinforcement Learning
Zhepeng Cen"1, Yihang Yao"!, William Han', Zuxin Liu", Ding Zhao'
Carnegie Mellon University, Salesforce Al Research
Equal contribution, {zcen, yihangyaJ@@ndrre cmu. edu
Abstract
Reinforcement learning (RL) has emerged as powerful post-training technique incentivize the reasoning ability of large language models (LLMs). However, LLMs can respond very inconsistently to RL finetuning: some show substantial performance gains, while others plateau even degrade. To understand this divergence, we analyze the per-step influence of the RL objective and identify two key conditions for effective post-training: (1) RL-informative rollout accuracy, and (2) strong data co-influence, which quantifies how much the training data affects performance on other samples. Guided by these insights, we propose behavior injection, a task-agnostic data augmentation scheme applied prior to RL. Behavior injection enriches the supervised finetuning (SFT) data by seeding exploratory and exploitative behaviors, effectively making the model more RL- ready. We evaluate our method across two reasoning benchmarks with multiple base models. The results demonstrate that our theoretically motivated augmentation can significantly increase the performance gain from RL over the pre-RL model. Website: https://bbidge:: www.-reaooomnn reasoning. git.ub...bbi. io/
Introduction
Large language models (LLMs) have demonstrated remarkable reasoning capabilities and strong performance across broad range of tasks I including mathematical problem solving 4 5 code generation 7 g and embodied decision-makiin
[0] When guided by chain-of-thought LLMs are able to generate intermediate reasoning steps, leading to more
(CoT) prompting structured and interpretable outputs. Despite these advances, LLMs still struggle with complex, multi-
step reasoning tasks, such as mathematical competitions [12 and real-world agentic scenarios [33
15 A common strategy for improving performance is to scale up training corpora [16,
17
18 19
However, concerns have been raised that high-quality pretraining data may soon exhausted 20 and continued data scaling introduces significant computational overhead.
alternative and increasingly prominent direction is enhance LLM reasoning through reinforce- ment learning (RL) [21] 22), which enables reward-driven fine-tuning based on verifiable outcomes, ranging from ground- -ruth correctness 23
24
to feedback from executable environments 25 14 15]
In traditional low low-dimensional RL, performance gains are typically attributed to algorithmic improve- 30), with little attention paid to initialization since models are often trained from scratch. In contrast, the RL process for LLMs begins with pretrained or fine-tuned model, and we argue that the quality of this initialization, particularly after supervised fine-tuning (SFT), plays a critical role in determining the effectiveness of subsequent RL. In this work, we focus on the warm-start RL training pipeline (23), which first fine-tunes an LLM via SFT before applying reinforeement learning. Our goal is to make language models RL-ready: rather than modifying RL algorithms themselves, we propose a data-centric strategy to enhance the base model's ability to benefit from RL, thereby improving training efficiency and downstream performance.
ments [26] 27 and data quality [28
29
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Supervised
Original dataset
Behavior-scand scarce
on the IG5M cotaset
Finetuning
SFT model
055
BRIDGE: Behavior Injection Data Augmentation
Base model
Tes Pes-Aocurury
puiloe befavvice
Exploration Behaviors
Qwen
Ixgisitation delurtte
O Meta
Exploitation Behaviors
Bohawler injected model
Behavlonstarce SIT mode
Behavior-injected Augmented dataset
Supervised
Behavior-injected
Step
Finetuning
SFT model
RL finetuning
Figure 1: Overview of the BRIDGE pipeline: We augment the SFT data by introducing exploration and exploitation behaviors to prepare LLMs ready for RL finetuning.
Our key insights stem from two perspectives: (1) Analysis of the RL learning objective: we identify two critical factors that influence model improvement during RL tuning: rollout accuracy and the data co-influence coefficient, which quantifies how strongly RL training data affects generalization to the target domain. (2) Desirable behaviors for RL.: while exploration
exploitation are central in low-dimensional RL [1], they are often underexplored in the context of LLM post-training. Motivated by these findings, we introduce BRIDGE (BehavioR Injection Data auGmEntation), data-centric augmentation strategy applied during the SFT stage. BRIDGE injects desired behaviors into the model before RL, enabling to generate more informative trajectories during RL rollout and leading to greater final performance improvements. Our contributions summarized as:
In-depth analysis of LLM reinforvement learning. We provide a detailed examination of the RL training process, highlighting two key factors that drive learning efficiency: rollout accuracy distribution and the data co-influence coefficient.
2. Introduction the BRIDGE augmentation algorithm. We propose BRIDGE, which prepares the model for RL by explicitly injecting exploration and exploitation behaviors during SFT.
Comprehensive empirical evaluation. We evaluate BRIDGE across diverse tasks from iGSM and PromptBench. Extensive experiments and ablation studies demonstrate that BRIDGE enhances data co-influence and significantly improves performance the RL stage.
2 Related Work
RI--based post-training for LLMs. Reinforcement learning (RL) has become a central post- training approach for aligning and extending large language models. Large-scale efforts such as OpenAl-ol [32) and DeepSeek-RI (33) illustrated the gains obtainable from reward optimization on ceneral-purpose models. Since then, RL fine-tuning has been pushed into a variety of domain- specialized settings. In mathematics, serifier-guied or programmatically graded rewards help models master challenging problems 33 35
37 38), while logic benchmarks likewise benefit from RL-driven reasoning refinement (39 40 Interactive agents leverage RL for textual tool use and multi-
24
36
step planning [1 42
mobile-app control 25 device manipulation 44 and
web navigation
49 Additional applications include medical visual QA (50), software-engineeriig
assistance [1 social reasoning (52), and tool-centric instruction following 53
Analysis of LLM finetuning. To understand how model performance changes during finetuning,
researchers study SFT learning dynamics of LLM [54] terms of data influence (55) 56) likelihood analysis [57 58) In the context of RL finetuning, OpenAI ol [32] showed that RL sigmificantly improve reasoning by encouraging the generation longer COT. Subsequent studies (59) 60 61 62
validate this effect and show that RL enables inference- -etrence--tmeeeee scaling by favoring more expressive reasoning traces. This finding is aligned with theoretical analyses that characterize the expressivity of CoT 63 64
68 69 observe that RL fine-tuning often amplifies behaviors already accessible in the base model rather than introducing entirely new ones, which are crucial cognitive operations to performance growth in RL 70). Distinct from these works, we investigate the tuning dynamics by RI. learning objective, and we identify behaviors from the perspective of exploration and exploitation prepare models for RL tuning.
65 66). Furthermore, researchers [67]

---

Data-centric approaches for LLMs. A complementary line of work improves language models not by changing the training algorithm but by enriching the data they see. supervised fine-tuning (SFT), targeted augmentation has proved especially fruitful: synthetic derivations, curriculum sampling, or structure-presst preserving rewrites boost mathematical reasoning [16] 72
73
and extend code 74 4
generation domains [75]. Beyond manual augmentation, agentic pipelines automate data acquisition, where LLM-based agents write queries, run tools, and self-filter their outputs, yielding high-quality instruction data at large scale [76 [9 77 78 79 (0). For RL settings, several studies curate seed datasets whose answers can be programmatically verified, seeding reward-based training with reliable trajectories [8 [4] Others examine what kinds of pre-RL corpora make base model more amenable to later policy optimization, highlighting the importance of coverage, diversity, and error profiles 662 [70]. Our work aligns with this data-centric perspective but focuses on behavior-level augmentations that explicitly make models RI-ready, rather than merely enlarging or filtering the SFT corpus.
Method
In this section, we first investigate key factors that affect the model performance growth in the RL stage, and then introduce our method based on the analysis to make the pre-RL model "RL-ready", i... being able boost the performance after the RL stage.
3.1 Preliminaries
Following the previous pipeline 23
apply SFT to the base models, followed by an RL stage. Supervised finetuning (SFT). SFT a common practice to initialize LLM finetuning by demon- stration dataset DSFT, which trains the policy wy by minimizing the negative log-likelihood:
min Csrr(D; = -B(aaa) DSFT Σ log no ass q. a (2))
where (n....................... U1. a [ai...., arl are demonstration query and answer from SFT dataset. For simplicity, we use SFT model to refer to the model after SFT training.
RL finetuning. The objective of RL for LLM is to maximize the expectation of reward on query Q:
T Σ 2'v (q (g.0155))
max [Ri(Q; Q Raaaaooooaee ooono (-1q)
where [911................. 9 [01, l,, orl indicate the query and output respectively, r(..)) is reward function, and y is the discount factor. We mainly consider the rulc-based outcome reward, i.c., binary reward on the last token of output:
r(q.o) ( Ygold),
where v(o) is the final answer of the output and Ygoid is the ground-truth answer of the query. In this paper, we adopt GRPO (23] the RL algorithm, which samples N outputs {o; J. for each query and optimizes a surrogate objective:
J(Q: 0) = (a:),(oo)~<o(i
nee (a(l:)
[)(0:)g) Tata (o:lg)
1 e, + A Ar -DoII
(3)
min
A, clip
Rotd (o: ④
where is the clip ratio, 33 is the KI. regularization coefficient, and A, is the advantage computed by normalizing the reward each group A, = (ri mean(
3.2 Training Per-step Influence in RL Finetuning
To understand why LLMS respond to RL. divergently, we leverage per-step influence answer the question how the model performance changes after one RL training step.
55
82 to
3

---

Consider language model policy тe, suppose we update it on one query-outpu group (q. (o)))n) with learning rate 17. then the parameter update is A n since the RL objective to maximize J. According to Taylor expansion, the caused model performance change on other query q' can be written as
AJ('':) = 0 A0) - AJ( (VoJ(q': A0) + O
With a sufficiently small learning rate, the per-step influence of on the model performance is AJ(Q: n (V: V.J(qq
(4)
Then we derive the per-step influence for the GRPO objective as follows.
Proposition 3.1. Suppose there are correct outputs in the sampled group with size N, denote the correct and incorrect outputs as {oi+ Ji-1 and {oك-} \N-" respectively. When RL training on is strictly on policy, with a sufficiently small B, the per-step influence of q on the model performance is (Q,;0) = "1llB'' -..0''
n Σ
N- Σ
1 (5)
A(d',o')
o'll
-
where n/N indicates the accuracy rate for the rollout samples, Kal(q', o'), (q o)) (Va log ne (o' ('), Va log to(o q)) indicates the influence between the log-likelihood of query-output samples (q', o') and (q, 0).
The proof and discussion are in Appendix A.1 The derivation mainly based on two assumptions, the on on-policy training and small KL coefficient 3, both of which hold in practical implementation. Note that this proposition can also be extended to other RL algorithms by replacing the advantage function. More discussion is provided in Appendix A.2
This proposition shows that the per-step influence mainly determined by factors: (1) the accuracy rate of sampled output, a, and (2) the co-influence coefficient Kg between training data (q.o) and target data (q', o'). For the first factor, 0% or 100% rollout accuracy makes the coefficient leading AJ 0, while medium accuracy can amplify the influence, which aligns with previous theoretical results on reward variance {3 and practical observation [84). The second factor, expressed through Ko[ +). quantifies how strongly each training sample affects performance change on the target domain. Take correct target sample (i.e., A(q', o') > 0) for example, we expect a large positive Kcp when the update draws on correct samples 0,+), enabling the language model to learn more from this successful path and gain greater improvement in the RL step. Conversely, when the update uses incorrect samples (q, o,-). we expect Ko for the same correct target sample be small or negative, allowing the model to learn from the failure path. We
Va(1 1 -a)
defer the detailed explanation of the validation method to examine this factor to section 4.3
Section 3.2 takeaway
There are two key strategies enhance performance growth during the RL stage of LLMs:
Altering the accuracy distribution to increase the information coefficient Va(1 a). Shaping the co-influence coefficient of model, governed by K.p, to improve knowledge acquisition from both successful and failed rollout experiences.
3.3 BRIDGE: Behavior Injection Data Augmentation
Based on the above analysis, the next question is how to improve performance gains during RL in practical implementations. A straightforward approach to perform rejection sampling on queries from low-information accuracy regions, i.e., those with excessively high or low However, this method has three key drawbacks: (1) Increased computational cost: it requires pre-sampling to identify and discard www-iformation samples; (2) Distortion of the query distribution: it induces distribution shift issues in optimization; and (3) Limited impact on data co-influence: this approach does little to affect the data co-influence factors involving Ko.
Rather than directly reshaping the RI. training distribution, we focus on the characteristics of the base model, which fundamentally influence both the accuracy distribution and the co-influence

---

structure. In particular, we target two essential capabilities of RL agents: exploration and exploitation. Exploration encourages the model to traverse a broader range of observations and actions, helping it avoid suboptimal local modes. Exploitation enhances the model's i ability to leverage its current knowledge for effective decision-making which can in turn increase the co-influence of training data during RI. By injecting these two behaviors into the base LLMs,, we promote more effective learning dynamics and greater performance gains in RL training.
We implement behavior injection through our proposed method, BRIDGE (BehavioR Injection Data auGm auGm Entation), as illustrated Figure I Starting from a vanilla chain-of of-thought (CoT) dataset for instruction tuning, we augment the dataset by injecting exploration and exploitation behaviors. We then perform supervised fine-tuning (SFT) on the augmented dataset, followed by reinforcement fine-tuning (RL).
Objective
Algorithm BRIDGE
Dependency/
Input:
Vanilla QA pairs (g, a). injection prob p
Reasoning step
Output: Injected QA pairs (q, a')
Exploitation
1: Extract DAG G = (V, E) from (g. a).
Exploration
2: Construct exploration behaviors.
Curated behaviors
3: Obtain a locked node n, ahead.
Ground truth DAG
4: 61 = "Let's solve [ny], .... wait, [ry]] seems to be
not solvable yet, let's get back."
Subgoaling
Reflection
5: Construct exploitation behaviors.
6: Aggregate all the information info solve
unlocked but not solved node na
7: b2 "Let's solve [m,,], .... since [info], [n.,.]
=[subgoal
valuee"
(b) Exploitation behavior (c) Exploration behavior
8: a' +- b; + b2; # Inject with prob p.
9: Return: Injected QA pair (g. a')
Figure 2: DAG representation of behaviors.
To further ground our approach in a practical implementation, we adopt a directed acyclic graph (DAG) representation for reasoning tasks [85). shown in Figure
2 Formally, a reasoning task is represented as G = (V, E), where V is the set of nodes and E S V x V denotes the directed edges that capture dependencies and relationships between nodes. Given initial information and ioder-node dependencies, the goal is to compute intermediate node results step by step and ultimately derive the final answer at the target node. The ground-trutt reasoning chain corresponds to topological of and logical
the DAG. Notably, many reasoning and agentic tasks, such as math reasoning [86] 87 88 reasoning 886
naturally conform to this graph-based representation (77].
Our BRIDGE approach is detailed Algorithm|]] I We first extract the DAG representation from the original query and its corresponding CoT response. Note that such extraction can be achieved by string matching for structured CoT or an oracle LLM for unstructured COT. More discussion is in the generate exploration behaviors by attempting to
Appendix B.2 After constructing the behaviors, solve a locked (i.e., not yet solvable) node ahead of time, followed by reflection. For exploitation behaviors, actively aggregate available information to solve unlocked node (i.e., solvable but not yet solved) or compute a sub-goal instead of directly reaching the node value, reinforcing the known reasoning path. After generating these behaviors, inject them into the vanilla CoT corpus, resulting in an augmented dataset.
Experiment
4.1
Experiment settings
Tasks. To evaluate the performance growth during RL finetuning, we conduct experiments on two benchmarks: 1) iGSM [74], a grade-school math problem benchmark involving math and common sense reasoning tasks; 2) PromptBench [86), a benchmark involving arithematic and logical reasoning tasks. Note that the difficulty of the tasks in both benchmarks are controllable: iGSM controls the difficulty of the query by the number of operations needed to reach the final answer while PromptBench controls it by reasoning depth and number of redundant premises (i.e., the redundant edges in DAG) in query. We adopt the strict match accuracy the evaluation metrics for both tasks.
5

---

In practice, we apply slight modifications on iGSM to improve the finetuning stability, e.g. removing the modulo operation because we observe that the base models are unable to calculate the modulo with high accuracy and it cannot be significantly enhanced by training on a small SFT dataset 74 See Appendix B. for more details of the tasks.
We select these two benchmarks for analysis because we want to test the problem-solving abilities of LLMs rather than the knowledge storage. While other tasks where domain knowledge can confound the evaluation of reasoning capabilities, the purely synthetic data in iGSM and PromptBench avoid the data contamination issue fundamentally 77 89 Although built upon relatively small semantics domains, these tasks require citti-ddmeesional abilities of LLM such as basic concept understanding, search and planning, and basic arithmetic calculation, which mirrors the key aspects of problem- solving in general reasoning tasks.
Injected behaviors. We adopt two exploitative behaviors, subgoal computation and information anal- ysis, and one exploratory behavior, reflection, respectively. Specifically, subgoal computation refers computing intermediate results for complex equations; information analysis involves aggregating relevant information when deriving the equation for a node; reflection means actively attempting to solve unlocked node and then revisiting back. We add the subgoal computation each step of CoT but inject the analysis and reflection with probability iGSM and PromptBench tasks are illustrated in Appendix B.3
0.1. Detailed examples of behaviors
Baselines. We compare BRIDGE with several data augmentation baselines: 1) Vanilla, which uses
original SFT data without augmentation; 2) premise permutation augmentation (PP-Aug) (90 73],
which augments SFT data by randomly shuffling the premises in the query to improve the reasoning consistency; 3) reasoning chain augmentation (RC-Aug) 4 which is implemented by generating new answers with different topological orders for the same query from SFT dataset.
Other experiment settings. We use two families of base models, Qwen-2.5 [9]] and Llama-3.2 [92] to validate the effectiveness of our methods. As there is large domain gap between pretraining corpus and evaluation tasks, first train base LLMs and demonstration answers. Then we use GRPO with the same settings to finetune the SFT models for different augmentation methods. More training details attached in Appendix B.4
SFT dataset to expose them the query set
4.2 Main results
We present the performance comparison on iGSM and PromptBench in Table [nd respectively.
For the iGSM task, dataset consists of 2000 data while PP-Aug and Aug augments dataset size 8000. BRIDGE also uses 2000 data for SFT but augments them by injecting behaviors. We train each model on corresponding dataset for epochs. In the RL stage, we train on the data with the same difficulty. To avoid overfitting (e.g., the memorization in SFT 9 39] and evaluate the generalizability of the finetuned models, we test their performance on two problem sets separately: 1) in-distribution
data with 15 20 operations for finetuning. In SFT stage, the vanilla
Table 1: The evaluation results (%) on the iGSM task. We train SFT models for 5 epochs and finetune them with RL for 100 (Qwen 3B) 200 steps (Qwen 1.5B and Llama IB). We compare SFT and RL models on both in-distribution and out-of-distributin problem sets. A denotes the performance improvement by RL over SFT.
Vanilla
PP-Aug
RC-Aug
BRIDGE (Ours)
Base
In-Dist
OOD
In-Dist
OOD
In-Dist
OOD
In-Dist
OOD
SFT
40.2
29.0
46.6
33.4
33.2
27.4
44.8
34.2
Qwen-1.5B
RL
46.2
36.4
60.4
48.0
46.2
39.6
91.4
83.0
7.4
13.8
14.6
13.0
12.2
46.6
48.8
SFT
38.0
28.4
38.0
19.8
45.6
32.8
59.2
45.8
Qwen-3B
RL
57.2
47.4
62.4
52.0
60.6
45.6
89.6
83.6
A
19.2
19.0
24.4
32.2
15.0
12.8
30.4
37.8
SFT
29.6
19.0
32.4
23.2
31.0
21.4
40.4
26.6
Llama-1B BB
RI.
31.0
24.0
39.0
26.4
33.6
23.4
64.6
44.8
A
1.4
5.0
6.6
3.2
2.6
2.0
24.2
18.2

---

Table 2: The evaluation results (%) on the PromptBench arithmetic reasoning task. We train SFT models for 2 epochs and finetune them with RL for 100 steps. We compare performances on both in-distribution and OOD problem sets, denotes the performance improvement by RI. over SFT. Bold means the best performance.
Vanilla
PP-Aug
RC-Aug
BRIDGE (Ours)
Base
In-Dist
OOD
In-Dist
OOD
In-Dist
OOD
In-
OOD
SFT
13.6
9.4
16.8
8.8
18.6
13.0
2.2
0.8
Qwen-1.5B
RL
37.8
29.0
41.2
31.2
34.2
21.6
55.2
41.0
A
24.2
19.6
24.4
22.4
15.6
8.6
53.0
40.2
SFT
31.2
18.0
40.6
29.4
35.2
19.4
44.4
31.0
Qwen-3B
RI.
50.0
34.6
66.0
50.8
64.0
43.4
85.8
70.0
A 18.8
16.6
25.4
21.4
28.8
24.0
41.4
39.0
SFT
12.6
5.4
11.2
8.8
9.2
4.4
6.0
3.6
Llama-IB
RL
27.8
18.8
28.6
19.0
25.2
17.4
47.6
39.8
15.2
13.4
17.4
10.2
16.0
13.0
41.6
36.2
(In-Dist) set with operation number = 20 and 2) out-of-distribution (OOD) set with operation number 25. Each set consists of 500 problems. We use greedy sampling and compute the accuracy when evaluating on the test sets.
For the PromptBench task, respectively, both of which have 0 ~ 8 redundant premises. This simulates the real-world settings that we have labeled CoT answers for easy tasks and only have verifiers for harder tasks where the demonstration annotation is absent. In SFT stage, we train the models on 5000 data (PP-Aug and Aug augment 10000) for 2 epochs. RL stage, we train the models for 100 RL steps. We also test the performances on I) in-distribution set with reasoning depth = 5 and 0 ~~ 8 redundancy redundancy separately.
data with reasoning depth 4 and for SFT and RL training
and 2) out-of-distribution set with reasoning depth and 20
In iGSM task, most augmentation methods with different base models increase the SFT accuracy on in-distribution set with models while the improvement OOD set is less prominent. For RL, BRIDGE achieves both the highest performance growth and best accuracy on in-distribution and OOD) sets. In PromptBench task, although BRIDGE has relatively lower pre-RI. accuracy, obtains significantly more remarkable performance growth than baselines during RI. and thus achieves the best final score both in-distribution and OOD sets. The training curves and more training results in Appendix B.5 where we also present result of comparison between BRIDGE and accuracy-based rejection sampling.
4.3 Rollout accuracy and Co-influence of Different SFT models
To study how different augmentations lead to divergent RL performances, we test the rollout accuracy distribution and the finetuning per-step influence AJ of the SFT models on iGSM task. The results are illustrated in Fig
Specifically, we use the SFT model to rollout 8 samples on iGSM test set consisting of 2000 data with the same difficulty (operation number 15 20) as RI. training set. We compute the advantages of each answer (by normalizing the reward within the group) and the sampling accuracy distribution of each model based on the rollout results. When measuring the data co-influence Keg, the directly computation requires taking the inner product between two gradients with the same size as the model parameters, which intractable to scale to a large dataset. Instead, we adopt a low-rank approximation [82 that first estimates the LoRA [94] gradients of SFT model and then applies random projection to further reduce the dimensionality. Based on the advantages and the approximated data co-influence K, we can compute the approximated per-step influence of one query-output group on other data AJ by Eq. 5). More implementation details are in Appendix B.7
We present the sampling accuracy, approximated per-step influence and training curves of different SFT models in F In both settings, BRIDGE which can be explained in terms of both sampling accuracy and data co-influence in finetuning. All of three augmentations increase the accuracy SFT model, but RC- Aug achieves mainly by more fully correct samples (acc=1). On the contrary, BRIDGE has the largest ratio of samples with
fastest reward growth during the training,

---

Vanilla
PP-Aug
RC-Aug
BRIDGE
Qwen2.5-1.58
0.6
Influrccea
0.035
0.8
0.0.0
0.7
0.4
0.025
Thinnng
0.015
0.4
0.2
0.010
0.3
0.005
50
160
150
200
0.0
9CC-0
0-33c--1
acc-1
0/8
1/8
2/8
3/8 48
5/8
6/8
7/8
8/
Stop
Samp accuracy
Accuracy group
Llama3.2-11
0.07
0.7
0.06
0.6
0.4
0.05
0.5
Erequercc
0.04
0.3
0.03
0.4
0.02
0.1
50
100
150
200
0.0
300+0
0<cace<1
accml
0/9 18
2/8
3/8
4/
5/8
6/8
7/8 8/8
Stop
Samp acturacy
Accuracy group
Figure 3: Left: Training curve, Middle: SFT model rollout accuracy distribution, Right: Per-step influence visualization. The top and bottom plots correspond to the results of Qwen2.5-1B and Llama3.2-11 1ii.3.2-1BB 1B respectively. We group the approximated per-step influence for samples with different accuracy in right plots, where the influences of samples with all correct or all wrong answers
medium accuracy (0 <acc< 1) that contribute to model improving during RL training, even larger than PP-Aug which has better performance at the beginning of RL. Meanwhile, the right plots clearly show the discrepancy among different methods: while PP-Aug and RC-Aug models have similar per-step influence with vanilla SFT model, method significantly improves it by proper behavior injection into the SFT data, and thus better prepares the model for RI.
Section 4.2 and 4.3 takeaway
BRIDGE demonstrates superior capability to boost RL performance (Table 2 with:
Injected behaviors improve the per-step influence (T(Q; (Q:6) 0) in Eq. (Figure Right).
Injected behaviors shape the accuracy distribution of rollout samples (Figure Middle).
4.4 Ablations of Injected Behaviors
As add various behaviors the demonstration dataset to prepare the model for RL, it remains unclear how each behavior contributes to the final improvement. Therefore, we run an ablation by adding only one behavior and present the results in Figure
For Qwen2.5-1.5B models, the subgoal computation increases the SFT model accuracy and three behaviors improve the performance gain in RI, consistent with their per-step influences. For Llama3.2 2- IB, the models with single behavior still outperform the bascline but the performance gain in RL is marginal. Meanwhile, trained by BRIDGE and model with any single behavior after RL finetuning, suggesting the necessity of behavior combination for the Llama model.
observe that there is a large accuracy gap between the model
We also present ablation studies on the injection probability of behaviors for the iGSM task in F Comparing consistently enhances the final performance. Furthermore, performance differences across various non-zero injection probabilities are nuanced. This suggests that the exact behavior ratio is not critical, and L.MMs can adaptively learn utilize these behaviors during RI. long as the behavior injected into the model.
with > 0, we observe that the injected reflection and analysis behaviors

---

Vanilla
Vanilla + subgoal compute
Vanilla reflection
Vanilia + analysis
BRIDGE (all behaviors)
0.035
0.
0.04
0.039
Valldaton Accurats
0.925
0.000
0.020
0.5
0.008
Vanceeron
Par-pee
0.006
0.044
0.-
0.022
0.005
02
0.000
0.1
0.000
SPT models with different behaviors
SPT models with cifferent behavirrs
IG5M-qwen-1.5B
IGS M-LLa-Laaa
Figure 4: The ablation of models with different behaviors iGSM task. We present the validation accuracy curves as the RL finetuning performances, where the validation consists of 500 queries with 21 ~ 25 operations. We also compare the average per-step influence of the SFT models with different behaviors.
Injection probability
Perplexity Distribution
Entropy Distribution
0.9
1.30
Validtton Bccuracy
1.25
Perplexity
Entropy
0.10
0.6
0.5
p
1.15
0.2
0.3
1.10
0.00
30
100
150
200
Vanilia
Reflection Subgoal
Analysis
Vanilla
Reflection
Subgoal
Analysis
Training stop
Figure 5: The ablations behavior injection probability.
Figure 6: The perplexity and entropy SFT models.
4.5 Exploration and Exploitation Effects Injected Behaviors
In this section, we study the effects of behaviors exploration and exploitation. We leverage policy entropy and perplexity as the corresponding metrics: we first train the model on datasets with the same size for the same training steps. Then rollout the SFT models with temperature same as rollout in RL finetuning) and compute the policy entropy on model's generation; we also use the log- likellioooo on corresponding SFT dataset to compute the perplexity. The results presented in (6
(the
While all above injected behaviors contribute to the preparation of RL finetuning, their effects on RL work from different aspects: subgoal computation and analysis reduce the perplexity on demonstration data and enable the models generate the ground-truc -truthh tokens with higher probability, suggesting these behaviors facilitate the exploitation of L.M.. Meanwhile, the reflection behavior leads larger policy entropy in generation, validating its effectiveness in encouraging LLMSs to explore. BRIDGE, we inject both types of behaviors to LLMS to enhance both exploration and exploitation of LLMs, making them significantly more "RL-ready" compared to the vanilla models.
Section 4.4 and 4.5 takeaway
The injected behaviors can either improve exploration or exploitation capability, specifically, Behavior injection is not sensitive to the behavior density, long behaviors are success-
fully injected into the Rre-RL model. (Figure 5)
Both exploration and exploitation behaviors injected through BRIDGE effective to
improving per-step influence (QQ f) in (5) thus benefiting RL. (Figure 4
Reflection improves exploration while subgoal computation and analysis improve exploita- tion, as examined by model entropy and perplexity. (Figure

---

Conclusion
In this paper, we investigate why different language models manifest divergent performances during RL finetuning by analyzing the per-step influence of data for GRPO algorithm. We then propose to inject behaviors, which are RL-favorable in terms of exploration and exploitation, into the SFT dataset to prepare LLMs for RL finetuning. Through various experiments across different benchmarks, we demonstrate that our method effectively makes the models RL-ready and significantly outperforms other augmentation baselines.
One limitation of BRIDGE is that the evaluation is focused on the math and common-sense reasoning tasks. The future work involves extending our method broader agentic domains with a larger dataset size. Meanwhile, we believe the analysis tools (c.g., per-step influence computation) can inspire future work on data curation and behavior discovery the community. One potential negative social impact of this paper is the misuse of our method, and unsafe behavior injection can lead to harmful consequences.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint urXiv:2303. 08774, 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint 1.Xii::2302.197777. 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901. 2020.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[5] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Relichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXi::2110.1416 2021.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1xXi::2007.03374 2021.
[8] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
[9] Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng. Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents.
arXiv preprint arXiv:2502. 13130, 2025.
[10] Zuxin Liu, Jesse Zhang, Kavosh Asadi, Yao Liu, Ding Zhao, Shoham Sabach, and Rasool Fakoor. Tail: Task-ip adapters for imitation learning with large pretrained models. arXiv preprint arXiv:2310.05905, 2023.
[11] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning large language models. Advances in neural information processing systems, 35:24824-24883 2022.
10

---

[12] Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[13] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint 2rXiv:2307.33854 2023.
[14] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: control- lable world of apps and people for benchmarking interactive coding agents. arXiv preprint arXiv:2407. 18901, 2024.
[15] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala al. Androidworld: dynamic benchmarking environment for autonomous agents. arXiv preprini arXiv:2405.14577 2024.
[16] Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-II A 1.8 million math instruction tuning dataset. Advances in Neural Information Processing Systems, 37:34737-34774 2024.
[17] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: large-scale dataset for android device control. Advances in Neural Informa- tion Processing Systems, 36:59708 597288, 2023.
[18] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243 2024.
[19] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems, 37:54463- 54482, 2024.
[20] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of I1m scaling based on human-generated data. Forty-firsi International Conference on Machine Learning, 2024.
[21] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedhack. Advances neural information processing systems, 35:27730-27144, 2022.
[22] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 00153728-537444 2023.
[23] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jumxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03333 200.222000000,, 2024.
[24] Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain-of--ction -ctto-tt-thhuutt enhances Ilm reasoning via autoregressive search. arXiv preprint 2xXi::5502.02500 2025.
[25] Kevin Chen, Marco Cusumano- Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive Il agents. arXiv preprint arXiv:2502.01600, 2025.

---

[26] Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. Pmlr, 2018.
[27] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[28] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644 2020.
[29] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neurul information processing systems, 33:1179- 179- 2020.
[30] Zhang- Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Hamess- ing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306. 13085, 2023.
[31] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
[32] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 16720 2024.
[33] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
[34] Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. arXiv preprint arXiv:2401.08967 3, 2024.
[35] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.0144 01566, 2025.
[36] Yuxiao Qu, Matthew YR Yang. Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Rus- lan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning. arXiv preprint urXiv:2503.07572 2025.
[37] Yuxiao Qu, Tianjun Zhang. Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. Advances in Neural Information Processing Systems, 37:55249-55285, 2024.
[38] Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint 2rXiv224/.004282 2024.
[39] Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024.
[40] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprini arXiv:2502. 14768, 2025.
[41] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.077 07190. 2024.
[42] Huatong Song, Jinhao Jiang, Yinggian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXhv:5003.05592 2025.
12

---

[43] Mingyang Chen, Tianpeng Li. Haoze Sun, Yijie Zhou, Chenzheng Zhu, Fan Yang, Zenan Zhou, Weipeng Chen, Haofen Wang, Jeff Z Pan, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503. 19470, 2025.
[44] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprini arXiv:2503.09516 2025.
[45] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TŞ" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15122 2024.
[46] Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilck Hakkani- Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504. 2025.
[47] Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461-12495, 2024.
[48] Qingyuan Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Vsc-rl: Advancing au- tonomous vision- -aaionla agents with variational subgoal conditioood reinforcement learning. arXiv preprint (tti::500.07744444 2025.
[49] Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai. Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:241 I. 1.02337, 2024.
[50] Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-rl: Incentivizing medical reasoning capability of vision-language models (ylms) via reinforcement learning. arXiv preprint arXiv:2502.19662 2025.
[51] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Ilm reasoning via reinforcement learning on open software evolution. arXiv preprint a7X.v.250..18449 2025.
[52] Yi-Long Lu, Chunhui Zhang, Jiajun Song, Lifeng Fan, and Wei Wang. Tom-rl: Reinforvement learning unlocks theory of mind in small 1lms. arXiv preprint XXXv:25544.0698. 2025.
[53] Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemotron research-tool-n Tool-using language models with reinforced reasoning. arXiv preprint arXiv:2505. 00024, 2025.
[54] Yi Ren and Danica J Sutherland. Learning dynamics of I1m finetuning. arXiv preprint PrXiv:2407.10490 2024.
[55] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances Neural Information Processing Systems, 13:19920-19930, 2020.
[56] Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kemel- based view of language model fine-tuning. International Conference on Machine Learning, pages 13610-23641 PMLR, 2023.
[57] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Dangi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.08844 2024.
[58] Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.0 000067 2025.
13

---

[59] Zhongzhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen. Yingying Zhang, Fei Yin, Jiahua Dong. Zhijiang Guo, Le Song, and Cheng-Lin Liu. From system I to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. /7419, 2025.
[60] Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, nathan lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, and Chelsea Finn. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-thought. arXiv preprint wxXiv2250.0466/ 2025.
[61] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint wrXiv:2503.095:0 2025.
[62] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373, 2025.
[63] Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36:70757-70798, 2023.
[64] William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. arXiv preprint axXiv:3100009933 2023.
[65] Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain thought empowers transformers to solve inherently serial problems. arXiv preprini (tt.:2402.1187755 1, 2024.
[66] Alireza Amiri, Xinting Huang, Mark Rofin, and Michael Hahn. Lower bounds for chain-of- thought reasoning in hard-attention transformers. arXiv preprint arXiv:2502.02339) 2025.
[67] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
[68] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint urXiv:2504.13833 2025.
[69] Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.09912, 2025.
[70] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint 2503vv2503.01300 2025.
[71] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15476-15488, 2022.
[72] Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew C Yao. Augmenting math word prob- lems via iterative question composing. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 24605-24613, 2025.
[73] Yihang Yao, Zhepeng Cen, Miao Li, William Han, Yuyou Zhang, Emerson Liu, Zuxin Liu, Chuang Gan, and Ding Zhao. Your language model may think too rigidly: Achieving reasoning consistency with symmetry-enhanced training. arXiv preprint arXiv:2502. 258.2.20277700 2025.
[74] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations, 2024.
[75] John Yang, Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofrr Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint 2rX1v:2504.21798, 2025.
14

---

[76] Elad Levi and Ilan Kadar. Intellagent: A multi-agent framework for evaluating conversational ai systems. arXiv preprint arXiv:2501.1106) 2025.
[77] Akshara Prabhakar, Zuxin Liu, Weiran Yao, Jianguo Zhang, Ming Zhu, Shiyu Wang, Zhi- wei Liu, Tulika Awalgaonkar, Haolin Chen, Thai Hoang, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. arXiv preprini arXiv:2504.0300 2025.
[78] Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yan- feng Wang, and Siheng Chen. Synthesizing post-training data for Ilms through multi-agent simulation. arXiv preprint arXiv:2410. 14251, 2024.
[79] Hector Vargas Alvarez, Gianluca Fabiani, Nikolaos Kazantzis, Ioannis G Kevrekidis, and Constantinos Siettos. Nonlinear discrete-time observers with physics-informed neural networks. Chaos, Solitons & Fractals, 186: 1 15215, 2024.
[80] Saptarshi Sengupta, Harsh Vashistha, Kristal Curtis, Akshay Mallipeddi, Abhinav Mathur, Joseph Ross, and Liang Gou. Mag-v: A multi-agent framework for synthetic data generation and verification. arXiv preprini arXiv:2412.04494 2024.
[81] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprini arXiv:2504.02605 2025.
[82] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333. 2024.
[83] Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model good teacher? an optimization perspective. arXiv preprint arXiv:2503. 15477 2025.
[84] Zihan Wang. Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlicb, et al. Ragen: Understanding self-evolution in Ilm agents via multi-ture reinforcement learning. arXiv preprint 2rXiv:2504.2007, 2025.
[85] Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xic. Dynamic evaluation of large language models by meta probing agents. In Forty-first International Conference on Machine Learning, 2024.
[86] Kaijie Zhu, Jiaao Chen, Jindong Wang. Neil Zhenqiang Gong, Diyi Yang, and Xing Xic. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2023.
[87] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hicrarchical Language Structures. SSRN Electronic Journal, May 2023. Full version available at phttps
// //ssrn. www.co/maaaaaaa 2ttmm/abbstaact555633
[88] Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. Prompthench: A unified library for evaluation of large language models. Journal of Machine Learning Research, 2-(25)):1-22, 2024.
[89] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part
2.2, How to Learn From Mistakes on Grade-School Math Problems. In Proceedings of the 13th International Conference on Learning Representations, ICLR 225, April 2025. Full version available at https:://////. pattps://ssaaee //ssrn.///neeeee com/ abstract=5250631
[90] Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. arXiv preprint wwXiv:2402.08939 2024.
[91] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report, arXiv preprint arXiv:2412.15115, 2024.
15

---

[92] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[93] Katie Kang, Amrith Setlur, Dibya Ghosh, Jacob Steinhardt, Claire Tomlin, Sergey Levine, and Aviral Kumar. What do learning dynamics reveal about generalization Ilm reasoning? arXiv preprint 1td::2411.076680 2024.
[94] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[95] Hugging Face. Open rl: fully open reproduction of deepseek-rl, January 2025.
[96] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[97] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
[98] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint 02X5v22504.02566, 2025.
[99] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[100] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.
[101] John Schulman. Approximating kl divergence, 2020.
[102] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023,
[103] William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26(1 (6616)9-66),, 1984.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
More details about the assumption discussion and computation resources are provided the appendix.
Justification:
We made the discussion of limitations in section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[Yes]
Justification:
The detailed proof of the proposition is provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The information needed to reproduce the main experimental results of the paper is introduced in section 4 More details are provided in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and dataset are provided the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/pbblic/guu.eee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, they are introduced section 4 and Appendix B.4 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experiments contain statistical significance tests. We visualize the perplex- ity and entropy distribution as shown in Figure 6 It indicates that the proposed behaviors are from types of either exploration exploitation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
They are introduced in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httptt://euiipss.coo /tt
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: They are discussed in section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the assets are cited. The licenses are provided in the code. Specifically, we use VeRL with Apache-2.0 License, iGSM with MIT License, and PromptBench with MIT license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in the paper are well documented and the documen- tation (README files) is provided alongside the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
23

---

A Proofs and Discussions of Theoretical Results
A.1 The Proof and Discussion of Proposition 3.1
The proof is as follows:
Proof. We first compute the advantage in one group. For query q with n correct outputs and N wrong outputs (0 < N), the mean and standard deviation of rewards are n/N and v /n(N - n)/N respectively. Accordingly, the advantages of positive and negative samples are
1-n/N A(q.0+) Vn(N-R)/N O-n/N
N-n
(6)
A(q.0_) Wn(N-n)/N
N-n
Note that the advantage will be all 0 if 0r N.
Then we consider the GRPO objective in E3.3). when the RL training is on policy, the importance ratio (o Tol/Totuuoll ignore the KL divergence regularization when the coefficient is small. Consequently, the gradient of simplified GRPO objective on query q
and thus
remove the corresponding clip terms. Mcanwhile,
VeJ (q:00)
VorT(o:iq)) (q) Foll (o ( A
-Boo.) \~o
(8)
[)(0, Told (o:1q)
-B,o, 1~z N
V log ())
(9)
N
=E,o, ~x0 ~x0
Ve log TO To( 01+ 1q)
Ve log To(o;-lq)A)
(10)
Vn(N~ n)
11
N n
=B.o.) ~x0
Vo log
Vo log no (0)-19))
(11)
n
N n
1
=B =B(o.)-To a)
Ve log To (0
V e log पo 19)
(12)
n
N n
where a n/N and we use the policy gradient VoTo(qlo) Th V log ne (q in derivation.
Similarly, if we do not expand the outputs to correct and incorrect ones in one group, the gradient of GRPO objective on the whole query set is
(ei(Q: Eq'=Q,
V mo''ooo (o'{(')
(13)
Therefore, obtain the per-step influence by the inner product of VoJ(q::) and (Q;
Remark A.I (Discussions on the assumptions). Our derivation of per-step influence relies on the on-policy training and small KL regularization coefficient assumptions, both of which approximately hold in practical implementation. The on policy training holds when one rollout step corresponds to one optimization step (i.e., one gradient descent step), which is consistent with our implementation
(see Appendix [.44) other framework such as open-R1 (95], or DeepSeekMath [33] where each generation is trained only for once. Moreover, the clip term further helps reduce the gap between Ty and mald even when strictly on-policy training is not guaranteed. For the KL regularization, we
observe that most implementations set a very small coefficient 33 (e.g., 0.01 or smaller), consistent with our assumption. Overall, the per-step influence mainly dominated by the data co-influence as derived in Proposition 3.1
24

---

Remark A.2 (Relation of data co- co-influence to neural tangent kernel (NTK)), The data co-influence Kg is also related NTK [9], kernel used describe how neural network evolves during training via gradient descent. Denote = (q, o) as the query-output pair, If we regard the language model as neural network with X as input and the logit z on the whole vocabulary (with size [V] of each token as output, we have
Vo log
1 T
E
11 mT
(1x)
vooz)))
(14)
where is the length of output o, or denotes the 1-th token of the output, 2 E R is the logits of t-th token. Here To (-Ix). e(o,) E Rl = (-Ix) is the output distribution of t-th token and e(o,) means the one-hot vector. Note that the result is divided by because there is nothing. Although we feed the model with the full answer sequence in X. the tokens in later position will not contribute to computing the output distribution of each token because the existence causal mask, which is natively integrated in auto-regressive language model.
Then the data co- co-influence can be decomposed
Kolx: x'1
(Vo log ())o))) Vo log mo(ool''''l))
(15)
(T (+1x) (Vo) (-Ix' c(d,))
TT
(16)
where the middle term (Vezr(-lx). Vez, (+xx)) is empirical NTK of the language model [54]
A.2 Extend Proposition 3.1 to Other RL Algorithms
Following previous notation, i.e., given query with n correct outputs and N - n wrong outputs, the accuracy a n/N and the advantages of correct and wrong output are A,, A_ respectively. We consider three RL variants:
Dr.RRPO 97
The advantages are A+ =1 N. A
N. Plug-in them to Eq
10), then
n(N- n)
12
V TeT
Bi.,) (~TT N2
8 log To (0++
Ve log no (0,-
(17)
71
n
E(oo) -0) a:)
Vo log no (0.1
V log To (o
(18)
N n
Then its per-step influence is to replace the original coefficient / aal a) by a(1 - a) and other parts remain the same GRPO.
multiples the advantage by a coefficient (it is a in original paper and we use C instead to avoid ambiguity). The advantages are A4 C (1 ), C (- ). Similar to Dr.GRPO, the coefficient in per-step influence is Ca(1 - a) and other parts remain the same.
GPG 198].
DAPO 199. The advantage computation in DAPO is the same as the GRPO so the coefficient is still /a a). DAPO introduces other modifications such as query filtering. Therefore, the corresponding per-step influence should replace the original query set Q by filtered query set Q' which only includes queries with rollout accuracy E (0, 1).
25

---

B More Details of Method and Experiment
B.1 Evaluation Benchmark
We provide a query along with vanilla demonstration answer for iGSM and PromptBench tasks. Meanwhile, we provide the corresponding DAG representation for each query for easy understanding.
iGSM. iGSM, we construct DAG by first generating the nodes, which are with the form of "each X's Y". There are two types of nodes abstract node and instance node based on the type of "Y". If "Y" is a class name, the node will be classified as an abstract node; if "Y" an instance of the class, the node will be classified an instance node. For example, consider class "Classroom" with its two instances "Painting room", "Computer room". the "each Oakwood Middle School's Classroom' an abstract node while "each Oakwood Middle School's Computer room" is instance node. After generating the nodes, we then generate the dependency of instance nodes randomly as edges construct DAG (we do not need to add dependency abstract nodes because it has implicit dependency. e.g.. "each Oakwood Middle School's Classroom" = "each Oakwood Middle School's Painting room" + "each Oakwood Middle School's Computer room"). We can then generate query and answer based on the DAG. In query, we only list the explicit dependencies on instance nodes which may include the conditions for redundant nodes; answer, the LLM should be able unlock all instance and abstract nodes to the final target node with a topological order. More implementation details (e.g.. how to generate random nodes and random edges, how to set the name of nodes) are systematically introduced in the original paper (74).
To improve training stability. modified the original iGSM. Here are the main modifications: (1) we remove the modulo operation in all computation steps; (2) we filter out the queries whose ground-truth answers
larger than 1000 smaller than - 1000 reduce the reliance on complex computation; (3) the original answer template is "Define [node name] as [a random letter). then [the random letter] = [... computation equations]". We rewrite keeping the logic of problem-solving (i.e., the order of node selection) and computation details. See below for an example of the new answer template.
increase the answer diversity while
One example with operation number op = 10 shown in the following text boxes. The DAG representation of this question is visualized in Figure 7
Plaing Sturs Junorrigh Clasermon
Gisphie Denign Studic's Managor Bockpack
Rio StaPS Highla Pastery ocm
Oalowcod Hiddle School's Choonuum
Puilery Classroam's Designer ling
Qhlasmoo Mitdle Serazl'ss Grapric Design Studio
Pointing Room's
Crosovier
Bockpack
School's Paiming
Classroom's Frinted Casust Backpack
Pattery Classrcom's Monager Bockipock
Oakwood Hidcllo School Pottwry Clasorricm
Cresiviow Midolo
School's Giophic
Dani ge Studio
Oulivood Hidell Schcols Puinting
Painting Room's Monager Bookipack
Room
Oruphic Desigh
Shuthote
Backpack
Figure 7: DAG representation of an iGSM task (number of operations 10). The red, light blue, dark blue, and gray rectangles mean the target, leaf, intermediate, and redundant nodes, respectively. The solid arrow indicates the explicit dependency which is explicitly given in query while the dash arrow means the implicit dependency which needs to be inferred by LLM. For example, it is not directly given how to compute the value of each "Rising Stars Junior High's Classroom" in the query. The LLM is required to understand its semantic meaning and then infer that it equals to "Rising Stars Junior High's Pottery Classroom".
Note that the RL training data (15 20 operations) and OOD test set (25 operations) in practical experiments much harder than the example above.
26

---

Query:
The number of cach Graphic Design Studio's Manager Backpack equals the difference of each Rising Stars Junior High's Classroom and each Oakwood Middle School's Classroom. The number of each Painting Room's Designer Bag equals 0 more than each Oakwood Middle School's Pottery Classroom.
The number each Painting Room's Printed Casual Backpack equals 9 more than the sum of each Painting Room's Designer Bag and each Pottery Classroom's Printed Casual Backpack.
The number of each Crestview Middle School's Graphic Design Studio equals the difference of each Painting Room's Designer Bag and each Oakwood Middle School's Painting Room. The number each Pottery Classroom's 's Designer Bag equals each Painting Room's Designer Bag. The number of each Pottery Classroom's 's Manager Backpack equals O.
The number of each Pottery Classroom's Printed Casual Backpack equals 1.
The number of each Rising Stars Junior High's Pottery Classroom equals times each Oakwood Middle School's Classroom.
The number cach Oakwood Middle School's Painting Room equals The number of cach Crestview Middle School's Painting Room equals 5.
The number of each Painting Room's Manager Backpack equals 0 more than the of each Graphic Design Studio's Backpack, each Oakwood Middle School's Pottery Classroom and each Oakwood Middle School's Painting Room.
The number of each Oakwood Middle School's Pottery Classroom equals 5 times each Pottery Classroom's 's Printed Casual Backpack.
The number of each Oakwood Middle School's Graphic Design Studio equals 8 more than each Painting Room's 's Designer Bag.
How many Manager Backpack does each Graphic Design Studio have?
Vanilla CoT:
Let's compute the answer step by step. According to the information given. the number of each Pottery Classroom's Printed Casual Back- pack Let's denote as So I.
Next, let Q represent the number of each Oakwood Middle School's Pottery Classroom. Then Q = 55 S = = 5.
Now,
find the number of each Painting Room's Designer Bag. Let's denote it U. Then 5.
U + Q
number each Oakwood Middle School's Graphic Design Studio. Let it be m. Then m 8 + U = 8 + 5 13.
WWe can then calculate
The number of each Oakwood Middle School's Painting Room is 6. Let's denote it as W. So W 6.
Then, let's denote the number of each Oakwood Middle School's Classroom as Then v m
Q+ + W = 13 + 5 + 24.
WWe can then calculate the number of cach Rising Stars Junior High's Pottery Classroom. Let it be
B. Then B = 2 v 2 24 48.
Now, can find the number of cach Rising Stars Junior High's Classroom. Let it be p. Then p
= 48.
Next, let represent the number of each Graphic Design Studio's Manager Backpack. Then y v 48 24 = 24.
p-v
Thus, the answer is 24.
PromptBench. PromptBench generates data in two stages: (1) DAG construction and (2) Natural language description of the DAG. For DAG construction, first generate directed acyelic graph (DAG) with specified depth and number of redundancies. The DAG is constructed top-down: we begin by generating the root node and then recursively sampling dependencies between each node and its parent node(s). If a node uses a binary operator, it has two parent nodes; if it uses a unary operator, has one parent node. This process continues recursively until the desired depth is reached. During DAG construction, each node is assigned unique name generated by a random string generator. Once the DAG structure complete, we sample values for all leaf nodes from a predefined set and compute the values of internal nodes in bottom-up manner. For the natural language description stage, we describe the constructed DAG and its associated computation using predefined templates to generate a textual problem description. To improve the training stability, we also filter out the queries whose ground-truth answers are larger than 1000 or smaller than 1000. Full details of the generation process can be found in the original PromptBench paper (86).
27

---

One example with depth = 4, number of redundancy 2 is shown in the following text boxes. The DAG representation of this question is visualized Figure
Rap
add
aBB
aak
aaj
sah
aRg
ABT
aag
aBU
nat
Figure 8: DAG representation of PromptBench task. The red, light blue, dark blue, and gray circles mean the target, leaf, intermediate, and redundant nodes, respectively.
Query:
The value of aac is 5. aaf gets its value The value of aaj is 5.
adding together the value of aab and
The value of aak is 9. The value of aar 2. The value of aaa is 9. The value of aat is 3. aao gets its value by adding together value of and aan. The value of aad is 5.
aas gets its value by multiplying together the value of aaq and aar.
The value of aaq is 5.
aan gets its value by adding together the value of aak and aaj. The value of aau is 10.
aai gets its value by multiplying together the value of aah and aag. aap gets its value by subtracting the value of aao from the value of aaf.
The value of aah is 2.
aav gets its value by multiplying together the value of aat and aau. nab gets its value by squaring the value that aaa has.
The value of aag is 6.
aac gets its value by subtracting the value of aac from the value of aad.
What is the value of aap?
Vanilla CoT:
Let's compute answer step by step.
Let's solve aaa, aaa is 9
Let's solve aab, aab aaa = 81 Let's solve aac, is 5
Let's solve aad,
Let's solve aac, aae = aac aad Let's solve aaf, aaf aab + aac 81 Let's solve aah, aah is 2 Let's solve aaj, aaj is Let's solve aag, aag 6
Let's solve aai, aai = aag aah = Let's solve aak, aak is 9 Let's solve aan,
aaj aak 14
Let's solve aao, Let's solve aap, aap aao aaf Thus, the answer is 55.
26
Why are iGSM and Promptbench good testbeds for LLM reasoning evaluation? We can observe that both tasks require the problem-solving capabilities of LLM from multiple perspectives, e.g., basic arithmetic calculation, search and planning, which mirrors reasoning tasks realistic scenarios. Meanwhile, these tasks require few priors (it only needs very basic concept understanding in iGSM task) and focus on the abilities instead of knowledge. Moreover, all the data are synthetic and the queries are almost infinite (sec estimation in (74).. Therefore, it avoiding the data contamination.
28

---

which can significantly confound the experiment results, while keeping substantial diversity of the query set.
B.2 The Extraction of DAG representation
For the datasets we used in this paper (iGSM and promptbench), all questions and answers are rule-based and follow a fixed sentence structure, which allows us extract the DAG representation through simple string matching.
Take an iGSM problem for example, each variable mentioned in the question is treated as a node, and we define an edge from one node to another if the value of the former depends on the latter. Consider premise The number of each Painting Room's Printed Casual Backpack equals 9 more than the sum of each Painting Room's 's Designer Bag and each Pottery Classroom's 's Printed Casual Backpack, view each Painting Room's 's P Printed Casual Backpack nodes each Painting Room's 's Designer Bag and each Pottery Classroom's 's Printed Casual Backpack and we draw edges from these construct the node set V and edge set E of the graph. Similarly, we can also apply the same procedure the answer. However, we will miss the redundant nodes if the answer only includes the minimal topological path to the final node.
node. Since it depends on
nodes to the former. By iterating all the premises in this way.
For other QA datasets without fixed format, can employ oracle LLM (e.g.. GPT4) parse the node and edge from the QA. Specifically, we can view all intermediate variables conclusions corollaries as node and the edge is still their dependency. Here is prompt extract (V, E):
Prompt to extract DAG:
You are a helpful data analyst. You be given question-answw pair. Your task extract the DAG of the question. Here are some instructions for extracting the DAG:
First separate the answer into multiple steps. If one step includes multiple intermediate variables conclusions, further separate it until each step includes only onc.
Identify NODE in each step. The node
intermediate variables / conclusions corollary. nodes.
Identify dependency between
Remember that the final graph should be acyclic. You should return the DAG in the following format:
the name of the nodes, e.g., node 1: node 2: y. node 3:
dependent list of each node, e.g., node 1: []. node 2: [1]. node [1. 2] [few-shot examples]
[QA]
B.3 Examples of Injected Behaviors
In this section, we give an example of injected behaviors on iGSM task.
Injected behaviors in iGSM:
Let's compute the answer step by step.
The number of each Oakwood Middle School's Painting Room is 6. Let's denote it as W. So W
6.
Then, let's denote the number of each Oakwood Middle School's Classroom as Then v
Q + W 13 + 5 + = 18 + 6 24.
-Then, let's denote the number of each Graphic Design Studio's Manager Backpack But we haven't calculated the number cach Rising Stars Junior High's Classroom yet, thus the value of y is still unknown.
Now. can find the number of each Graphic Design Studio's Manager Backpack. Remember that it has been denoted as y. We know that it equals the difference between each Rising Stars Junior High's Classroom and each Oakwood Middle School's Classroom. Then y = p 24 24.
Thus, the answer is 24.
29

---

The blue, violet, and cyan parts correspond to the subgoal computation, reflection, and information analysis behaviors. In subgoal computation, we enforee LLM to conduct only one operation (i.e.. +. x) in every derivation and keep all intermediate computations for complex equations. In reflection, the LLM explores a locked (not solvable yet) node "each Graphic Design Studio's 's Manager Backpack" and then goes back. In information analysis, the LLM integrates the information gathered from the query for better derivation of computation, which also an exploitative behavior.
B.4 Training Details
SFT dataset and baseline implementations. For the iGSM task, we use 2000 SFT data for Vanills and BRIDGE. The PP-Aug and RC- Aug augment each data for additional three times they have 8000 SFT data in total. For the PromptBench task, we use 5000 SFT data for Vamilla and BRIDGE. so they have 10000 SFT data in The Pp-Aug and RC-Aug augment each data for additional total.
SFT training. We first train the LLMs by SFT to narrow the domain gap between the pretraining corpus and evaluation tasks and enforce the model to answer the question with the demonstrated template, where the final reward is easy to extract and verify. The other training configurations attached below, which are shared by all base models on both the iGSM and PromptBench tasks.
Table 3: Configurations SFT training
Configurations
value
training epoch
batch size
128
learning rate
x 10
learning rate scheduler
constant
The system prompt along with query and answer templates are shown follows:
System prompt:
A conversation that the assistant solves the user's problem. The assistant first thinks about the reasoning process the mind and then provides the with the answer. The reasoning process and answer are enclosed within <think> </think and <answer> <lanswer> tags, respectively, i.e. <think> all the reasoning process here </think> <answer> final answer here </answer>.
SFT data template (lake Qwen tokenizer for example) <lim__stae wtt.sstbbssttm//immmmmmm _nddl> [system prompt)
<tim < startbuserrllm endl>
[query)
<lim_ wwt.sttbbasisttattlm_eeeddd
<think> {CoT answer) </think>
<answer> The final answer is (final answer) <lanswer>
RL training. After the SFT stage, we apply RL to further fine-tune the SFT models. Our im- plementation based on VeRL (100). When computing KI. divergence, we use the low variance implementation [101, aligning with the GRPO implementation 23]. The shared parameters are listed Table 4
In rollout, we set the maximum generation length as 2560 for iGSM and 1536 for PromptBench.
We top p = 1.0 for sampling generation. Besides, we observe that Llama3.2-1B may deviate from answer template so we add format reward to it, i.e., it will obtain a 0.05 reward if it strictly follows the given answer template (i.e., "<think> </think> <answer> </answer>") addition to the correctness reward.
B.5 More Experiment Results
We attach the training curves of main experiments in Fig. 9
30

---

Table Configurations of RL training
Configurations
value
batch size
256
learning rate
10-6
learning rate scheduler
constant
rollout number per query N
32
rollout temperature
1.0
rollout backend
vllm [102]
KL coefficient B
0.001
Vanilla
PP-Aug
RC-Aug
BRIDGE
Qwen-1.58
Qwen-3B
Liama-18
1.0
0.8
0.7
Training Reward
0.8
Training Reward
Tranning Reward
0.5
0.4
0.3
0.2
0.2
100
50
150
Step
Step
Step
Figure 9: The training curve of experiments in the iGSM task.
B.6 Comparison between BRIDGE and Accuracy-based Rejection Sampling
Based on the analysis of two factors affecting the performance in RL in Proposition 3.1 there is an intuitive idea that filters out queries with excessively high or low rollout accuracy (close to I or 0) as discussed in the beginning of sec. 3.3 In this section, we compare the performance of method with the rejection egjettin-sampling- sampling-base based RL. Specifically, we roll out the SFT models (pre-RL models) for times on larger dataset that shares the same distribution as the original RL training set. We then retain only the queries with to 7 correct answers, resulting in a filtered RL training dataset with medium accuracy ratio. Note that new dataset has the same size previous RL training dataset. Then we run RI. on the rejection sampled dataset starting from the same SFT model. The results are shown in Fig.
Vanilla
PP-Aug
RC-Aug
BRIDGE
Qwen Qwen-1.58 1.5B
Liama 18
1.0
0.8
Training Reward
0.8
Training Reward
0.7
0.5
0.4
0.4
0.2
0.3
0
20
40
60
80
100
20
40
60
80
100
Step
Step
Figure 10: The training curve of experiments in the PromptBench task.
31

---

Vanilla
Vanilla + rejection sampling
BRIDGE
BRIDGE + rejection sampling
Qwen2.5-1.58
Lamma32-18
0.9
0.8
Validation Accuracy
Validation Accuracy
0.6
0.5
0.4
0.3
0.3
50
100
150
200
100
150
200
Training step
Training step
Figure 11: The comparison BRIDGE with pepeection-sapplnng based filtering.
We can observe that the rejection sampling does not improve the RL performance over Vanilla baseline and is significantly worse than BRIDGE. Meanwhile, when applying rejection sampling to the BRIDGE, the difference very minor, and it even degrades the final performance on the Qwen- 1.5B model. The reason for its poor performance is that rejection sampling can only shape the accuracy distribution of RL. rollouts at the initial stage. As training progresses, the distribution may shift and is not guaranteed to remain within the medium-accurace range. More importantly, this method does not improve the data co-influence. Based on the above results, we can conclude that it is more effective to improve the data co-influence when preparing LLM for RL, which relies on behavior injection, rather than simply adjusting rollout accuracy by distorting the query distribution.
B.7 Details of Data Co-influence and Per-step Influence Estimation
Here a more detailed description on the computation of per-step influence shown Fig. right.
1. Begin by 2,000 queries. For each query, we generate 8 rollouts using the model, resulting in a total of 16,000 query-output pairs. Each query is then assigned to an accuracy group based on the accuracy of its rollouts.
2. For each query-output pair, we compute 1) advantage, and 2) the policy gradient Vo log (olg))
3. Using (5), we estimate the per-step influence for each query q. In this formulation, (d', ranges over all 16,000 output pairs.
4. We then group queries by their accuracy (0/8, 1/8, ..., 8/8), and compute the average per-step influence for each group. These results are visualized in Fig. Brrgght. Notably, the groups with 0/8 and 8/8 accuracy have zero per-step influence because the corresponding advantages are zero under the GRPO formulation.
Since computing the inner product [-. is computationally intractable, we mainly follow LESS in data co-influence estimation. Specifically, there are two main steps reducing the dimensionality of V log Ne (o]q)) (I) we first compute the LoRA gradient VoLas log Re (o rather than the full-parameter gradient by backpropagating the likelihood loss to the LoRA modules. (2) Then apply random projection on the LoRA gradient and get a vector with a smaller dimension, which preserves the inner product of two gradients [033. For the LoRA module, we specified rank of 64, an a value of 128, a dropout rate of 0.1, and learned LoRA matrices for all attention matrices. For the random projection, the final dimension of the projection output is 8192. Then data co-influence coefficient [p ( by the inner product between two query-outpin samples.
estimate the
After obtaining the data co-influence coefficient, we can compute the per-step influence AJ by plugging the data co-influence and advantages. Note that we did not multiple the learning rate when computing the results in Fig.
32

---

B.8 Computation Overhead
All experiments can be run on a server with Ax0 (80G). Each SFT experiment takes less than 1h. Regarding the RL experiments, it takes ~ 8h for Qwen-1.55 and Llama- IB to complete 200-step RL and ~ 6h for Qwen-3B to run 100-step RL on iGSM while it spends 2h run RL on PromptBench 1.5B (100 steps) for Qwen- 1,5B and Llama- 1B due to shorter rollout length.
33


Reviews:
[{"review_id": "zNUwBmOyOf", "reviewer": "Reviewer_NxGR", "summary": "This paper proposes BehavioR Injection Data auGmEntation (BRIDGE), a strategy designed to augment the training data for SFT in order to improve the performance of LLMs trained via RL. Since RL training is typically preceded by SFT, BRIDGE enhances the SFT dataset by constructing examples that incorporate both exploration and exploitation behaviors, thereby better preparing LLMs for subsequent RL training. Experiments are conducted on two families of LLMs: Qwen and Llama, and evaluated on two benchmarks: iGSM and PromptBench. The results show that LLMs trained on datasets augmented with BRIDGE achieve significant performance improvements after RL training, outperforming models trained with other data augmentation strategies.", "questions": "1. Could you provide more details on how to extract the DAG representation from a query and its corresponding response?\n\n2. As shown in Table 2, it seems that the experimental results for Qwen2.5-3B are missing from PromptBench.\n\n3. Could you explain why the injected behaviors influence the accuracy distribution of the rollout samples?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "> **Strengths**\n\n1. BRIDGE is motivated by per-step influence analysis, which highlights two key factors for enhancing the performance of LLMs: (1) rollout accuracy and (2) the co-influence coefficient between training and target data.\n\n2. Experimental results show that LLMs trained on data augmented with BRIDGE achieve exceptional performance after RL training.\n\n3. In addition to experiments demonstrating the effectiveness of BRIDGE-augmented data, the paper includes ablation studies that analyze various injected behaviors. These studies provide valuable insights into constructing SFT datasets to improve LLM performance following RL training.\n\n> **Weaknesses**\n\n1. The LLMs used in the experiments have no more than 3 billion parameters. It would be preferable to include experimental results on more widely used LLMs in academic research, such as those with approximately 7 billion parameters.\n2. Considering that the Adam optimizer is predominantly used for training LLMs, the mathematical formulation for estimating the per-step influence of q on model performance may lack accuracy.", "was_revised": true, "final_justification": "5: Accept", "rebuttal": "We thank the reviewer for your thoughtful and constructive feedback. Here are our responses:\n- **W1. Include experimental results on larger models (such as 7B).**\n  \n  We run experiments on iGSM tasks with Qwen2.5-7B and llama3.1-8B models. The settings are the same as the previous experiments: we first apply SFT for 5 epochs on the base model and then run RL for 100 steps. Here are the results of their accuracy (the numbers before and after the `/` indicate the accuracy on in-domain and OOD test set respectively).\n  \n  | Base     |    | Vanilla     | PP-Aug      | RC-Aug      | BRIDGE   |\n  | --- | --- | --- | --- | --- | ---- |\n  | Qwen 7B  | SFT      | 54.4 / 37.6 | 61.8 / 41.2 | 53.4 / 33.8 | 61.8 / 49.4         |\n  |          | RFT      | 74.2 / 64.8 | 76.0 / 62.0 | 73.0 / 60.8 | **91.0** / **83.2** |\n  |          | $\\Delta$ | 19.8 / 27.2 | 14.2 / 20.8 | 19.6 / 27.0 | **29.2** / **33.8** |\n  | LLama 8B | SFT      | 61.6 / 50.8 | 69.0 / 57.8 | 60.8 / 43.8 | 53.0 / 46.4         |\n  |          | RFT      | 68.4 / 59.2 | 76.4 / 62.8 | 65.6 / 57.8 | **89.4** / **87.6** |\n  |          | $\\Delta$ | 6.8 / 8.4   | 7.4 / 5.0   | 4.8 / 14.0  | **36.4** / **41.2** |\n  \n  These results demonstrate that BRIDGE maintains its significant advantage over all baselines on these larger models. The substantial improvement from RL (Δ) when using BRIDGE suggests that our method effectively equips the base model with generalizable reasoning capabilities, making it more amenable to reinforcement learning, rather than simply encouraging it to memorize specific training examples. We believe these new results substantially strengthen our paper's claims.  \n  \n- **W2. The estimating the per-step influence of q on model performance may lack accuracy due to the Adam optimizer.**\n  \n  Thank you for raising this important concern. We acknowledge that estimating per-step influence under adaptive optimizers like Adam introduces certain approximations. However, we would like to offer a few points of clarification: 1) The estimation is particularly accurate at the initial steps of training, where the Adam optimizer behaves similarly to vanilla SGD. Since the performance gain in RL is mainly in the early stage of the training, we find the approximation still meaningful and informative. 2) More faithful modeling of Adam optimizer requires tracking its state, which is computationally intractable in our large-scale RL setting. Previous work [1] attempts to estimate this state using separate data, but such approaches also introduce additional approximation. 3) Our approach aligns with prior studies (e.g., [2]), which also adopts SGD approximations when analyzing training dynamics under Adam. Such approximations have been shown to offer valuable insights without optimizer state. \n  \n  Overall, while we agree that more precise modeling of per-step influence under Adam is an interesting direction, we believe our current approximation still provides a reasonable and useful estimate. We appreciate the suggestion and consider this a valuable avenue for future work.\n\n- **Q1. More details on how to extract the DAG representation from a query and its corresponding response**\n  \n  Thanks for your question. we are happy to provide more details here.\n  - For the datasets we used in this paper (iGSM and promptbench),  all questions and answers are rule-based and follow a fixed sentence structure, which allows us to extract the DAG representation through simple **string matching**. Take a iGSM problem for example, each variable mentioned in the question is treated as a node, and we define an edge from one node to another if the value of the former depends on the latter. Consider a premise `The number of each Painting Room’s Printed Casual Backpack equals 9 more than the sum of each Painting Room’s Designer Bag and each Pottery Classroom’s Printed Casual Backpack.`. we view `each Painting Room’s Printed Casual Backpack` as a node. Since it depends on nodes `each Painting Room’s Designer Bag` and `each Pottery Classroom’s Printed Casual Backpack` and we draw edges from these two nodes to the former. By iterating all premises in this way, we construct the node set V and edge set E of the graph. Similarly, we can also apply the same procedure to the answer. However, we will miss the redundant nodes (which are defined in the question but do not contribute to the computation of the final node, e.g., the gray node in fig.7 in appendix) if the answer only includes the minimal topological path to the final node.\n  - For other QA datasets without a fixed format, we can employ an oracle LLM (e.g., GPT4) to parse the node and edge from the QA. Specifically, we can view all intermediate variables / conclusions / corollaries as node and the edge is still their dependency. Here is a prompt to extract (V, E):\n    ```\n    You are a helpful data analyst. You will be given a question-answer pair. Your task is to extract the DAG of the question. Here are some instructions for extracting the DAG:\n    - First separate the answer into multiple steps. If one step includes multiple intermediate variables / conclusions, further separate it until each step includes only one.\n    - Identify NODE in each step. The node can be intermediate variables / conclusions / corollary.\n    - Identify the dependency between the nodes.\n    - Remember that the final graph should be acyclic.\n\n    You should return the DAG in the following format:\n    - the name of the nodes, e.g., node 1: x, node 2: y, node 3: z\n    - the dependent list of each node, e.g., node 1: [], node 2: [1], node 3: [1, 2]\n\n    [few-shot examples]\n    [QA]\n    ```\n    Given a QA sample from MATH dataset:\n    ```\n    Question:\n    Define \n    \\[\n    p = \\sum_{k = 1}^\\infty \\frac{1}{k^2} \\quad \\text{and} \\quad q = \\sum_{k = 1}^\\infty \\frac{1}{k^3}.\n    \\]\n    Find a way to write \n    \\[\n    \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3}\n    \\]\n    in terms of $p$ and $q$.\n\n    Answer:\n    We count the number of times $\\frac{1}{n^3}$ appears in the sum \n    \\[\n    \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3},\n    \\]\n    where $n$ is a fixed positive integer. (In other words, we are conditioning the sum on $j + k$.) \n    We get a term of $\\frac{1}{n^3}$ each time $j + k = n.$ The pairs $(j,k)$ that work are $(1,n - 1),$ $(2,n - 2),$ $\\dots,$ $(n - 1,1),$ for a total of $n - 1$ pairs. \n    Therefore, \n    \\begin{align*} \n    \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3} &= \\sum_{n = 1}^\\infty \\frac{n - 1}{n^3} \\\\ \n    &= \\sum_{n = 1}^\\infty \\left( \\frac{n}{n^3} - \\frac{1}{n^3} \\right) \\\\ \n    &= \\sum_{n = 1}^\\infty \\left( \\frac{1}{n^2} - \\frac{1}{n^3} \\right) \\\\ \n    &= \\sum_{n = 1}^\\infty \\frac{1}{n^2} - \\sum_{n = 1}^\\infty \\frac{1}{n^3} \\\\ \n    &= \\boxed{p - q}. \n    \\end{align*}\n    ```\n    The extract result is:\n    ```\n    Node names:\n    node 1: ( S = \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3} )\n    node 2: Number of pairs ((j, k)) with (j + k = n) is ( n-1 )\n    node 3: ( S = \\sum_{n=1}^\\infty \\frac{n-1}{n^3} )\n    node 4: ( \\frac{n-1}{n^3} = \\frac{1}{n^2} - \\frac{1}{n^3} )\n    node 5: ( S = \\sum_{n=1}^\\infty \\left(\\frac{1}{n^2} - \\frac{1}{n^3}\\right) )\n    node 6: ( S = \\sum_{n=1}^\\infty \\frac{1}{n^2} - \\sum_{n=1}^\\infty \\frac{1}{n^3} )\n    node 7: ( S = p - q )\n    \n    Dependency list:\n    node 1: []\n    node 2: []\n    node 3: [1, 2]\n    node 4: []\n    node 5: [3, 4]\n    node 6: [5]\n    node 7: [6]\n    ```\n    This example shows that we can still extract DAG from other QA data without a pre-defined format.\n  \n  We will update the details and examples in the revised manuscript.\n  \n- **Q2. In Table 2, the results for Qwen2.5-3B are missing from PromptBench.**\n  \n  Thanks for your reminder. We ran the experiment on Qwen2.5-3B on PromptBench. Here are the results (RL is trained for 100 steps):\n  \n  | Base    |   | Vanilla     | PP-Aug      | RC-Aug      | BRIDGE  |\n  | --- | --- | --- | ---- | ---- | ---- |\n  | Qwen 3B | SFT      | 31.2 / 18.0 | 40.6 / 29.4 | 35.2 / 19.4 | 44.4 / 31.0         |\n  |         | RFT      | 50.0 / 34.6 | 66.0 / 50.8 | 64.0 / 43.4 | **85.8** / **70.0** |\n  |         | $\\Delta$ | 18.8 / 16.6 | 25.4 / 21.4 | 28.8 / 24.0 | **41.4** / **39.0** |\n  \n  In the results, we can observe that BRIDGE achieves the best performance among all the baselines, which is consistent to the results we presented in the main context.\n\n- **Q3. Why the injected behaviors influence the accuracy distribution of the rollout samples?**\n  \n  Because the training data (with and without behavior injection) differs in both content and structure, which naturally leads to differences in the resulting SFT models’ performance. Specifically, there are two main contributing factors: 1) Exploitation behaviors often improve accuracy, as they encourage the model to better utilize the available information, reinforcing effective problem-solving strategies. 2) Exploration behaviors, on the other hand, may reduce accuracy of the SFT model. This is because they deliberately introduce incorrect intermediate reasoning steps (e.g., through reflection), which can cause the model to make errors on questions it would otherwise solve correctly.\n  \n  While these behaviors have different effects on the SFT model’s accuracy, they are all valuable from an RL training perspective. As shown in Fig. 5, incorporating both exploration and exploitation behaviors improves the overall performance of the RL-finetuned model, suggesting that behavior injection plays a crucial role in enhancing the model’s learning dynamics.\n  \nWe sincerely appreciate your effort again and we hope our responses can address your concerns.\n  \n[1] Xia, Mengzhou, et al. \"Less: Selecting influential data for targeted instruction tuning.\"\n\n[2] Ren, Yi, and Danica J. Sutherland. \"Learning dynamics of llm finetuning.\""}, {"review_id": "FrZMqiExlG", "reviewer": "Reviewer_JjKq", "summary": "This paper presents BRIDGE (BehavioR Injection Data auGmEntation), a method to prepare language models for more effective reinforcement learning fine-tuning (RFT) by injecting exploration and exploitation behaviors during supervised fine-tuning (SFT). The core contributions of this paper are 1) analysis of reinforcement learning behavior and its correlation with rollout data; 2) proposed data augmentation strategy BRIDGE; 3) empirical evaluation. The authors identify two critical factors influencing RFT performance: rollout accuracy and data co-influence. Guided by this analysis, BRIDGE enhances SFT datasets by injecting exploration and exploitation behaviors, making models \"RL-ready\" and improving their learning efficiency and generalization in the RFT stage.", "questions": "- Why does BRIDGE yield quite low accuracy on SFT stage (Table 2)?\n- Do authors scale up to 7B/32B models?\n- How does the theory apply to other RL algorithms?\n- What's the effect of training on BRIDGE-construcrted for more epoch? Will it affect the generalizability of the model?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- Pre-RL preparation is important. Many prior literature [1,2] pin-point this but few works delve deep into the direction.\n- The idea is well-motivated by theory\n- Data efficiency. This paper shows a data-efficient preparation approach to incentivize language models' potentials in RL training, where the proposed strategy requires 2000/5000 data samples for two tasks according to the appendix.\n- Remarkable empirical improvements.\n\n\nWeakness:\n- The general impact is upper-bounded by the scalability of DAG representation.\n- The theoretical analysis is specific to GRPO, where the coefficient $\\sqrt{\\alpha(1-\\alpha)}$ depends on GRPO's advantage estimation. It is unclear how the theory intuition will changes on other RL algorithms\n- It is unclear how the training epochs will affect the BRIDGE performance. As shown in table 1-2, different epochs affect the final performance a lot but the effect of more training epochs is not presented.\n\n\n[1] Guo, Daya, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n[2] Wang, Zengzhi, Fan Zhou, Xuefeng Li, and Pengfei Liu. \"Octothinker: Mid-training incentivizes reinforcement learning scaling.\" arXiv preprint arXiv:2506.20512 (2025).", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful and constructive feedback. Here are our responses:\n- **W1. The general impact is upper-bounded by the scalability of DAG representation.**\n  \n  Thank you for raising this important point. We agree that our method relies on the DAG representation. However, we believe that this representation is broadly applicable to many question types solvable by sequentially resolving intermediate steps within a CoT reasoning, which naturally contains a underlying DAG structure. To extract the DAG from more general questions, given a CoT-style answer, we can use an oracle LLM (e.g., GPT-4) to parse the structure by identifying intermediate variables, conclusions, or corollaries as nodes, and defining edges based on their dependencies. This allows us to construct a DAG even when the original question does not explicitly follow a fixed template. \n  \n  Here is an example. We use the following prompt to extract DAG:\n  ```\n  You are a helpful data analyst. You will be given a question-answer pair. Your task is to extract the DAG of the question. Here are some instructions for extracting the DAG:\n  - First separate the answer into multiple steps. If one step includes multiple intermediate variables / conclusions, further separate it until each step includes only one.\n  - Identify NODE in each step. The node can be intermediate variables / conclusions / corollary.\n  - Identify the dependency between the nodes.\n  - Remember that the final graph should be acyclic.\n\n  You should return the DAG in the following format:\n  - the name of the nodes, e.g., node 1: x, node 2: y, node 3: z\n  - the dependent list of each node, e.g., node 1: [], node 2: [1], node 3: [1, 2]\n\n  [few-shot examples]\n  [QA]\n  ```\n  Given a QA sample with the highest difficulty (level 5) from MATH dataset:\n  ```\n  Question:\n  Define \n  \\[\n  p = \\sum_{k = 1}^\\infty \\frac{1}{k^2} \\quad \\text{and} \\quad q = \\sum_{k = 1}^\\infty \\frac{1}{k^3}.\n  \\]\n  Find a way to write \n  \\[\n  \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3}\n  \\]\n  in terms of $p$ and $q$.\n\n  Answer:\n  We count the number of times $\\frac{1}{n^3}$ appears in the sum \n  \\[\n  \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3},\n  \\]\n  where $n$ is a fixed positive integer. (In other words, we are conditioning the sum on $j + k$.) \n  We get a term of $\\frac{1}{n^3}$ each time $j + k = n.$ The pairs $(j,k)$ that work are $(1,n - 1),$ $(2,n - 2),$ $\\dots,$ $(n - 1,1),$ for a total of $n - 1$ pairs. \n  Therefore, \n  \\begin{align*} \n  \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3} &= \\sum_{n = 1}^\\infty \\frac{n - 1}{n^3} \\\\ \n  &= \\sum_{n = 1}^\\infty \\left( \\frac{n}{n^3} - \\frac{1}{n^3} \\right) \\\\ \n  &= \\sum_{n = 1}^\\infty \\left( \\frac{1}{n^2} - \\frac{1}{n^3} \\right) \\\\ \n  &= \\sum_{n = 1}^\\infty \\frac{1}{n^2} - \\sum_{n = 1}^\\infty \\frac{1}{n^3} \\\\ \n  &= \\boxed{p - q}. \n  \\end{align*}\n  ```\n  The extract result is:\n  ```\n  Node names:\n  node 1: ( S = \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3} )\n  node 2: Number of pairs ((j, k)) with (j + k = n) is ( n-1 )\n  node 3: ( S = \\sum_{n=1}^\\infty \\frac{n-1}{n^3} )\n  node 4: ( \\frac{n-1}{n^3} = \\frac{1}{n^2} - \\frac{1}{n^3} )\n  node 5: ( S = \\sum_{n=1}^\\infty \\left(\\frac{1}{n^2} - \\frac{1}{n^3}\\right) )\n  node 6: ( S = \\sum_{n=1}^\\infty \\frac{1}{n^2} - \\sum_{n=1}^\\infty \\frac{1}{n^3} )\n  node 7: ( S = p - q )\n    \n  Dependency list:\n  node 1: []\n  node 2: []\n  node 3: [1, 2]\n  node 4: []\n  node 5: [3, 4]\n  node 6: [5]\n  node 7: [6]\n  ```\n  This indicates that we can extract DAG representation from other QA data and our method is able to generalize to a broad range of tasks. \n  \n- **W2 & Q3. The theoretical analysis is specific to GRPO, where the coefficient depends on GRPO's advantage estimation... How does the theory apply to other RL algorithms?**\n  \n  Thanks for this question. The key of applying the theory to other RL algorithm is to replace the advantage function. Following the notation in our original paper, i.e., given a query with $n$ correct outputs and $N-n$ wrong outputs, the accuracy $\\alpha = n/N$ and the advantages of correct and wrong output are $A_+, A_-$ respectively. We consider three RL variants:\n  - Dr.GRPO [1]. The advantages are $A_+ = 1-\\frac{n}{N}, A_- = -\\frac{n}{N}$. Plug-in them to Eq.(10) (in appendix), the corresponding gradient is $$\n    \\nabla_\\theta \\mathcal{J} = \\mathbb{E} \\frac{n(N-n)}{N^2}[\\frac{1}{n}\\sum_{i=1}^n \\log\\pi_\\theta(o_{i+}|q) - \\frac{1}{N-n}\\sum_{j=1}^{N-n}\\log\\pi_\\theta(o_{j-}|q)] \\\\\n    =  \\mathbb{E} \\alpha(1-\\alpha) [...]\n    $$ Then its per-step influence is to replace the original coefficient $\\sqrt{\\alpha(1-\\alpha)}$ by $\\alpha(1-\\alpha)$ and other parts remain the same as GRPO.\n  - GPG [2]. It multiples the advantage by a coefficient $C$ (it is $\\alpha$ in original paper and we use $C$ instead to avoid ambiguity). The advantages are $A_+ = C\\cdot(1-\\frac{n}{N}), A_- = C\\cdot (-\\frac{n}{N})$. Similar to Dr.GRPO, the coefficient in per-step influence is $C \\alpha (1-\\alpha)$ and other parts remain the same.\n  - DAPO [3]. The advantage computation in DAPO is the same as the GRPO so the coefficient is still $\\sqrt{\\alpha (1-\\alpha)}$. DAPO introduces other modifications such as query filtering. Therefore, the corresponding per-step influence should replace the original query set $Q$ by a filtered query set $Q'$ which only includes queries with rollout accuracy $\\in(0,1)$.\n  \n  The results on other RL variants show that \"section 3.2 takeaway\" holds for different RL algorithms, i.e., the rollout accuracy and co-influence are still two key factors to the performance gain in RL.\n\n- **W3 & Q4. ... As shown in table 1-2, different epochs affect the final performance a lot but the effect of more training epochs is not presented. What's the effect of training on BRIDGE-construcrted for more epoch? Will it affect the generalizability of the model?**\n  \n  We want to first clarify that the performances in table 1 and 2 are not comparable: table 1 shows the results on iGSM task while table 2 shows the results on PromptBench. They have different SFT dataset sizes. Moreover, the SFT epoch only affects the SFT training and we adopt the exact same settings for the follow-up RL training.\n  \n  We provide the results of training with different SFT epochs on iGSM tasks (where we adopt epoch num = 5 in original paper). The following table shows the accuracy on validation set (the same as the set in fig.5). \n  \n  | Base      | Epoch num | 2    | 5    | 10   | 20   |\n  | --------- | --------- | ---- | ---- | ---- | ---- |\n  | Qwen 1.5B | SFT       | 38.4 | 38.6 | 39.4 | 36.0 |\n  |           | RFT       | 86.8 | 89.4 | 88.2 | 77.6 |\n  | Llama 1B  | SFT       | 20.4 | 27.4 | 28.6 | 24.0 |\n  |           | RFT       | 55.2 | 57.4 | 54.6 | 40.4 |\n  \n  In general, more SFT epoch slightly improves the performance of the SFT model when epoch num $\\leq 10$. The performance of RL model is not very sensitive to the SFT epoch in the range of 2 ~ 10. However, we also observe a performance drop if training SFT for 20 epochs. The main reason is that the excessive SFT training decreases the entropy of LLM output, which reduces the exploration in RL training and leads to slower performance increase.\n\n- **Q1. Why does BRIDGE yield quite low accuracy on SFT stage (Table 2)?**\n  \n  That's a good question. The relatively low accuracy of the SFT model mainly stems from over-reflection after BRIDGE injects reflection behaviors. Specifically, the model imitates the injected reflection pattern in the BRIDGE SFT data and repeatedly attempts to solving an unlocked node (i.e., a node whose parent nodes have not yet been resolved) and then reflects. This often leads to unnecessarily long reasoning chains that reach the generation length limit, causing the model to fail to complete the answer and identified as wrong.\n\n  While this behavior results in low initial accuracy, it is effectively corrected during RL fine-tuning. As shown in our results, RL training significantly improves its final performance.\n  \n- **Q2. Scale up to 7B/32B models**\n  \n  According to reviewer's suggestion, we ran the experiments on iGSM tasks with Qwen2.5-7B and llama3.1-8B models. The settings are the same as the previous experiments: we first train the base model by SFT for 5 epochs and then run RL for 100 steps.\n  \n  | Base     |    | Vanilla     | PP-Aug      | RC-Aug      | BRIDGE              |\n  | -------- | -- | --- | ---- | --- | ---- |\n  | Qwen 7B  | SFT      | 54.4 / 37.6 | 61.8 / 41.2 | 53.4 / 33.8 | 61.8 / 49.4         |\n  |          | RFT      | 74.2 / 64.8 | 76.0 / 62.0 | 73.0 / 60.8 | **91.0** / **83.2** |\n  |          | $\\Delta$ | 19.8 / 27.2 | 14.2 / 20.8 | 19.6 / 27.0 | **29.2** / **33.8** |\n  | LLama 8B | SFT      | 61.6 / 50.8 | 69.0 / 57.8 | 60.8 / 43.8 | 53.0 / 46.4         |\n  |          | RFT      | 68.4 / 59.2 | 76.4 / 62.8 | 65.6 / 57.8 | **89.4** / **87.6** |\n  |          | $\\Delta$ | 6.8 / 8.4   | 7.4 / 5.0   | 4.8 / 14.0  | **36.4** / **41.2** |\n  \n  We can observe that BRIDGE achieves the best performance with 7B and 8B models, which shows the scalability of our method. For tasks that require capability of 32B+ model size, we leave them as future works due to our current constraints on computation resources.\n\nWe sincerely appreciate your effort again and we hope our responses can address your concerns.\n\n\n[1] Understanding R1-Zero-Like Training: A Critical Perspective\n\n[2] GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning\n\n[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale"}, {"review_id": "vdCtveee0i", "reviewer": "Reviewer_to1g", "summary": "This paper investigates why reinforcement fine-tuning (RFT) sometimes yields large gains for large language models (LLMs) and sometimes fails to help. The authors identify two key conditions that make a model RL-ready: (1) an intermediate rollout accuracy (the base model’s initial success rate on task rollouts should be neither too low nor too high) and (2) a strong data co-influence (training on one sample should generalize to others, indicating the model can propagate learned knowledge). To prepare an LLM to better benefit from RFT, the paper proposes Behavior Injection, a data-centric augmentation strategy called BRIDGE. The idea is to enrich the supervised fine-tuning (SFT) data with behaviors that encourage exploration and exploitation before any RL training. Concretely, the authors augment the model’s chain-of-thought demonstrations with additional reasoning steps: subgoal computation (explicitly breaking down problems into intermediate steps), information analysis (aggregating relevant info in reasoning), and reflection (exploring an alternative path or revisiting a sub-problem). By seeding these behaviors in the SFT phase, the base model produces more informative and varied trajectories during RL, which in turn leads to larger performance improvements from the RL fine-tuning. The paper evaluates BRIDGE on two challenging reasoning benchmarks (grade-school math iGSM and the arithmetic reasoning subset of PromptBench), using multiple base models of different sizes. In experiments, models prepared with behavior injection consistently achieve significantly higher gains from RFT compared to baseline approaches (including no augmentation, premise shuffling, and existing chain-of-thought augmentations). For example, BRIDGE-prepared models not only start with solid performance but after RL fine-tuning they outperform others by a large margin on both in-distribution and out-of-distribution test sets. Overall, the paper’s contributions include a theoretical analysis of RL dynamics for LLMs, a novel data augmentation method to improve those dynamics, and empirical evidence that this method makes LLMs more amenable to reinforcement learning.", "questions": "1. How well does behavior injection generalize to more complex or interactive tool-use tasks beyond static QA or math problems?\n\n2. What’s the impact of the demonstration source or quality? Were they human-written, model-generated, or noisy? How robust is the method to imperfect traces?\n\n3. How does behavior injection compare to prompting approaches like “think step-by-step” at inference time? Does fine-tuning offer clear advantages?\n\n4. Are there cases where the model overuses tools unnecessarily? Any error analysis on when behavior injection might lead to misuse would be helpful.", "limitations": "The authors explicitly discuss some limitations and potential negative impacts, which is commendable. They acknowledge that the scope of evaluation is limited to math and common-sense reasoning tasks, and suggest that future work should extend BRIDGE to broader domains (e.g. more complex agent environments) and with larger datasets. I agree with this assessment – the method’s current validation, while thorough for what it is, doesn’t cover all possible use cases. Another limitation implicitly noted is that the approach thus far is demonstrated on tasks with ground-truth verifiable rewards; tasks involving subjective or human-provided rewards are not directly tested, which could be addressed in the future. The authors also mention a potential negative societal impact: the misuse of their method. Since behavior injection is a data augmentation technique, a malicious actor could in principle inject undesirable behaviors into a model (for example, tendencies towards biased, unethical, or unsafe outputs) and then amplify them via RL. This is a realistic concern – if someone “prepares” a model with harmful behaviors and then optimizes for them with RL, it could yield a more capable but harmful agent. The paper advises caution here, which is appropriate. Mitigating this risk would involve careful curation of behaviors and perhaps oversight during RL (an issue that overlaps with general AI safety considerations). On the positive side, the method is aimed at improving reasoning and alignment with correct answers, which is a beneficial goal; there is nothing inherently dangerous in BRIDGE itself if used responsibly. One limitation not explicitly discussed in the paper (but worth noting) is the manual effort or expertise required to design injected behaviors. As raised above, BRIDGE currently relies on human intuition to decide what constitutes a helpful exploratory or exploitative behavior for a task. This could be seen as a form of expert data augmentation. If a user of this method does not have a good understanding of the task’s solution structure, they might struggle to apply BRIDGE effectively. While not a dire flaw, this is a practical consideration: the need for domain-specific behavior engineering could limit adoption or require further research (perhaps the authors could mention this as an area for future improvement). Aside from that, I did not identify major overlooked issues. The paper is quite conscientious in evaluating its own scope. Computational cost is reasonably addressed (the method doesn’t drastically increase training cost, aside from some overhead to generate augmented data and compute influence metrics for analysis). The ethical considerations are relatively mild for this work (since it’s about improving model training, not directly producing problematic content). The mention of misuse covers the main worry. In sum, the authors have adequately discussed limitations and societal impacts, and I appreciate their transparency. Any omissions are minor and can be tackled in subsequent research or the rebuttal.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\nQuality (analysis): The paper provides a thoughtful theoretical analysis of what drives learning in the RL fine-tuning stage. It introduces a per-step influence framework that isolates how rollout accuracy and a co-influence coefficient contribute to gradient signal during RFT. This analysis yields an intuitive insight: if a model gets every training query either entirely correct or entirely wrong, the RL reward signal is uninformative (gradient is zero in those extreme cases). The most learning happens when the model’s initial answers are partially correct. This perspective is well-grounded and novel in the LLM context, and it directly motivates the proposed solution (shape the SFT model such that its knowledge is incomplete but improvable under RL). The analytical findings are clearly connected to the method, lending a strong foundation to the paper’s approach.\nOriginality: The idea of “behavior injection” to make an LLM more RL-ready is innovative. Rather than modifying the RL algorithm or reward scheme, the authors take a data-centric approach: they augment the training data to instill certain behaviors before performing RL. While data augmentation for better reasoning is not new, this work is the first (to my knowledge) to explicitly inject exploratory and exploitative reasoning patterns to facilitate subsequent policy optimization. This behavior-level augmentation goes beyond simply adding more data or shuffling it – it changes how the model approaches problems (e.g. sometimes pausing to reflect or breaking problems into subparts). This approach of prepping a model for RL by curating its reasoning habits is a fresh contribution and could inspire other “RL-preparation” techniques.\nSignificance: The ability to improve an LLM’s performance via RL with less hassle is quite significant. Many important applications of LLMs (e.g. solving complex math problems, planning, tool use) rely on reinforcement fine-tuning for alignment or enhanced reasoning. However, RFT can be unstable or yield minimal gains if the base model isn’t already in a good state. This paper’s findings and method address that bottleneck. Empirically, the gains reported are impressive: for instance, on the iGSM math benchmark, the BRIDGE-augmented models see large accuracy jumps after RL (up to ~30-50% absolute improvement over their pre-RL accuracy, far exceeding baseline gains). Importantly, these gains hold on both seen (in-distribution) and harder (out-of-distribution) problems, indicating robustness. Such improvements suggest that BRIDGE could make RL fine-tuning a more routinely effective tool, potentially impacting how practitioners approach alignment and domain adaptation of LLMs. Even if the method is demonstrated on math and logic tasks here, the general idea could be applicable to a wide range of reasoning-heavy domains.\nQuality (experiments): The experimental evaluation is thorough. The authors test their method on two quite different reasoning datasets (one focused on multi-step math, another on logical/arithmetic reasoning with varying irrelevant information) using two model families (Qwen and LLaMA, with multiple model sizes). They compare against strong augmentation baselines – e.g. PP-Aug (premise permutation) and RC-Aug (reasoning chain augmentation from prior work) – rather than just against a vanilla model. This shows the authors have situated their work in context and are pushing beyond existing techniques. The results consistently favor BRIDGE, and the paper reports not just final accuracy but also the performance gain due to RL (“Delta by RFT” in tables), which directly measures how much RFT helped in each case. This is a appropriate metric given the paper’s focus. Additionally, the authors include ablation studies: they try injecting each behavior (reflection, subgoaling, analysis) in isolation to gauge their individual contributions. These ablations (illustrated in their Figures 4–6) reveal that each injected behavior does help (e.g. reflection consistently boosts final performance, while analysis primarily speeds up learning) and that the combination (BRIDGE) is most effective for robust gains. Such detailed empirical analysis strengthens confidence in the approach. The paper also measures proxy metrics like model perplexity and action entropy, showing that the injected behaviors indeed increased the model’s tendency to explore (higher entropy with reflection) and exploit (lower perplexity with step-by-step reasoning). Overall, the experiments are convincing and well-designed.\nClarity: The paper is well-written and structured in a logical manner. The problem context is explained succinctly (with relevant references to prior work on RL for LLMs and data augmentation). The introduction clearly states the motivation and contributions without overstating them. Technical sections are dense but generally understandable: key new terms like “data co-influence coefficient” are defined, and there’s an effort to provide intuition (e.g. Proposition 3.1 and the discussion around it give insight into how initial accuracy and cross-sample influence affect learning). The inclusion of figures (such as the overview diagram in Figure 1 and the influence visualization in Figure 3) helps the reader grasp the concepts. The supplementary material appears to provide useful extras (e.g. examples of the augmented behaviors in Appendix B.2 and training details in B.3), which shows the authors’ care in ensuring reproducibility and clarity. Minor points: there are a few notational heavy parts (the equations for influence may be hard to parse on a first read), but the authors do summarize key takeaways in plain language (e.g. end of Section 4.3, they summarize what Figures 3 showed). In sum, the clarity is sufficient for a NeurIPS submission, and the professional tone and organization make it accessible to the target audience (researchers familiar with LLMs and RL).\n\nWeakness: \n\nWhile the paper is strong overall, a few areas could be clarified or expanded to enhance impact and applicability:\n\nLimited Domain Scope: The experiments are limited to math and structured reasoning tasks with programmatically verifiable rewards. While well-chosen for controlled analysis, it remains unclear how well BRIDGE generalizes to domains like dialogue, code generation, or interactive environments where rewards may be noisier or more subjective. The authors acknowledge this and suggest it as future work, but even a discussion on how to adapt BRIDGE to such settings would improve confidence in its broader applicability.\n\nManual Behavior Design: The effectiveness of BRIDGE depends on hand-crafted behaviors (e.g., reflection, subgoaling) tailored to specific tasks. This may limit generalization unless clearer guidelines or semi-automated methods for behavior selection are provided. Including such guidance would improve the method’s transferability across domains.\n\nBase Model Dependence: BRIDGE assumes a reasonably capable base model that can already express chain-of-thought reasoning. It’s unclear how the method would perform with significantly weaker or much larger models. Some empirical or theoretical discussion on how model capacity interacts with the benefits of behavior injection would be helpful.\n\nEmpirical Scope and Metric Validity: While the experiments are strong, exploring how BRIDGE performs with other RL algorithms or with larger datasets could strengthen generality claims. Additionally, the use of approximate co-influence metrics is intriguing but could benefit from validation or sensitivity analysis, especially if intended as a general-purpose diagnostic tool.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful and constructive feedback. Here are our responses:\n- **W1 & Q1. How well does BRIDGE generalize to domains beyond math and structured reasoning tasks with programmatically verifiable rewards? How well does behavior injection generalize to more complex or interactive tool-use tasks beyond static QA or math problems?** \n  \n  Thank you for the insightful questions. BRIDGE injects behaviors based on a DAG representation. As long as an underlying graph structure can be identified (e.g., from the answer CoT), which holds for many reasoning tasks, our method can generalize beyond math or structured QA domains. However, BRIDGE is mainly designed to enhance a model’s ability to leverage reasoning behaviors, such as decomposition and reflection, rather than injecting domain-specific knowledge. Therefore, while it can be applied to other domains, the improvements generally may be less pronounced in tasks where domain expertise is the primary challenge instead of multi-step reasoning (e.g., the non-reasoning tasks).\n  \n  Regarding more complex or interactive tool-use tasks, we can still adapt our method by identifying a reasoning graph structure in the action sequence or interaction flow. While we believe exploring how BRIDGE operates in such interactive settings is an exciting direction, we consider it beyond the current scope and leave it to future work.\n\n- **W2. Manual Behavior Design: The effectiveness of BRIDGE depends on hand-crafted behaviors (e.g., reflection, subgoaling) tailored to specific tasks. This may limit generalization unless clearer guidelines or semi-automated methods for behavior selection are provided.**\n  \n  Thank you for highlighting this important point. While the behaviors may appear hand-crafted, they are in fact designed based on general exploration and exploitation principles from RL, making them broadly applicable beyond the specific tasks studied in this paper. Our ablation results in fig.5 further demonstrate that all these behaviors contribute to the model’s performance during RL training.\n  \n  Moreover, the specific behaviors we adopt such as reflection and subgoaling are grounded in well-established findings from cognitive science [1] and have also been shown to be effective in LLM finetuning [2]. We agree that developing more systematic or semi-automated methods for behavior selection would be a valuable direction for future work, and we see our current design as a first step toward that goal.\n\n- **W3. Base Model Dependence: BRIDGE assumes a reasonably capable base model that can already express chain-of-thought reasoning.** \n  \n  We respectfully disagree with the assumption that BRIDGE requires a base model with strong reasoning capabilities. In our experiments, we intentionally use pretrained base models (Qwen2.5, Llama3) rather than the reasoning-focused post-training model (e.g., Qwen2.5-math or DeepSeek-distilled model). In fact, BRIDGE is designed to teach reasoning behaviors to models that initially lack such capabilities. By injecting structured behaviors like subgoaling and reflection, our method guides the model toward improved reasoning during both SFT and RL stages. \n  \n  Our empirical results also validate: 1) The baseline models without behavior injection show limited gains from SFT and RL, indicating their initial lack of strong reasoning ability. 2) BRIDGE significantly boosts performance in the RL stage, demonstrating its effectiveness in enhancing reasoning in less capable base models.\n\n- **Q2. What’s the impact of the demonstration source or quality? Were they human-written, model-generated, or noisy? How robust is the method to imperfect traces?**\n    \n  The demonstrations used in our experiments (i.e., the vanilla CoT) are rule-based and programmatically generated as part of the iGSM and PromptBench datasets. These ground-truth traces are consistent across all compared methods, including BRIDGE, PP-Aug, and RC-Aug. More implementation details can be found in Appendix B.1 and the codes.\n\n  Importantly, the focus of BRIDGE is not to evaluate or depend on the inherent quality of the base demonstrations, but rather to inject structured reasoning behaviors (e.g., subgoaling, reflection) into them. These behaviors are applied systematically and can be incorporated even in the presence of imperfect or noisy demonstrations. The core research question we address is whether behavior injection can improve RL fine-tuning effectiveness, regardless of the initial demonstration source. We agree that studying the robustness of BRIDGE to imperfect or low-quality traces is an important direction. While this is not the primary focus of the current paper, we consider it valuable future work and appreciate the suggestion.\n\n- **Q3. How does behavior injection compare to prompting approaches like “think step-by-step” at inference time? Does fine-tuning offer clear advantages?**\n  \n  Yes, the finetuning shows clear advantages over prompt engineering. We evalute the accuracy on a iGSM test set (operation num=21~25) as follows\n  \n  | Model     | Base (no SFT) | Base + prompting (no SFT) | Random Guess | BRIDGE (SFT) | BRIDGE (RFT) |\n  | --------- | ------------- | ---------------------- | ------------ | ------------ | ------------ |\n  | Qwen 1.5B | <10.0          | <10.0                   | 14.2         | 38.6         | 89.4         |\n  | Llama 1B  | <10.0          | <10.0                   | 14.2         | 27.4         | 57.4         |\n  \n  The prompt engineering is not effective for synthetic questions as the base models are not familiar with this OOD tasks, which is even lower than random guess (random guess means the model answers \"0\" for all questions). As a comparison, the finetuning (SFT or RFT) shows much larger performance improvement.\n\n- **Q4. Are there cases where the model overuses tools unnecessarily? Any error analysis on when behavior injection might lead to misuse would be helpful.**\n\n  Based on our observations, the exploration behavior, particularly reflection, can indeed be overused by the SFT model trained with BRIDGE. In some cases, the model repeatedly attempts to solve an unlocked node without having computed all of its parent nodes, leading to output patterns such as:\n  ```\n  Then, let's denote the number of [Node] as y. But we haven't calculated the number of [one parent of the Node], thus the value of y is still unknown;\n  # repeat the above for several times\n  ```\n  This phenomenon is consistent with he reflection pattern injected during SFT, where the model learns to re-attempt unsolved steps. While such repetition reflects the intended exploration behavior, it can introduce errors and result in reduced performance at the SFT stage. However, as shown in fig. 5, although the model overuses reflection after SFT training, it will be corrected in further RL training with reward signals and those behaviors are finally beneficial to the performance after RL.\n\n- **Concerns on AI safety**\n  \n  Thanks for your comprehensive discussion on the potential misuse of BRIDGE that can lead to the violation of AI safety. We totally understand and agree with your concerns. We will provide more discussions of the potential negative impact in revision.\n\nWe sincerely appreciate your effort again and we hope our responses can address your concerns.\n\n[1] Polya, George. \"How to solve it.\" (1957).\n\n[2] Gandhi, Kanishk, et al. \"Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.\""}, {"review_id": "waPq8MHWzC", "reviewer": "Reviewer_SDEM", "summary": "This paper introduces, BRIDGE, an algorithm to add data to the SFT phase before performing RL training. Specifically, it converts question answer pairs to directed acyclic graph. Using the graph structure, the authors identify locked and unlocked nodes which are used to craft exploration and exploitation behaviors. They compare the BRIDGE algorithms to several baselines for multiple base models on 2 tasks. They use the notion of per-step influence function to analyze their algorithm and the baselines.", "questions": "* How do you extract DAG G = (V, E) from (q, a)?\n* How do you identify locked and unlocked nodes?\n* Could BRIDGE be applied throughout RL training and not only prior?", "limitations": "From my current understanding of the method, the limitations are not well described. For example, could BRIDGE be used on other math and common sense reasoning benchmarks such as GSM8k and MATH?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* Using per-step influence function to analyse RL training is quite interesting. I wish the authors analyzed this quantity in more details and throughout training.\n* The results on both iGSM and PromptBench are strong for multiple base models.\n\n## Weaknesses\n\n### Clarity\n\n* > 1: Extract DAG G = (V, E) from (q, a).\n\nIt is very unclear how to transform any question-answer pair into a DAG.\n\n* > We generate exploration behaviors by attempting to solve a locked (i.e., not yet solvable) node ahead of time, followed by reflection. For exploitation behaviors, we aggregate available information to solve an unlocked node (i.e.,solvable but not yet solved), reinforcing the known reasoning path\n\nIt is unclear how to identify locked and unlocked nodes\n\n* Figure 2: no legend for darker nodes.\n\n* Figure 3: why are the width of the bar not all the same?\n\n* The connection between per-step influence and BRIDGE is unclear.\n\n* Figure 3 Right should be discussed in more details.\n\n### Benchmarking\n\niGSM and prompt bench are both synthetic and uncommon benchmarks. Could the paper be extended to more common benchmarks such as GSM8k?", "was_revised": true, "final_justification": "The authors addressed my main concern that their method would not scale to non synthetic data. The demonstrated that their method could be applied to GSM8k and show great improvement. I would encourage the author to rewrite the paper to focus more on GSM8k than on the synthetic datasets.", "rebuttal": "We thank the reviewer for your review and feedback. Here are our responses:\n- **W1 & Q1. How do you extract DAG from QA**\n\n  Thanks for pointing it out. We provide more details about the extraction process here.\n  \n  For the datasets we used in this paper (iGSM and promptbench), all questions and answers are rule-based and follow a fixed sentence structure, which allows us to extract the DAG representation through simple **string matching**. In iGSM problem, each variable mentioned is treated as a node and the edge is the dependency between nodes. Consider a premise `The number of each Painting Room’s Printed Casual Backpack equals 9 more than the sum of each Painting Room’s Designer Bag and each Pottery Classroom’s Printed Casual Backpack.`. we view `each Painting Room’s Printed Casual Backpack` as a node. Since it depends on nodes `each Painting Room’s Designer Bag` and `each Pottery Classroom’s Printed Casual Backpack` and we draw edges from these two nodes to the former. By iterating all premises in this way, we construct the node set V and edge set E of the graph. We also provide a complete example in Appendix B.1.\n  \n  For other QA datasets without a fixed format, we can employ an oracle LLM (e.g., GPT4) to parse the node and edge from the QA. Specifically, we can view all intermediate variables / conclusions / corollaries as node and the edge is still their dependency. Here is an example prompt to extract (V, E):\n  ```\n  You are a helpful data analyst. You will be given a question-answer pair. Your task is to extract the DAG of the question. Here are some instructions for extracting the DAG:\n  - First separate the answer into multiple steps. If one step includes multiple intermediate variables / conclusions, further separate it until each step includes only one.\n  - Identify NODE in each step. The node can be intermediate variables / conclusions / corollary.\n  - Identify the dependency between the nodes.\n  - Remember that the final graph should be acyclic.\n  \n  [few-shot examples]\n  [QA]\n  [format instruction]\n  ```\n  Given a QA sample from MATH dataset:\n  ```\n  Question:\n  Define \\[ p = \\sum_{k = 1}^\\infty \\frac{1}{k^2} \\quad \\text{and} \\quad q = \\sum_{k = 1}^\\infty \\frac{1}{k^3}. \\]\n  Find a way to write \\[ \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3} \\] in terms of $p$ and $q$.\n  \n  Answer:\n  We count the number of times $\\frac{1}{n^3}$ appears in the sum \\[ \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3}\\], where $n$ is a fixed positive integer. (In other words, we are conditioning the sum on $j + k$). \n  We get a term of $\\frac{1}{n^3}$ each time $j + k = n.$ The pairs $(j,k)$ that work are $(1,n - 1),(2,n - 2),\\dots,(n - 1,1)$ for a total of $n - 1$ pairs. \n  Therefore, \n  \\begin{align*} \n  \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3} &= \\sum_{n = 1}^\\infty \\frac{n - 1}{n^3} \\\\ \n  &= \\sum_{n = 1}^\\infty \\left( \\frac{n}{n^3} - \\frac{1}{n^3} \\right) \\\\ \n  &= \\sum_{n = 1}^\\infty \\left( \\frac{1}{n^2} - \\frac{1}{n^3} \\right) \\\\ \n  &= \\sum_{n = 1}^\\infty \\frac{1}{n^2} - \\sum_{n = 1}^\\infty \\frac{1}{n^3} \\\\ \n  &= \\boxed{p - q}. \n  \\end{align*}\n  ```\n  The extract result is:\n  ```\n  Node names:\n  node 1: ( S = \\sum_{j = 1}^\\infty \\sum_{k = 1}^\\infty \\frac{1}{(j + k)^3} )\n  node 2: Number of pairs ((j, k)) with (j + k = n) is ( n-1 )\n  node 3: ( S = \\sum_{n=1}^\\infty \\frac{n-1}{n^3} )\n  node 4: ( \\frac{n-1}{n^3} = \\frac{1}{n^2} - \\frac{1}{n^3} )\n  node 5: ( S = \\sum_{n=1}^\\infty \\left(\\frac{1}{n^2} - \\frac{1}{n^3}\\right) )\n  node 6: ( S = \\sum_{n=1}^\\infty \\frac{1}{n^2} - \\sum_{n=1}^\\infty \\frac{1}{n^3} )\n  node 7: ( S = p - q )\n  \n  Dependency list:\n  node 1: []\n  node 2: []\n  node 3: [1, 2]\n  node 4: []\n  node 5: [3, 4]\n  node 6: [5]\n  node 7: [6]\n  ```\n  We will update the details and make it more clear in revision.\n\n- **W2 & Q2. How do you identify locked and unlocked nodes?**\n  \n  With the extracted DAG, we construct the solution step by step while keeping track of the solved nodes. A locked node is one whose parent nodes have not all been solved yet. In contrast, an unlocked node is a node that has not yet been solved, but all of its parent nodes are already resolved.\n  \n- **W3. Figure 2: no legend for darker nodes.**\n  \n  The blue node generally indicates any non-target node. The dark blue node indicates the intermediate node that has parent node, while the light blue indicates the source node that does not have parent node.\n  \n- **W4. Figure 3: why are the width of the bar not all the same?**\n\n  The bars differ in width because the middle bar represents all rollouts with accuracy in the range $\\in (0, 1)$, which covers multiple discrete accuracy $1/8, ..., 7/8$ and is therefore wider than the left bar (accuracy = 0) and right bar (accuracy = 1). We highlight the $(0, 1)$ interval separately because these rollouts have non-zero advantages and per-step influence. This visualization emphasizes the distribution and impact of partially correct rollouts on the model’s improvement.\n  \n- **W5 & W6. The connection between per-step influence and BRIDGE is unclear. Figure 3 Right should be discussed in more details.**\n  \n  Thanks for bringing up the concerns. Let's begin with fig.3 right. The computation of per-step influence is briefly described in Lines 250–259. We provide additional details:\n  - Begin by 2,000 queries. For each query, we generate 8 rollouts using the model, resulting in a total of 16,000 query–output pairs. Each query is then assigned to an accuracy group based on the accuracy of its rollouts.\n  - For each query-output pair, we compute 1) advantage, and 2) the policy gradient $\\nabla_\\theta \\log \\pi_\\theta (o|q)$.\n  - Using Eq.(5), we estimate the per-step influence for each query $q$. In this formulation, $(q', o')$ ranges over all 16,000 query–output pairs.\n  - We then group queries by their accuracy (0/8, 1/8, ..., 8/8), and compute the average per-step influence for each group. These results are visualized in fig.(3) right. Notably, the groups with 0/8 and 8/8 accuracy have zero per-step influence because the corresponding advantages are zero under the GRPO formulation. \n  - Since computing the inner product $\\mathcal{K}_\\theta[.,.]$ is computationally intractable, we apply dimension reduction following previous work. More details are in Appendix B.6.\n  \n  The results demonstrate that BRIDGE increases the per-step influence, contributing to more effective RL updates and explains the improved performance we observe after fine-tuning.\n\n  For the first concern \"The connection between per-step influence and BRIDGE is unclear\", we respectfully disagree with it. As we derive in line 107~109, the per-step influence measures how model performance changes after training on a specific query with its outputs. We evaluate the per-step influence of different SFT models and present the experiment results in fig.3 right. The results clearly show that BRIDGE consistently exhibits higher per-step influence. This directly supports our claim in **sec.3.2 takeaway** that BRIDGE provides more effective learning signals during RL training, resulting in larger performance gains compared to other baselines.\n  \n- **W7. iGSM and prompt bench are both synthetic and uncommon benchmarks. Could the paper be extended to more common benchmarks such as GSM8k?**\n  \n  Yes. To extend to GSM8K, we apply our DAG extraction method to 2,000 examples from the GSM8K training set. Using the extracted DAGs, we leverage an oracle LLM (GPT4.1) to augment the data by injecting information analysis and reflection behaviors into the answer CoT. We then do SFT on Qwen2.5-1.5B and Llama3.2-1B models for 5 epochs and use the remaining training data for RL (150 steps). The results are listed below:\n  | Base model|  | Vanilla | BRIDGE   |\n  |---| --- | --- | -------- |\n  | Qwen 1.5B | SFT  | 55.8    | 61.6 |\n  |   | RFT      | 63.5 | **77.1** |\n  |   | $\\Delta$ | 7.7  | **15.5** |\n  | Llama 1B  | SFT | 44.2    | 46.8 |\n  |   | RFT      | 48.6 | **59.6** |\n  |   | $\\Delta$ | 4.4  | **12.8** |\n  \n  Moreover, we want to emphasize that while evaluating on real-world benchmarks like GSM8K is important, synthetic benchmarks offer unique advantages as testbeds for RL research: 1) Synthetic benchmarks focus on to structured reasoning and problem-solving ability, minimizing the confounding effects of background knowledge, which is required by many public benchmarks but ususally lacked by some LLMs. 2) There is a severe data contamination of public benchmarks in many LLMs, as suggested by recent works [1][2], while the synthetic data fundamentally prevent it.\n  \n- **Q3. Could BRIDGE be applied throughout RL training and not only prior?**\n  \n  Thank you for this suggestion. It could be a promising direction for future research. While our current work uses BRIDGE prior to RL, we agree it could be applied iteratively during training. For example, one could periodically pause RL, use the partially trained model to generate improved reasoning traces, and then apply BRIDGE to synthesize new training examples for an intermediate round of supervised fine-tuning before resuming RL (which is also mentioned in Deepseek R1 paper [3]). We believe such iterative training can be another learning paradigm to further boost performance and improve sample efficiency. We will add a discussion of this potential extension in the revised manuscript.\n\nWe hope our detailed responses and new experimental results have fully addressed the reviewer's concerns. We would be grateful for their reconsideration of our manuscript in light of these clarifications.\n  \n[1] Shao, Rulin, et al. \"Spurious rewards: Rethinking training signals in rlvr.\"\n\n[2] Wu, Mingqi, et al. \"Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.\"\n\n[3] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\""}]

Author rebuttal:

Meta-review: This paper tackles the open question of why reinforcement fine-tuning (RFT) benefits some language models but not others, and proposes a simple yet powerful solution: behavior injection (BRIDGE), a data-centric augmentation that seeds exploration and exploitation behaviors during supervised fine-tuning to make models more “RL-ready.” The reviewers appreciated the combination of careful analysis (identifying rollout accuracy and data co-influence as key factors in RFT effectiveness) with a practical method that is broadly applicable. Empirical results across math and logical reasoning benchmarks show consistent and significant improvements over strong augmentation baselines, with ablations confirming the role of injected behaviors. While some reviewers noted that the experiments are focused on relatively narrow domains and the theoretical analysis is limited in scope, the authors clarified these concerns in discussion and positioned their method as a general data-centric approach. Overall, the paper offers both useful conceptual insights and strong empirical gains, and I recommend acceptance.
Decision: Accept (poster)