Title: Behavior Injection: Preparing Language Models for Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Large Language Model

Data augmantation
Abstract: Reinforcement learning (RL) has emerged as a powerful post-training technique to incentivize the reasoning ability of large language models (LLMs). However, LLMs can respond very inconsistently to RL finetuning: some show substantial performance gains, while others plateau or even degrade. To understand this divergence, we analyze the per-step influence of the RL objective and identify two key conditions for effective post-training: (1) RL-informative rollout accuracy, and (2) strong data co-influence, which quantifies how much the training data affects performance on other samples. Guided by these insights, we propose behavior injection, a task-agnostic data augmentation scheme applied prior to RL. Behavior injection enriches the supervised finetuning (SFT) data by seeding exploratory and exploitative behaviors, effectively making the model more RL-ready. We evaluate our method across two reasoning benchmarks with multiple base models. The results demonstrate that our theoretically motivated augmentation can significantly increase the performance gain from RL over the pre-RL model.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces, BRIDGE, an algorithm to add data to the SFT phase before performing RL training. Specifically, it converts question answer pairs to directed acyclic graph. Using the graph structure, the authors identify locked and unlocked nodes which are used to craft exploration and exploitation behaviors. They compare the BRIDGE algorithms to several baselines for multiple base models on 2 tasks. They use the notion of per-step influence function to analyze their algorithm and the baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* Using per-step influence function to analyse RL training is quite interesting. I wish the authors analyzed this quantity in more details and throughout training.\n* The results on both iGSM and PromptBench are strong for multiple base models.\n\n## Weaknesses\n\n### Clarity\n\n* > 1: Extract DAG G = (V, E) from (q, a).\n\nIt is very unclear how to transform any question-answer pair into a DAG.\n\n* > We generate exploration behaviors by attempting to solve a locked (i.e., not yet solvable) node ahead of time, followed by reflection. For exploitation behaviors, we aggregate available information to solve an unlocked node (i.e.,solvable but not yet solved), reinforcing the known reasoning path\n\nIt is unclear how to identify locked and unlocked nodes\n\n* Figure 2: no legend for darker nodes.\n\n* Figure 3: why are the width of the bar not all the same?\n\n* The connection between per-step influence and BRIDGE is unclear.\n\n* Figure 3 Right should be discussed in more details.\n\n### Benchmarking\n\niGSM and prompt bench are both synthetic and uncommon benchmarks. Could the paper be extended to more common benchmarks such as GSM8k?\n\nQUESTIONS:\n* How do you extract DAG G = (V, E) from (q, a)?\n* How do you identify locked and unlocked nodes?\n* Could BRIDGE be applied throughout RL training and not only prior?\n\nLIMITATIONS:\nFrom my current understanding of the method, the limitations are not well described. For example, could BRIDGE be used on other math and common sense reasoning benchmarks such as GSM8k and MATH?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates why reinforcement fine-tuning (RFT) sometimes yields large gains for large language models (LLMs) and sometimes fails to help. The authors identify two key conditions that make a model RL-ready: (1) an intermediate rollout accuracy (the base model’s initial success rate on task rollouts should be neither too low nor too high) and (2) a strong data co-influence (training on one sample should generalize to others, indicating the model can propagate learned knowledge). To prepare an LLM to better benefit from RFT, the paper proposes Behavior Injection, a data-centric augmentation strategy called BRIDGE. The idea is to enrich the supervised fine-tuning (SFT) data with behaviors that encourage exploration and exploitation before any RL training. Concretely, the authors augment the model’s chain-of-thought demonstrations with additional reasoning steps: subgoal computation (explicitly breaking down problems into intermediate steps), information analysis (aggregating relevant info in reasoning), and reflection (exploring an alternative path or revisiting a sub-problem). By seeding these behaviors in the SFT phase, the base model produces more informative and varied trajectories during RL, which in turn leads to larger performance improvements from the RL fine-tuning. The paper evaluates BRIDGE on two challenging reasoning benchmarks (grade-school math iGSM and the arithmetic reasoning subset of PromptBench), using multiple base models of different sizes. In experiments, models prepared with behavior injection consistently achieve significantly higher gains from RFT compared to baseline approaches (including no augmentation, premise shuffling, and existing chain-of-thought augmentations). For example, BRIDGE-prepared models not only start with solid performance but after RL fine-tuning they outperform others by a large margin on both in-distribution and out-of-distribution test sets. Overall, the paper’s contributions include a theoretical analysis of RL dynamics for LLMs, a novel data augmentation method to improve those dynamics, and empirical evidence that this method makes LLMs more amenable to reinforcement learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nQuality (analysis): The paper provides a thoughtful theoretical analysis of what drives learning in the RL fine-tuning stage. It introduces a per-step influence framework that isolates how rollout accuracy and a co-influence coefficient contribute to gradient signal during RFT. This analysis yields an intuitive insight: if a model gets every training query either entirely correct or entirely wrong, the RL reward signal is uninformative (gradient is zero in those extreme cases). The most learning happens when the model’s initial answers are partially correct. This perspective is well-grounded and novel in the LLM context, and it directly motivates the proposed solution (shape the SFT model such that its knowledge is incomplete but improvable under RL). The analytical findings are clearly connected to the method, lending a strong foundation to the paper’s approach.\nOriginality: The idea of “behavior injection” to make an LLM more RL-ready is innovative. Rather than modifying the RL algorithm or reward scheme, the authors take a data-centric approach: they augment the training data to instill certain behaviors before performing RL. While data augmentation for better reasoning is not new, this work is the first (to my knowledge) to explicitly inject exploratory and exploitative reasoning patterns to facilitate subsequent policy optimization. This behavior-level augmentation goes beyond simply adding more data or shuffling it – it changes how the model approaches problems (e.g. sometimes pausing to reflect or breaking problems into subparts). This approach of prepping a model for RL by curating its reasoning habits is a fresh contribution and could inspire other “RL-preparation” techniques.\nSignificance: The ability to improve an LLM’s performance via RL with less hassle is quite significant. Many important applications of LLMs (e.g. solving complex math problems, planning, tool use) rely on reinforcement fine-tuning for alignment or enhanced reasoning. However, RFT can be unstable or yield minimal gains if the base model isn’t already in a good state. This paper’s findings and method address that bottleneck. Empirically, the gains reported are impressive: for instance, on the iGSM math benchmark, the BRIDGE-augmented models see large accuracy jumps after RL (up to ~30-50% absolute improvement over their pre-RL accuracy, far exceeding baseline gains). Importantly, these gains hold on both seen (in-distribution) and harder (out-of-distribution) problems, indicating robustness. Such improvements suggest that BRIDGE could make RL fine-tuning a more routinely effective tool, potentially impacting how practitioners approach alignment and domain adaptation of LLMs. Even if the method is demonstrated on math and logic tasks here, the general idea could be applicable to a wide range of reasoning-heavy domains.\nQuality (experiments): The experimental evaluation is thorough. The authors test their method on two quite different reasoning datasets (one focused on multi-step math, another on logical/arithmetic reasoning with varying irrelevant information) using two model families (Qwen and LLaMA, with multiple model sizes). They compare against strong augmentation baselines – e.g. PP-Aug (premise permutation) and RC-Aug (reasoning chain augmentation from prior work) – rather than just against a vanilla model. This shows the authors have situated their work in context and are pushing beyond existing techniques. The results consistently favor BRIDGE, and the paper reports not just final accuracy but also the performance gain due to RL (“Delta by RFT” in tables), which directly measures how much RFT helped in each case. This is a appropriate metric given the paper’s focus. Additionally, the authors include ablation studies: they try injecting each behavior (reflection, subgoaling, analysis) in isolation to gauge their individual contributions. These ablations (illustrated in their Figures 4–6) reveal that each injected behavior does help (e.g. reflection consistently boosts final performance, while analysis primarily speeds up learning) and that the combination (BRIDGE) is most effective for robust gains. Such detailed empirical analysis strengthens confidence in the approach. The paper also measures proxy metrics like model perplexity and action entropy, showing that the injected behaviors indeed increased the model’s tendency to explore (higher entropy with reflection) and exploit (lower perplexity with step-by-step reasoning). Overall, the experiments are convincing and well-designed.\nClarity: The paper is well-written and structured in a logical manner. The problem context is explained succinctly (with relevant references to prior work on RL for LLMs and data augmentation). The introduction clearly states the motivation and contributions without overstating them. Technical sections are dense but generally understandable: key new terms like “data co-influence coefficient” are defined, and there’s an effort to provide intuition (e.g. Proposition 3.1 and the discussion around it give insight into how initial accuracy and cross-sample influence affect learning). The inclusion of figures (such as the overview diagram in Figure 1 and the influence visualization in Figure 3) helps the reader grasp the concepts. The supplementary material appears to provide useful extras (e.g. examples of the augmented behaviors in Appendix B.2 and training details in B.3), which shows the authors’ care in ensuring reproducibility and clarity. Minor points: there are a few notational heavy parts (the equations for influence may be hard to parse on a first read), but the authors do summarize key takeaways in plain language (e.g. end of Section 4.3, they summarize what Figures 3 showed). In sum, the clarity is sufficient for a NeurIPS submission, and the professional tone and organization make it accessible to the target audience (researchers familiar with LLMs and RL).\n\nWeakness: \n\nWhile the paper is strong overall, a few areas could be clarified or expanded to enhance impact and applicability:\n\nLimited Domain Scope: The experiments are limited to math and structured reasoning tasks with programmatically verifiable rewards. While well-chosen for controlled analysis, it remains unclear how well BRIDGE generalizes to domains like dialogue, code generation, or interactive environments where rewards may be noisier or more subjective. The authors acknowledge this and suggest it as future work, but even a discussion on how to adapt BRIDGE to such settings would improve confidence in its broader applicability.\n\nManual Behavior Design: The effectiveness of BRIDGE depends on hand-crafted behaviors (e.g., reflection, subgoaling) tailored to specific tasks. This may limit generalization unless clearer guidelines or semi-automated methods for behavior selection are provided. Including such guidance would improve the method’s transferability across domains.\n\nBase Model Dependence: BRIDGE assumes a reasonably capable base model that can already express chain-of-thought reasoning. It’s unclear how the method would perform with significantly weaker or much larger models. Some empirical or theoretical discussion on how model capacity interacts with the benefits of behavior injection would be helpful.\n\nEmpirical Scope and Metric Validity: While the experiments are strong, exploring how BRIDGE performs with other RL algorithms or with larger datasets could strengthen generality claims. Additionally, the use of approximate co-influence metrics is intriguing but could benefit from validation or sensitivity analysis, especially if intended as a general-purpose diagnostic tool.\n\nQUESTIONS:\n1. How well does behavior injection generalize to more complex or interactive tool-use tasks beyond static QA or math problems?\n\n2. What’s the impact of the demonstration source or quality? Were they human-written, model-generated, or noisy? How robust is the method to imperfect traces?\n\n3. How does behavior injection compare to prompting approaches like “think step-by-step” at inference time? Does fine-tuning offer clear advantages?\n\n4. Are there cases where the model overuses tools unnecessarily? Any error analysis on when behavior injection might lead to misuse would be helpful.\n\nLIMITATIONS:\nThe authors explicitly discuss some limitations and potential negative impacts, which is commendable. They acknowledge that the scope of evaluation is limited to math and common-sense reasoning tasks, and suggest that future work should extend BRIDGE to broader domains (e.g. more complex agent environments) and with larger datasets. I agree with this assessment – the method’s current validation, while thorough for what it is, doesn’t cover all possible use cases. Another limitation implicitly noted is that the approach thus far is demonstrated on tasks with ground-truth verifiable rewards; tasks involving subjective or human-provided rewards are not directly tested, which could be addressed in the future. The authors also mention a potential negative societal impact: the misuse of their method. Since behavior injection is a data augmentation technique, a malicious actor could in principle inject undesirable behaviors into a model (for example, tendencies towards biased, unethical, or unsafe outputs) and then amplify them via RL. This is a realistic concern – if someone “prepares” a model with harmful behaviors and then optimizes for them with RL, it could yield a more capable but harmful agent. The paper advises caution here, which is appropriate. Mitigating this risk would involve careful curation of behaviors and perhaps oversight during RL (an issue that overlaps with general AI safety considerations). On the positive side, the method is aimed at improving reasoning and alignment with correct answers, which is a beneficial goal; there is nothing inherently dangerous in BRIDGE itself if used responsibly. One limitation not explicitly discussed in the paper (but worth noting) is the manual effort or expertise required to design injected behaviors. As raised above, BRIDGE currently relies on human intuition to decide what constitutes a helpful exploratory or exploitative behavior for a task. This could be seen as a form of expert data augmentation. If a user of this method does not have a good understanding of the task’s solution structure, they might struggle to apply BRIDGE effectively. While not a dire flaw, this is a practical consideration: the need for domain-specific behavior engineering could limit adoption or require further research (perhaps the authors could mention this as an area for future improvement). Aside from that, I did not identify major overlooked issues. The paper is quite conscientious in evaluating its own scope. Computational cost is reasonably addressed (the method doesn’t drastically increase training cost, aside from some overhead to generate augmented data and compute influence metrics for analysis). The ethical considerations are relatively mild for this work (since it’s about improving model training, not directly producing problematic content). The mention of misuse covers the main worry. In sum, the authors have adequately discussed limitations and societal impacts, and I appreciate their transparency. Any omissions are minor and can be tackled in subsequent research or the rebuttal.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents BRIDGE (BehavioR Injection Data auGmEntation), a method to prepare language models for more effective reinforcement learning fine-tuning (RFT) by injecting exploration and exploitation behaviors during supervised fine-tuning (SFT). The core contributions of this paper are 1) analysis of reinforcement learning behavior and its correlation with rollout data; 2) proposed data augmentation strategy BRIDGE; 3) empirical evaluation. The authors identify two critical factors influencing RFT performance: rollout accuracy and data co-influence. Guided by this analysis, BRIDGE enhances SFT datasets by injecting exploration and exploitation behaviors, making models \"RL-ready\" and improving their learning efficiency and generalization in the RFT stage.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Pre-RL preparation is important. Many prior literature [1,2] pin-point this but few works delve deep into the direction.\n- The idea is well-motivated by theory\n- Data efficiency. This paper shows a data-efficient preparation approach to incentivize language models' potentials in RL training, where the proposed strategy requires 2000/5000 data samples for two tasks according to the appendix.\n- Remarkable empirical improvements.\n\n\nWeakness:\n- The general impact is upper-bounded by the scalability of DAG representation.\n- The theoretical analysis is specific to GRPO, where the coefficient $\\sqrt{\\alpha(1-\\alpha)}$ depends on GRPO's advantage estimation. It is unclear how the theory intuition will changes on other RL algorithms\n- It is unclear how the training epochs will affect the BRIDGE performance. As shown in table 1-2, different epochs affect the final performance a lot but the effect of more training epochs is not presented.\n\n\n[1] Guo, Daya, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n[2] Wang, Zengzhi, Fan Zhou, Xuefeng Li, and Pengfei Liu. \"Octothinker: Mid-training incentivizes reinforcement learning scaling.\" arXiv preprint arXiv:2506.20512 (2025).\n\nQUESTIONS:\n- Why does BRIDGE yield quite low accuracy on SFT stage (Table 2)?\n- Do authors scale up to 7B/32B models?\n- How does the theory apply to other RL algorithms?\n- What's the effect of training on BRIDGE-construcrted for more epoch? Will it affect the generalizability of the model?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes BehavioR Injection Data auGmEntation (BRIDGE), a strategy designed to augment the training data for SFT in order to improve the performance of LLMs trained via RL. Since RL training is typically preceded by SFT, BRIDGE enhances the SFT dataset by constructing examples that incorporate both exploration and exploitation behaviors, thereby better preparing LLMs for subsequent RL training. Experiments are conducted on two families of LLMs: Qwen and Llama, and evaluated on two benchmarks: iGSM and PromptBench. The results show that LLMs trained on datasets augmented with BRIDGE achieve significant performance improvements after RL training, outperforming models trained with other data augmentation strategies.\n\nSTRENGTHS AND WEAKNESSES:\n> **Strengths**\n\n1. BRIDGE is motivated by per-step influence analysis, which highlights two key factors for enhancing the performance of LLMs: (1) rollout accuracy and (2) the co-influence coefficient between training and target data.\n\n2. Experimental results show that LLMs trained on data augmented with BRIDGE achieve exceptional performance after RL training.\n\n3. In addition to experiments demonstrating the effectiveness of BRIDGE-augmented data, the paper includes ablation studies that analyze various injected behaviors. These studies provide valuable insights into constructing SFT datasets to improve LLM performance following RL training.\n\n> **Weaknesses**\n\n1. The LLMs used in the experiments have no more than 3 billion parameters. It would be preferable to include experimental results on more widely used LLMs in academic research, such as those with approximately 7 billion parameters.\n2. Considering that the Adam optimizer is predominantly used for training LLMs, the mathematical formulation for estimating the per-step influence of q on model performance may lack accuracy.\n\nQUESTIONS:\n1. Could you provide more details on how to extract the DAG representation from a query and its corresponding response?\n\n2. As shown in Table 2, it seems that the experimental results for Qwen2.5-3B are missing from PromptBench.\n\n3. Could you explain why the injected behaviors influence the accuracy distribution of the rollout samples?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}