Sample-efficient LLM Optimization with Reset Replay

## Abstract

Abstract Recent advancements in post-training Large Language Models (LLMs), particularly through Reinforcement Learning (RL) and preference optimization methods, are key drivers for enhancing their reasoning capabilities.
However, these methods are often plagued by low sample efficiency and a susceptibility to primacy bias, where overfitting to initial experiences degrades policy quality and damages the learning process.
To address these challenges, we introduce LLM optimization with Reset Replay (LoRR), a general and powerful plugin designed to enhance sample efficiency in any preference-based optimization framework.
LoRR’s core mechanism enables training at a high replay number, maximizing the utility of each collected data batch.
To counteract the risk of overfitting inherent in high-replay training, LoRR incorporates a periodic reset strategy with reusing initial data, which preserves network plasticity.
Furthermore, it leverages a hybrid optimization objective, combining supervised fine-tuning (SFT) and preference-based losses to further bolster data exploitation.
Our extensive experiments demonstrate that LoRR significantly boosts the performance of various preference optimization methods on both mathematical and general reasoning benchmarks.
Notably, an iterative DPO approach augmented with LoRR achieves comparable performance on challenging math tasks, outperforming some complex and computationally intensive RL-based algorithms.
These findings highlight that LoRR offers a practical, sample-efficient, and highly effective paradigm for LLM finetuning, unlocking greater performance from limited data.

## 1 Introduction

Recent advancements in large language models (LLMs) highlight the potential to enhance reasoning capabilities by leveraging human feedback (Leike et al., 2018).
The widely used factor driving this progress is Reinforcement Learning (RL) from human feedback (Ouyang et al., 2022), in which more recent RL approaches assign verifiable rewards (Cui et al., 2025; Ma et al., 2025) based on whether the model’s output matches a ground-truth.
On the other hand, offline methods like Direct Preference Optimization (DPO) (Rafailov et al., 2023) and SimPO (Meng et al., 2024) have streamlined this process by training policies directly from preference data, bypassing the need for an explicit reward model.
However, a fundamental limitation of applying these generalized RL algorithms to real-world tasks is their low sample efficiency, requiring an extensive number of interactions with the dataset from model rollouts, to learn satisfactory policies through model sampling and reward filtering.
At the same time, they may be susceptible to primacy bias (Nikishin et al., 2022), a tendency to overfit initial training experiences.
Because of experience provided by the same dataset rather than interactions with the environment, this early overfitting can degrade the policy’s quality, crippling the learning process.
Together, the demand for vast amounts of data and the risk of policy degradation from overfitting present a core challenge for effective LLM optimization.

Despite the attractiveness of sampling-efficient optimization algorithms for finetuning LLMs in real-world tasks, few works have explored this topic.
For example, recent work (Xia et al., 2024; Zhou et al., 2023) has emphasized that data quality is more crucial than quantity.
However, the training framework has remained mostly unchanged.
In each training iteration of on-policy optimization, a sample is gathered by LLM rollouts based on the current policy (Zhang et al., 2025; Tu et al., 2025).
For convergence to be achieved, existing methods typically require collecting a large number of samples and performing reward validation, which renders sample efficiency a well-known problem.

In traditional RL, a line of work (Nikishin et al., 2022; D’Oro et al., 2023; Xu et al., 2024)
solves this concept by scaling the number of replays, which can remarkably quicken the convergence process and boost the final performance.
This finding offers the understanding that the data collected can be utilized more effectively through a greater number of parameter-update steps.
While none of the prior works make efforts to enable the updating of LLMs at a high replay number, meaning training the LLMs multiple times using the collected experiences for periodic interactions with a dataset.
When compared with RL, LLM optimization has parameter updates that occur less frequently.
As a result, there is a greater chance in LLM that the dataset may not be fully utilized, either by supervised fine-tuning (SFT) or RL-based optimization.
If an LLM fails to learn adequately from the current data, it will be unable to roll out new high-quality data, which in turn hurts sample efficiency.

To address the data exploitation issue mentioned
above, we propose an algorithm called LLM optimization with Reset Replay (LoRR) into finetuning processes.
This is a powerful plugin that can enhance sample efficiency on any preferred optimization method.
LoRR is built upon three essential elements: a high-replay training setting, a reset strategy, and a hybrid optimization technique.
First, in contrast to existing on-policy sampling, our algorithm leverages a replay buffer where the sampled experiences are more diverse.
This enhanced data diversity is critical for boosting the sample efficiency of fine-tuning by maximizing data reuse with mixing experiences. Capitalizing on this principle, we implement a high-replay-ratio finetuning process, whereby an LLM is updated multiple times per interactive batch.
Second, learning at a high replay number would make LLMs incur a risk of overfitting to the primacy bias, thereby negatively impairing future learning (Nikishin et al., 2022). To overcome this overfitting phenomenon, we introduce a Shrink & Perturb strategy (Ash & Adams, 2020) to periodically reset the network parameters of the LLM, a procedure that preserves the network plasticity critical for continual learning (Sokar et al., 2023; Lyle et al., 2023).
Furthermore, given that LoRR updates LLM networks multiple times per batch, we bolster data efficiency by hybridizing SFT and preference-based optimization.
Surprisingly, this unique synergy of components achieves a significant advance, enabling LLM finetuning at a replay number an order of magnitude greater than that reported in previous literature.

The contributions of this paper are concluded as follows:

- •
We provide demonstrations of the existence of the primacy bias in LLM optimization, and show the impact of different experiences on the training process, laying the foundation for mixing rollouts and hybrid optimization.
- •
We propose a mechanism LoRR for alleviating overfitting and enhancing sample efficiency, by periodically training with a high replay number and then resetting a part of the LLM;
- •
Our experiments show that LoRR significantly improves performance when applied to preference methods and can outperform existing complex RL-based algorithms under the same conditions for multi-round improvements.

## 2 Related Works

Sample Efficiency for RL.
A central goal in traditional RL is to improve sample efficiency, as acquiring experience through environment interaction is often costly and time-consuming (Chen et al., 2021; D’Oro et al., 2023; Schwarzer et al., 2023).
A prominent strategy for achieving this is to maximize the utility of collected data by increasing the replay number, also known as the update-to-data ratio.
Despite offering little benefit when applied to standard baselines (Fedus et al., 2020), replay number scaling has been observed to enhance the performance of already well-optimized algorithms.
This refers to the number of gradient updates performed on an agent’s parameters for each new data point collected (Nikishin et al., 2022; D’Oro et al., 2023).
The underlying principle is that an agent can gain more knowledge by repeatedly learning from previously collected experiences, thus reducing the demand for new samples and accelerating learning (Schwarzer et al., 2023).
However, while appealing in principle, naively scaling the replay ratio introduces a critical stability issue, overfitting (Nikishin et al., 2022; Lyle et al., 2023; Sokar et al., 2023).
From earlier experiences when training, it results in losing the network plasticity to learn good policies in the following learning process.
Researchers (D’Oro et al., 2023) further raise the algorithm, using the shrink and perturbation instead of the reset to maintain the network plasticity.
While high replay with reset is becoming an active area of investigation in single-/multi-agent RL, this topic remains underexplored within finetuning LLMs, especially as more and more RL algorithms (Rafailov et al., 2023; Cui et al., 2025) are being applied to LLMs.
This gap motivates our work to develop a framework that can harness the benefits of a high replay number while explicitly mitigating the associated risks in the LLM optimization context.

RL for LLM Reasoning.
RL has been extensively employed in the context of LLMs primarily for aligning models with human preferences (Ouyang et al., 2022; Meng et al., 2024), also in some applications like jailbreaking (Liu et al., 2024).
Recently, a growing body of work investigates the amplification of mathematical reasoning in open-source language models, particularly the Qwen2.5 family, through RL (Zeng et al., 2025; Yan et al., 2025; Ma et al., 2025; Cui et al., 2025).
Early efforts (Zeng et al., 2025; Ma et al., 2025) employed explicit, verifiable rewards demonstrating notable performance gains on math benchmarks.
Subsequent research (Wang et al., 2025b) focused on data efficiency, enabling effective learning even from minimal labeled or unlabeled examples.
However, the broader generalizability of these observed gains, particularly within mathematical domains, has faced critical scrutiny. For example, performance enhancements noted on Qwen2.5, even when driven by random or inaccurate rewards, did not consistently translate to other LLMs, suggesting model-specific idiosyncrasies (Zhao et al., 2025).
Consequently, while RL may offer a potent tool for specialized LLM enhancement, its successful deployment necessitates a nuanced understanding of its interaction with specific learning dynamics.
Our work aims to enhance the sample efficiency of RL to improve the general learning ability of LLMs, extending beyond mere mathematical benchmarks.

Self-improvement of LLMs primarily occurs during the training phase, where models enhance their performance by evaluating and refining their own outputs without relying on external human supervision (Kumar et al., 2025). However, a prevailing view suggests that LLMs cannot achieve self-improvement without any external information (Huang et al., 2025). Consequently, an increasing number of methods, especially RL-based, leverage external supervisory signals to facilitate self-improvement, such as outcome labels (Tu et al., 2025; Zhang et al., 2025), value functions (Wang et al., 2024a), and process reward models (Cui et al., 2025). Our approach is based on self-improvement, and it keeps training and partially resets the data and model for each batch.

## 3 The Primacy Bias in LLM Optimization

This section examines how initial training phases can disproportionately affect the LLM finetuning process due to primacy bias.
To start, let us review how previous work (Nikishin et al., 2022; Zhou et al., 2022) defined the primacy bias in deep reinforcement learning as follows:

###### Definition 1 .

A tendency to overfit initial experiences that damages the rest of the learning process.

In the context of LLM finetuning, current optimization paradigms often rely on RL or preference-based rewards and collect data via off-policy methods.
Typically, a policy periodically rolls out several responses and selects preferences based on the reward verifier.
This makes us curious whether there will be primacy bias in finetuning.
For this, we present two experiments focused on the LLM finetuning, aiming to illustrate the phenomenon’s existence in improper learning from early data.
Firstly, we demonstrate that over-training an LLM exclusively on its initial data can irreparably damage the rest of the learning process.
Secondly, our experiments reveal that the applicability of the initial experience of LLMs is uncertain during the training process.

Heavy Priming Causes LLM Overfitting. The extent to which LLM finetuning relies on its initial data affects its learning process.
In order to explicitly reveal the impact of primacy bias in LLM finetuning, we investigate an extreme case of over-reliance on early data: Does overfitting to just one batch of initial data destroy the generalization of an LLM?
To this end, we design a set of experiments to optimize preferences over math datasets based on Llama-3B.
As shown in Figure [1](https://arxiv.org/html/2508.06412v2#S3.F1), we start by using the default hyperparameters for pure DPO (red line), which means each batch gets one update per step.
Then, we set up an identical version of DPO with heavy priming mode (blue line), where we first train the LLM with the first batch 200 times. Only after this extensive initial priming do we resume standard training.
The results show that even after training on all the data, the LLM affected by heavy priming was unable to recover from the initial overfitting in the math domain.
Besides, both normal training and overfitting have more or less impact on other domain-specific performance, such as health and psychology, suggesting that knowledge of the original model is being forgotten (Ibrahim et al., 2024; Huan et al., 2025).
This finding strongly suggests that the primacy bias has compounding effects similar to RL (Nikishin et al., 2022): an LLM that is overfitted early on will rollout poorer quality data, which in turn leads to less efficient learning, further impairing its ability to learn iteratively without reset, and so on.

Figure: Figure 1: DPO training process gains in MMLU-pro (Avg.) and its subdomains. It compares an LLM trained with standard procedures against one subjected to heavy priming on its initial data.
Refer to caption: x1.png

Figure: (a)
Refer to caption: x2.png

Initial Experiences are LLM-specific.
Existing preference learning algorithms typically rely solely on roll-out data while ignoring the original responses generated by the base model, which significantly constrains the model’s capabilities to the performance bounds of the base model (Ma et al., 2025).
However, directly incorporating original data into the optimization process introduces non-trivial challenges: for instance, an LLM cannot initially determine whether the collected original data is useful for learning, leading to potential inefficiencies or even degradation in performance.
To explore this issue, we try in practice to optimize LLMs by blending different ratios of data, i.e., combining a portion of the original dataset responses with roll-out preference data scored by a reward verifier.
In addition, we integrate diverse optimization losses, such as DPO for preference exploration and SFT for answer fitting.
We evaluate two models, Llama3.2 and Phi3, with results shown in Figure [2](https://arxiv.org/html/2508.06412v2#S3.F2).
Surprisingly, these experiments reveal model-specific patterns rather than consistent trends.
Llama3.2, which appears to have acquired strong math domain capabilities during pre-training, leverages roll-out data to generate high-quality samples for further improvement; in contrast, Phi3 requires prior data for SFT fitting, and incorporating initial experience even has a negative impact.
These findings highlight that the challenges of integrating original data into preference learning are not universal but LLM-specific, underscoring the need for more thoughtful strategy design to leverage initial experience strategically.

###### Definition 1 .

## 4 Scaling Replay Number with Reset for Finetuning LLM

In reinforcement learning, the term replay number typically refers to the number of agent updates for every single step taken in the environment (Wang et al., 2017; Fedus et al., 2020).
Inspired by this, we set LLM finetuning algorithms to learn at a high replay number using preference data or verifiable reward.
More specifically, the core idea is to update the LLM parameters multiple times for each batch finetuning with the dataset, which is achieved by setting a high replay number while keeping the batch size constant.
Surprisingly, despite this seemingly straightforward concept and the amount of work (Nikishin et al., 2022; D’Oro et al., 2023; Schwarzer et al., 2023) that has successfully achieved high replay rates in RL, almost nothing has been reported in the field of LLM finetuning.
In fact, both RL and LLM optimization often show that learning without reset leads to a significant primacy bias (Nikishin et al., 2022) and negatively impacts the quality of policies.

Figure: Algorithm 1 Reset Replay

To address this issue, we introduce an algorithm called LLM optimization with Reset Replay (LoRR) into finetuning processes.
The reset technique we integrated is the Shrink & Perturb strategy (Ash & Adams, 2020), which resets the network parameters of LLMs periodically to maintain the network plasticity.
Originally proposed as a method to "warm-start" neural network training for incorporating new data without losing generalization, this technique has recently been adopted in RL systems (D’Oro et al., 2023; Lyle et al., 2023; Xu et al., 2024) to combat overfitting in high replay number settings. The formulation of the reset strategy for LLM optimization is defined as

$$ $\pi_{\theta}\leftarrow\alpha\pi_{\theta}+(1-\alpha)\pi_{\theta_{\text{init}}},$ (1) $$

where $\pi_{\theta}$ is a current LLM policy learned in the finetuning phase and $\pi_{\theta_{\text{init}}}$ is the initial policy.
The interpolation factor $\alpha$ determines the extent to which the existing LLM parameters are preserved during an update. A higher $\alpha$ indicates a greater influence from the new updates, while a lower $\alpha$ means more of the old parameters are retained.

Next, we integrate the details of the above strategy into our proposed LoRR, as shown in Algorithm [1](https://arxiv.org/html/2508.06412v2#alg1).
LoRR follows the general workflow of standard on-policy preference optimization but employs two additional operations.
The main operation is in line 20, resetting the policy model $\pi_{\theta}$ in replay loops.
Given the reset interval $|\mathcal{B}|$ with L times of replay for each batch $\mathcal{B}$, LoRR performs Shrink & Perturb to inject plasticity into the policy model, thereby restoring the learning capability of specified networks.
The second is hybrid optimization in line 24 via RL and SFT, which has been shown to work differently for diverse on-policy data (Tajwar et al., 2024; Ma et al., 2025).
This augmentation operation with different ratios can take full advantage of LoRR updates, bringing the benefits of diversity to input representations for LLM learning.
Below, we’ll go over each operation in detail.

Replays with Reset Data.
As preference algorithms (Meng et al., 2024; Tran et al., 2023) collect data using the on-policy setting, we develop a replay strategy into the reset loop.
Specifically, we first sample multiple trajectories $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{K}\}$ for each prompt $\mathbf{x}$, and then re-annotate them with a reward verifier $r$ that selects the highest-scoring one as $\mathbf{y}_{w}$ and the lowest-scoring one as $\mathbf{y}_{l}$ (Pace et al., 2024).
The preference pairs $(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})$ generated by the reward signals are collected in a set of pairs $\mathcal{T}$, a.k.a., a new transition batch.
However, to avoid duplicating the update with each replay, a certain ratio of the initial samples is mixed into the transition batch, where the pair is formalized as $(\mathbf{x},\mathbf{y},\mathbf{y}^{\prime}_{l})$(^1^11The lose sample $\mathbf{y}^{\prime}_{l}$ is from the dataset $\mathcal{D}$ if it have preference data, otherwise, it is rolled out in the initial model $\pi_{\theta_{\text{init}}}$ with the same collection method.).
In this high replay number setting, for every replay $\ell$, we set a linear rollout ratio $\varepsilon$ working on each batch.
The ratio goes up as the number of replays declines, which means the model further learns from the experience of the initial sample.
In practice, given an initial ratio $\varepsilon_{\text{init}}=1$, each replay adjusts the ratio to be $\varepsilon\leftarrow\varepsilon_{\text{init}}(1-\frac{\ell}{2L})$, controlling whether transitions are sampled from the current rolled out sample or the initial sample, to increase the diversity of the transition batch.
The intuition behind it is to mitigate experiences of primed LLMs’ uncertainty, allowing the model to simultaneously utilize the on/off-policy data (Tajwar et al., 2024), especially at a high replay number, and thus approach the target distribution.

Replays with Hybrid Optimization.
Both SFT and preference learning methods operate within the same optimal policy-reward subspace, positioning SFT as a special case of implicit reward learning (Wang et al., 2025a).
Beyond the preference optimization, RL with SFT has been shown to increase capacity in LLM finetuning (Ma et al., 2025).
In §[3](https://arxiv.org/html/2508.06412v2#S3) we have explored the uncertainty of LLM experience, but what about preference optimization combined with SFT?
Thus, in high replay number settings, we directly mix different proportions of SFT losses, allowing the policy model to learn $\mathbf{y}_{w}$ directly.
Mathematically, given a preference loss(^2^22Our default optimizing loss is DPO, but this can be replaced at will, depending on scenarios.) $\mathcal{L}_{\text{DPO}}$ and an SFT loss $\mathcal{L}_{\text{SFT}}$, the hybrid optimization in each reply is

$$ $\mathcal{L}_{\theta}=\lambda\mathcal{L}_{\text{SFT}}+(1-\lambda)\mathcal{L}_{\text{DPO}},$ (2) $$

where $\theta$ denotes the parameter of the current policy $\pi_{\theta}$ and $\lambda$ is an SFT ratio.
The $\mathbf{y}_{w}$ of the rollout is intuitively consistent with the reward throughout the iteration; hence, as the number of replays increases, so does our $\lambda$.
Formally, we set an initial value $\lambda_{\text{init}}=0$ that the first optimization is based on preferences only, and then for each optimization after a reset, the SFT ratio adjusts as $\lambda\leftarrow\frac{\ell}{2L}+\lambda_{\text{init}}$.
As this ratio grows, it was theoretically proven (Tajwar et al., 2024) that the policy distribution slowly aligns with the expected distribution.
And thanks to the reset strategy, the hybrid optimization slows down the primary bias in LLM.

The novel combination of the two straightforward yet effective extended techniques allows LLM to not overfit but also to efficiently utilize the diversity of samples.
Moreover, LoRR is general and easy to plug into the mainstream off-policy LLM optimization with only minor modification, as detailed in Algorithm [1](https://arxiv.org/html/2508.06412v2#alg1). Next, we’ve conducted extensive experiments on various math and reasoning tasks to empirically validate LoRR’s effectiveness.

## 5 Experiments

In this section, we mainly evaluate LoRR on mathematical and reasoning problems, highlighting the superior performance of LoRR as a plugin with different optimization methods (§[5.2](https://arxiv.org/html/2508.06412v2#S5.SS2)).
We then investigate the performance of LoRR on iterative DPOs and compare it to existing RL-based methods with continuous multi-round improvements, analyzing potential differences in reasoning paradigms (§[5.3](https://arxiv.org/html/2508.06412v2#S5.SS3)).
Further, we provide an in-depth understanding of the following components: ratio type and replay number in the Appendix §[C](https://arxiv.org/html/2508.06412v2#A3).
For all the tasks, we set $\alpha$ to 0.5 and apply to downstream projections of networks for the Shrink & Perturb strategy.

### 5.1 Experimental Setup

Datasets. Our training set is augmented with mathematical problems from a subset of Meta-Math (Yu et al., 2024) and MMIQC (Liu et al., 2025). This data has been processed into high-quality preference data by previous work (Lai et al., 2024) and used for finetuning various mathematical models. To ensure that the amount of data in §[5.3](https://arxiv.org/html/2508.06412v2#S5.SS3) is consistent with the baseline (e.g., DPO-VP (Tu et al., 2025)), the same 8K data are sampled for training. In addition, we optimize on a universal dataset, UltraFeedback (Cui et al., 2023), for comparing the reasoning performance of models.

Models and Finetuning Implementation. We first optimize preferences using three model families: Llama3.2-3B (Grattafiori et al., 2024), Phi3-mini-4K (Abdin et al., 2024), and Qwen2.5-3B (Yang et al., 2025), all in the Instruct setup.
For a fair comparison, we use the same reward model as SimPO (Meng et al., 2024): ArmoRM-Llama3-8B-v0.1 (Wang et al., 2024b) for ranking generated data, significantly enhancing performance.
We test the performance of LoRR under five different sets of preference optimizations: DPO (Rafailov et al., 2023), KTO (Ethayarajh et al., 2024), IPO (Azar et al., 2024), rDPO (Park et al., 2024), and SimPO (Meng et al., 2024), where the hyperparameters remain unchanged and are marked as Base in tables.
For comparison, we set the number of replays to $3$ in LoRR and integrate in these preference optimizations to see the performance of LoRR.
Furthermore, we add an iterative preference training approach (Iter. n) to each method, where the number of iterations is $3$, and each iteration rolls data out through the reward model, so as to compare whether LoRR improves in the case where a policy model meets the same amount of data.
For multi-round self-improvement experiments, we mainly use the Qwen2.5-Math-7B model for optimization, which is consistent with most works (Guan et al., 2025; Cui et al., 2025; Zeng et al., 2025).

Evaluation In our evaluation, we mainly focus on several well-established mathematical benchmarks: GSM8K (Cobbe et al., 2021), MATH/MATH500 (Hendrycks et al., 2021), TabMWP (Lu et al., 2023), Minerva (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), and AIME/AMC (Li et al., 2024). All of the benchmarks adopt Pass@1 as the evaluation criterion.
Considering reasoning experiments training on UltraFeedback, we further evaluate the generalization ability on MMLU-Pro (Wang et al., 2024c), where we randomize the order of multiple-choice options to avoid contamination.

### 5.2 Main Results

Our initial evaluation focuses on quantifying the performance improvement of the LoRR plugin on different preference optimizations.
The experimental procedure involves finetuning models on the previously mentioned 8K math dataset, first using standard preference learning to create baseline models, and then enhancing the process with LoRR.
Results of these base preference learning and LoRR-based versions for the math ability are shown in Tables [1](https://arxiv.org/html/2508.06412v2#S5.T1), [5](https://arxiv.org/html/2508.06412v2#A2.T5), and [6](https://arxiv.org/html/2508.06412v2#A2.T6).
As we can see, the introduction of LoRR, particularly when configured with a replay number of 3, leads to a remarkable boost in performance, far exceeding the outcomes of the base optimization methods.
Specifically, the different finetuning models have max 6.54% to 16.99% enhancements relative to the SFT model, using the LoRR setting.
It also produces a significant boosting effect compared to methods where the interaction with the dataset is fixed (e.g., the number of iterations is 3).
Some optimizations (e.g., training Llama3.2 with rDPO) were originally trained to cause side effects, yet those are higher than SFT models under LoRR.
In a side-by-side comparison, SimPO has the most significant performance effect when plugged in LoRR, maintaining first place performance on all three models, followed by rDPO.
Note that the small sample size of AIME24 resulted in no significant enhancement, possibly because the SFT data is sampled by $\pi_{\theta_{\text{init}}}$, but this policy has no resolution capability in its knowledge.
Overall, LoRR is able to enhance most of the scenarios under different training methods.

To evaluate the broader applicability of LoRR, we extend our experiments to a general domain. We optimize the Llama3.2 on the UltraFeedback dataset and subsequently test on the challenging MMLU-Pro reasoning task.
As baselines, we employ two different preference optimization algorithms, DPO and SimPO, and each algorithm incorporates a different contribution of SFT loss.
We then augmented these baseline methods with LoRR to isolate its impact. Figure [3](https://arxiv.org/html/2508.06412v2#S5.F3) illustrates the comparative performance, highlighting the improvements conferred by our method.
Specifically, it show that varying the proportion of SFT loss yield minimal performance improvements and induce overfitting after approximately 400 training steps.
In stark contrast, LoRR achieves a significant performance increase by efficiently reusing sample data through its reset-and-replay cycle.

**Table 1: Llama3.2 finetuning results on six mathematical benchmarks. We train SFT models on the 8K math data for base, iterative, and LoRR settings, where the training modes include DPO, KTO, IPO, rDPO, and SimPO. For Instruct settings, we use off-the-shelf models as the SFT model.**
| Training Method | GSM8k | MATH | TabMWP | Minerva | AMC23 | AIME24 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama3.2-3B-Instruct | 37.7 | 35.1 | 27.5 | 33.8 | 12.5 | 6.7 | 25.55 |
| Llama3.2 + DPO (Base) | 40.6 | 36.7 | 27.3 | 33.4 | 15.0 | 3.3 | 26.05 |
| + DPO (Iter. n) | 41.9 | 36.1 | 28.1 | 35.2 | 15.0 | 6.7 | 27.2 |
| + DPO (Base+LoRR) | 43.0 | 37.9 | 28.2 | 36.0 | 15.0 | 10.0 | 28.35 |
| Llama3.2 + KTO (Base) | 39.0 | 36.1 | 27.3 | 35.2 | 17.5 | 6.7 | 26.97 |
| + KTO (Iter. n) | 39.4 | 36.2 | 26.8 | 32.4 | 20.0 | 10.0 | 27.47 |
| + KTO (Base+LoRR) | 40.5 | 37.3 | 28.8 | 36.2 | 20.0 | 10.0 | 28.80 |
| Llama3.2 + IPO (Base) | 37.8 | 36.0 | 26.7 | 33.6 | 15.0 | 6.7 | 25.97 |
| + IPO (Iter. n) | 40.9 | 36.9 | 27.7 | 35.4 | 12.5 | 6.7 | 26.70 |
| + IPO (Base+LoRR) | 40.3 | 37.1 | 28.9 | 36.0 | 12.5 | 6.7 | 26.92 |
| Llama3.2 + rDPO (Base) | 37.7 | 35.9 | 27.6 | 33.4 | 15.0 | 3.3 | 25.48 |
| + rDPO (Iter. n) | 40.6 | 37.8 | 29.5 | 36.6 | 12.5 | 6.7 | 27.28 |
| + rDPO (Base+LoRR) | 40.9 | 38.1 | 28.4 | 39.4 | 20.0 | 6.7 | 28.92 |
| Llama3.2 + SimPO (Base) | 39.0 | 36.8 | 28.4 | 33.8 | 12.5 | 3.3 | 25.63 |
| + SimPO (Iter. n) | 40.9 | 37.6 | 26.4 | 34.4 | 15.0 | 6.7 | 26.83 |
| + SimPO (Base+LoRR) | 45.7 | 38.6 | 32.3 | 39.4 | 20.0 | 3.3 | 29.88 |

Figure: (a)
Refer to caption: x6.png

### 5.3 Results on iterative DPO

Further, to create a rigorous benchmark against RL and preference learning methods, we conduct iterative DPO (a.k.a., multi-round DPO) on the advanced Qwen2.5-Math-7B model (Yang et al., 2024), ensuring a controlled comparison by using the same base model and datasets with similar sample sizes.
We implement the LoRR with 3 epochs of DPO.
At the end of each epoch, we dynamically generate new SFT and preference data using the current policy $\pi_{\theta}$.
This newly generated preference data is labeled by a reward model, making it closer to an on-policy setting, as discussed in (Meng et al., 2024).
The resulting policy, which we name it as Qwen2.5-7B-DPO-LoRR, is then compared against several state-of-the-art RL-based methods:

- •
Qwen2.5-Math-7B-Instruct (Yang et al., 2024): The instruction-tuned base model from the Qwen family, selected for its strong mathematical reasoning capabilities.
- •
rStar-Math-7B (Guan et al., 2025): A model trained on a large volume of data generated via self-evolution with Monte Carlo Tree Search, which is then filtered using a Process Preference Model.
- •
Eurus-2-7B-PRIME (Cui et al., 2025): A model trained using policy rollouts and outcome labels, leveraging implicit process rewards.
- •
Simple-RL-Zero (Zeng et al., 2025) and Qwen2.5-7B-PURE-VR (Cheng et al., 2025): Two similar models, trained by RL with verifiable rewards and some tricks, were also trained without additional SFT data.
- •
Qwen2.5-7B-ReLIFT (Ma et al., 2025) and Qwen2.5-7B-LUFFY (Yan et al., 2025): Two models are both reinforcement learning frameworks that bridge the gap between RL and SFT by incorporating off-policy reasoning traces into the training process.
- •
Qwen2.5-DPO-R1-Zero (Zhang et al., 2025) and Qwen2.5-7B-DPO-VP (Tu et al., 2025): Both investigate the effectiveness of iterative DPO in facilitating self-improvement for LLMs.

**Table 2: Pass@1 results across different methods on math tasks. All the models have been fine-tuned based on the Qwen2.5-Math-7B. The results with $\dagger$ are the ones we evaluated from the official model published by the authors, and the rStar results with $*$ are obtained from Guan et al. (2025). In the relatively fair comparison, Bold indicates the best results and underline indicates the second one.**
| Model Pass@1 Acc. | MATH500 | Minerva | Olybench | AMC23 | AIME24 | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Qwen2.5-Math-7B^† | 66.2 | 10.7 | 24.1 | 47.5 | 23.3 | 34.4 |
| Qwen2.5-Math-7B-Instruct^† | 84.6 | 37.1 | 39.9 | 62.5 | 16.7 | 48.2 |
| rStar-Math-7B^∗ (Guan et al., 2025) | 78.4 | - | 47.1 | 47.5 | 26.7 | - |
| Eurus-2-7B-PRIME^† (Cui et al., 2025) | 74.0 | 39.7 | 35.6 | 57.5 | 23.3 | 46.0 |
| Qwen2.5-7B-ReLIFT^† (Ma et al., 2025) | 81.4 | 23.9 | 44.3 | 62.5 | 13.3 | 45.1 |
| Qwen2.5-7B-LUFFY^† (Yan et al., 2025) | 80.6 | 30.5 | 45.6 | 72.5 | 13.3 | 48.5 |
| Simple-RL-Zero^† (Zeng et al., 2025) | 77.8 | 32.7 | 40.7 | 57.5 | 20.0 | 45.7 |
| Qwen2.5-7B-PURE-VR^† (Cheng et al., 2025) | 72.8 | 13.6 | 28.1 | 50.0 | 23.3 | 37.6 |
| Qwen2.5-DPO-R1-Zero^† (Zhang et al., 2025) | 75.4 | 27.6 | 38.8 | 60.0 | 23.3 | 45.0 |
| Qwen2.5-7B-DPO-VP^† (Tu et al., 2025) | 76.0 | 30.9 | 38.2 | 62.5 | 26.7 | 46.9 |
| Qwen2.5-7B-DPO-LoRR-iter1 | 73.8 | 23.5 | 35.1 | 55.0 | 23.3 | 42.1 |
| Qwen2.5-7B-DPO-LoRR-iter2 | 75.4 | 33.8 | 37.0 | 55.0 | 30.0 | 46.2 |
| Qwen2.5-7B-DPO-LoRR-iter3 | 75.6 | 36.0 | 37.2 | 60.0 | 30.0 | 47.8 |

These baselines all use the ZERO version, so all answers $\mathbf{y}$ to the maths questions are rolled out by policies not initially in the dataset.
We evaluate our model and baselines on GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), and several Olympiad-level benchmarks (He et al., 2024), including AMC23, AIME24, and OlympiadBench.
However, most of these have different training data and reward models, making it difficult to make fair comparisons.
In keeping with most RL methods, we similarly use the 8K math questions we mentioned to train our model with iterative DPO.
Table [2](https://arxiv.org/html/2508.06412v2#S5.T2) lists the pass@1 accuracies of different models, where the first category is not comparable due to the effect of the amount of training data, etc., and the second category is relatively fair, although the different methods differ in their settings, e.g., the data and epoch.
We also present Table [3](https://arxiv.org/html/2508.06412v2#S5.T3) for further analysis of training data consumption and GPU usage to describe the difference among them.
Note that we do not use rule-based reward signals because LoRR can train on general datasets, not just for answer matching.

Based on the results in Tables [2](https://arxiv.org/html/2508.06412v2#S5.T2) and [3](https://arxiv.org/html/2508.06412v2#S5.T3), we draw the following conclusions:

- •
Trained with LoRR, a simple iterative DPO achieves mathematical reasoning that is comparable to RL-based methods. Our final model, Qwen2.5-7B-DPO-LoRR, attains an average score of 47.8, the highest among all methods in the relatively fair comparison group. This performance not only exceeds other DPO variants like Qwen2.5-7B-DPO-VP (46.9) but also outperforms prominent RL-based models such as Simple-RL-Zero (45.7), all while using the same base model and initial training data constraints.
- •
The iterative application of DPO demonstrates consistent and significant performance improvements. We observe a clear monotonic progression in the average score across iterations: from 42.1 in the first iteration, to 46.2 in the second, and culminating in 47.8 in the third. This highlights the effectiveness of our reset replays for data reusing.
- •
This superior performance is achieved with remarkable data and computational efficiency, presenting a highly practical and scalable paradigm.
Our method requires only 8K preference pairs for the entire iterative training process, and the entire training for our model was completed in just 2 days on a single A100 GPU. This is a fraction of the data used by other top-performing models like Qwen2.5-7B-ReLIFT (45K) and orders of magnitude less than rStar-Math-7B ($\sim$3.6M) with more GPUs, demonstrating our approach’s ability to learn effectively from limited data.

In summary, our work demonstrates that LoRR plugin an iterative DPO can achieve leading performance in mathematical reasoning without the need for massive-scale SFT data, complex reward modeling pipelines, or extensive computational clusters, offering a powerful and accessible alternative to prevailing RL-based finetuning methodologies.

**Table 3: A comparison of data and GPUs among different methods.**
|  | Qwen2.5-Math-7B-Instruct | rStar-Math-7B | Eurus-2-7B-PRIME | Qwen2.5-7B-SimpleRL-Zero | Qwen2.5-7B-ReLIFT | Qwen2.5-7B-DPO-VP | Qwen2.5-7B-DPO-LoRR |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Paradigm | RL + ORM | MCTS + PPM | RL + PRM | RL + VR | RL + SFT | DPO + VR | DPO + SFT |
| Base Model | Qwen2.5-Math-7B | Qwen2.5-Math-7B | Qwen2.5-Math-7B | Qwen2.5-Math-7B | Qwen2.5-Math-7B | Qwen2.5-Math-7B | Qwen2.5-Math-7B |
| SFT Data | 2.5M (open-source and in-house) | $\sim$7.3M (MATH, NuminaMath, etc.) | 230K | 0 | 0 | 0 | 0 |
| RM Data | 618K (in-house) | $\sim$7K (in-house) | 0 | 0 | 0 | 0 | 0 |
| RM | Qwen2.5-Math-RM (72B) | 7B PPM | Eurus-2-7B-SFT | Rule-based RM | Rule-based RM | DeepSeek-V3 | ArmoRM-Llama3-8B-v0.1 |
| Improvement Data | 66K | $\sim$3.647M | 150K | 8K | 45k | 8K | 8K |
| Epoch | - | 2 | - | 1 | 3 | 6 | 3 |
| GPUs / Time | - | 80 H100 / few days | 8 A100 / few days | 32 H100 / 1.5 days | 16 A800 / - | 1 A100 / 3 days | 1 A100 / 2 days |

## 6 Conclusion

In this paper, we introduced the LLM optimization with Reset Replay (LoRR), to address the critical issues of low sample efficiency and primacy bias in LLM fine-tuning. Our method is a general-purpose plugin that synergizes high-replay training with periodic parameter resets and a hybrid optimization loss, enabling more effective data utilization.
Extensive experiments show that LoRR significantly enhances various preference optimization methods on both mathematical and general reasoning tasks. Notably, an iterative DPO approach augmented with LoRR achieves state-of-the-art performance, outperforming complex RL-based methods with substantially greater sample and computational efficiency.
This work demonstrates that by mitigating overfitting from intensive data reuse, LoRR unlocks the latent potential in training data, offering a practical path toward more powerful and efficient LLM optimization.
As for future work, there are two key aspects. On one hand, a higher replay number (e.g., $L=10$) tends to become less effective in the later stages of training, therefore, it may be feasible to address this issue through adaptive replay frequencies.
On the other hand, it is crucial to further understand and reveal the potential mechanisms of plasticity in LLM networks when they are fine-tuned.

## References

- Abdin et al. (2024)
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al.
Phi-3 technical report: A highly capable language model locally on your phone.
*arXiv preprint arXiv:2404.14219*, 2024.
- Ash & Adams (2020)
Jordan Ash and Ryan P Adams.
On warm-starting neural network training.
In *NeurIPS*, pp.  3884–3894, 2020.
- Azar et al. (2024)
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello.
A general theoretical paradigm to understand learning from human preferences.
In *AISTATS*, pp.  4447–4455, 2024.
- Chen et al. (2021)
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross.
Randomized ensembled double q-learning: Learning fast without a model.
In *ICLR*, pp.  1–25, 2021.
- Cheng et al. (2025)
Jie Cheng, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Gang Xiong, Yisheng Lv, and Fei-Yue Wang.
Stop summation: Min-form credit assignment is all process reward model needs for reasoning.
*arXiv preprint arXiv:2504.15275*, 2025.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Cui et al. (2023)
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.
Ultrafeedback: Boosting language models with high-quality feedback.
*arXiv preprint arXiv:2310.01377*, 2023.
- Cui et al. (2025)
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al.
Process reinforcement through implicit rewards.
*arXiv preprint arXiv:2502.01456*, 2025.
- D’Oro et al. (2023)
Pierluca D’Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G. Bellemare, and Aaron C. Courville.
Sample-efficient reinforcement learning by breaking the replay ratio barrier.
In *ICLR*, pp.  1–23, 2023.
- Ethayarajh et al. (2024)
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.
Kto: Model alignment as prospect theoretic optimization.
*arXiv preprint arXiv:2402.01306*, 2024.
- Fedus et al. (2020)
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney.
Revisiting fundamentals of experience replay.
In *ICML*, pp.  3061–3071, 2020.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Guan et al. (2025)
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang.
rstar-math: Small llms can master math reasoning with self-evolved deep thinking.
*arXiv preprint arXiv:2501.04519*, 2025.
- He et al. (2024)
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.
In *ACL*, pp.  3828–3850, 2024.
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the MATH dataset.
In *NeurIPS*, 2021.
- Huan et al. (2025)
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue.
Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning.
*arXiv preprint arXiv:2507.00432*, 2025.
- Huang et al. (2025)
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou.
Large language models cannot self-correct reasoning yet.
In *ICLR*, pp.  1–17, 2025.
- Ibrahim et al. (2024)
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish.
Simple and scalable strategies to continually pre-train large language models.
*Transactions on Machine Learning Research*, pp.  95266–45, 2024.
- Kumar et al. (2025)
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan.
Llm post-training: A deep dive into reasoning large language models.
*arXiv preprint arXiv:2502.21321*, 2025.
- Lai et al. (2024)
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia.
Step-dpo: Step-wise preference optimization for long-chain reasoning of llms.
*arXiv preprint arXiv:2406.18629*, 2024.
- Leike et al. (2018)
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg.
Scalable agent alignment via reward modeling: a research direction.
*arXiv preprint arXiv:1811.07871*, 2018.
- Lewkowycz et al. (2022)
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.
Solving quantitative reasoning problems with language models.
In *NeurIPS*, pp.  3843–3857, 2022.
- Li et al. (2024)
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al.
Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.
*Hugging Face repository*, 2024.
- Liu et al. (2025)
Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew C Yao.
Augmenting math word problems via iterative question composing.
In *AAAI*, pp.  24605–24613, 2025.
- Liu et al. (2024)
Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, and Jiang Bian.
Protecting your llms with information bottleneck.
In *NeurIPS*, pp.  29723–29753, 2024.
- Lu et al. (2023)
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan.
Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.
In *ICLR*, pp.  1–26, 2023.
- Lyle et al. (2023)
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney.
Understanding plasticity in neural networks.
In *ICML*, pp.  23190–23211, 2023.
- Ma et al. (2025)
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al.
Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions.
*arXiv preprint arXiv:2506.07527*, 2025.
- Meng et al. (2024)
Yu Meng, Mengzhou Xia, and Danqi Chen.
Simpo: Simple preference optimization with a reference-free reward.
In *NeurIPS*, pp.  124198–124235, 2024.
- Nikishin et al. (2022)
Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville.
The primacy bias in deep reinforcement learning.
In *ICML*, pp.  16828–16847, 2022.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
In *NeurIPS*, pp.  27730–27744, 2022.
- Pace et al. (2024)
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn.
West-of-n: Synthetic preference generation for improved reward modeling.
In *ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models*, pp.  1–19, 2024.
- Park et al. (2024)
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn.
Disentangling length from quality in direct preference optimization.
In *Findings of ACL*, pp.  4998–5017, 2024.
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
In *NeurIPS*, pp.  53728–53741, 2023.
- Schwarzer et al. (2023)
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro.
Bigger, better, faster: Human-level atari with human-level efficiency.
In *ICML*, pp.  30365–30380, 2023.
- Sokar et al. (2023)
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci.
The dormant neuron phenomenon in deep reinforcement learning.
In *ICML*, pp.  32145–32168, 2023.
- Tajwar et al. (2024)
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar.
Preference fine-tuning of llms should leverage suboptimal, on-policy data.
In *ICML*, pp.  47441–47474, 2024.
- Tran et al. (2023)
Hoang Tran, Chris Glaze, and Braden Hancock.
Iterative dpo alignment.
Technical report, Technical report, Snorkel AI, 2023.
- Tu et al. (2025)
Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, et al.
Enhancing llm reasoning with iterative dpo: A comprehensive empirical investigation.
*arXiv preprint arXiv:2503.12854*, 2025.
- Wang et al. (2025a)
Bo Wang, Qinyuan Cheng, Runyu Peng, Rong Bao, Peiji Li, Qipeng Guo, Linyang Li, Zhiyuan Zeng, Yunhua Zhou, and Xipeng Qiu.
Implicit reward as the bridge: A unified view of sft and dpo connections.
*arXiv preprint arXiv:2507.00018*, 2025a.
- Wang et al. (2024a)
Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An.
Q*: Improving multi-step reasoning for llms with deliberative planning.
*arXiv preprint arXiv:2406.14283*, 2024a.
- Wang et al. (2024b)
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang.
Interpretable preferences via multi-objective reward modeling and mixture-of-experts.
In *Findings of EMNLP*, pp.  10582–10592, 2024b.
- Wang et al. (2025b)
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al.
Reinforcement learning for reasoning in large language models with one training example.
*arXiv preprint arXiv:2504.20571*, 2025b.
- Wang et al. (2024c)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al.
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.
In *NeurIPS*, pp.  95266–95290, 2024c.
- Wang et al. (2017)
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas.
Sample efficient actor-critic with experience replay.
In *ICLR*, pp.  1–20, 2017.
- Xia et al. (2024)
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen.
Less: selecting influential data for targeted instruction tuning.
In *ICML*, pp.  54104–54132, 2024.
- Xu et al. (2024)
Linjie Xu, Zichuan Liu, Alexander Dockhorn, Diego Perez Liebana, Jinyu Wang, Lei Song, and Jiang Bian.
Higher replay ratio empowers sample-efficient multi-agent reinforcement learning.
In *CoG*, pp.  1–8, 2024.
- Yan et al. (2025)
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang.
Learning to reason under off-policy guidance.
*arXiv preprint arXiv:2504.14945*, 2025.
- Yang et al. (2024)
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al.
Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.
*arXiv preprint arXiv:2409.12122*, 2024.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yu et al. (2024)
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
Metamath: Bootstrap your own mathematical questions for large language models.
In *ICLR*, pp.  1–22, 2024.
- Zeng et al. (2025)
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He.
7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.
[https://hkust-nlp.notion.site/simplerl-reason](https://hkust-nlp.notion.site/simplerl-reason), 2025.
- Zhang et al. (2025)
Hanning Zhang, Jiarui Yao, Chenlu Ye, Wei Xiong, and Tong Zhang.
Online-dpo-r1: Unlocking effective reasoning without the ppo overhead, 2025.
*Notion Blog*, 2025.
- Zhao et al. (2025)
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang.
Absolute zero: Reinforced self-play reasoning with zero data.
*arXiv preprint arXiv:2505.03335*, 2025.
- Zhou et al. (2023)
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.
Lima: Less is more for alignment.
In *NeurIPS*, pp.  55006–55021, 2023.
- Zhou et al. (2022)
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville.
Fortuitous forgetting in connectionist networks.
In *ICLR*, pp.  1–24, 2022.

## Appendix A Implementation Details

In this section, we outline the specific parameters and data of the experiments.

**Table 4: Various preference optimization hyperparameters used for each training setting.**
| Method | $\beta$ | $\gamma$ | Learning rate |
| --- | --- | --- | --- |
| DPO | 0.01 | - | 5.0e-7 |
| KTO | 0.01 | 1.0 | 5.0e-7 |
| IPO | - | 0.5 | 5.0e-7 |
| rDPO | 0.01 | 0.6 | 5.0e-7 |
| SimPO | 2 | 0.55 | 1e-6 |

General training hyperparameters. We conduct preliminary experiments for the basic preference learning settings using a batch size of 128 and a single training epoch. Moreover, we set the maximum sequence length at 2048 and utilize the AdamW optimizer with a cosine learning schedule, which includes 10% warmup steps, for the preference optimization.
For each preference method, we follow the most parameterised exploration of SimPO (Meng et al., 2024), where Table [4](https://arxiv.org/html/2508.06412v2#A1.T4) shows the detailed hyperparameters for baselines.
For LoRR, we set the default parameters: a replay number $L=3$, sample number $K=5$, reset weight $\alpha=0.5$, and an iteration number $N=7$ for math.
For LoRR in iterative DPO, $N=3$ is set and $3$ epochs are guaranteed, while other parameters remain unchanged.
DeepSpeed
ZeRO2 with CPU offload is used to reduce GPU memory usage during training.

Building datasets for optimization.
For math tasks, our training dataset is supplemented with mathematical problems selected from subsets of Meta-Math (Yu et al., 2024) and MMIQC (Liu et al., 2025).
To maintain consistency in data volume with the baselines as discussed in §[5.3](https://arxiv.org/html/2508.06412v2#S5.SS3), we sampled the same 8K data points for training.
For the reasoning tasks, we conduct optimizations on a general-purpose dataset, UltraFeedback (Cui et al., 2023), to facilitate comparisons of the models’ reasoning capabilities.
Regarding training data, we produce a maximum of 5 responses for each prompt in each replay. Additionally, we leverage the ArmoRM model (Wang et al., 2024b) to label the preference relationships between these responses.

## Appendix B Additional Experiment Results

We further evaluate the performance of the LoRR plugin on Phi3 and Qwen2.5 models, with results presented in Tables [5](https://arxiv.org/html/2508.06412v2#A2.T5) and [6](https://arxiv.org/html/2508.06412v2#A2.T6) respectively.
These experiments follow the same setup as the Llama3.2 evaluation: finetuning on the 8K math dataset using standard preference learning (baseline) and LoRR-enhanced preference learning, with performance measured across the six mathematical benchmarks.
Similarly to the body text, the LoRR-based versions consistently outperform their base preference learning and iterative counterparts.
The results for Phi3 and Qwen2.5 reinforce the effectiveness of LoRR across different model architectures, consistently enhancing performance across most mathematical benchmarks when integrated with various preference optimization techniques.

**Table 5: Phi3 finetuning results on six mathematical benchmarks. We train SFT models on the 8K math data for base, iterative, and LoRR settings, where the training modes include DPO, KTO, IPO, rDPO, and SimPO. For Instruct settings, we use off-the-shelf models as the SFT model.**
| Training Method | GSM8k | MATH | TabMWP | Minerva | AMC23 | AIME24 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Phi3-mini-4K-Instruct | 52.8 | 21.8 | 19.2 | 23.4 | 20.0 | 3.3 | 23.42 |
| Phi3 + DPO (Base) | 52.6 | 21.8 | 18.7 | 24.4 | 12.5 | 3.3 | 22.22 |
| + DPO (Iter. n) | 51.9 | 19.9 | 17.2 | 18.2 | 22.5 | 3.3 | 22.17 |
| + DPO (Base+LoRR) | 53.9 | 23.8 | 17.6 | 25.2 | 15.0 | 6.7 | 23.70 |
| Phi3 + KTO (Base) | 52.8 | 21.6 | 19.0 | 23.8 | 17.5 | 3.3 | 23.00 |
| + KTO (Iter. n) | 51.3 | 21.1 | 18.9 | 18.2 | 20.0 | 3.3 | 22.13 |
| + KTO (Base+LoRR) | 54.2 | 23.8 | 19.7 | 24.2 | 20.0 | 3.3 | 24.20 |
| Phi3 + IPO (Base) | 49.7 | 20.2 | 16.6 | 18.4 | 15.0 | 3.3 | 20.53 |
| + IPO (Iter. n) | 51.9 | 21.1 | 19.5 | 20.2 | 20.0 | 3.3 | 22.67 |
| + IPO (Base+LoRR) | 51.9 | 21.7 | 19.7 | 19.8 | 17.5 | 6.7 | 22.88 |
| Phi3 + rDPO (Base) | 52.8 | 21.7 | 19.8 | 20.6 | 20.0 | 0.0 | 22.48 |
| + rDPO (Iter. n) | 50.2 | 20.6 | 17.7 | 19.8 | 17.5 | 3.3 | 21.52 |
| + rDPO (Base+LoRR) | 52.5 | 22.4 | 18.1 | 23.6 | 25.0 | 3.3 | 24.15 |
| Phi3 + SimPO (Base) | 51.7 | 21.2 | 19.5 | 22.2 | 17.5 | 3.3 | 22.57 |
| + SimPO (Iter. n) | 48.7 | 19.2 | 16.1 | 18.6 | 17.5 | 6.7 | 21.13 |
| + SimPO (Base+LoRR) | 53.2 | 24.2 | 19.0 | 24.0 | 22.5 | 6.7 | 24.93 |

**Table 6: Qwen2.5 finetuning results on six mathematical benchmarks. We train SFT models on the 8K math data for base, iterative, and LoRR settings, where the training modes include DPO, KTO, IPO, rDPO, and SimPO. For Instruct settings, we use off-the-shelf models as the SFT model.**
| Training Method | GSM8k | MATH | TabMWP | Minerva | AMC23 | AIME24 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2.5-3B-Instruct | 41.1 | 60.4 | 32.1 | 60.8 | 25.0 | 3.3 | 37.12 |
| Qwen2.5 + DPO (Base) | 44.3 | 60.9 | 32.4 | 60.2 | 25.0 | 0.0 | 37.13 |
| + DPO (Iter. n) | 42.5 | 60.8 | 30.5 | 60.0 | 27.5 | 0.0 | 36.88 |
| + DPO (Base+LoRR) | 45.4 | 61.1 | 32.5 | 61.2 | 27.5 | 0.0 | 37.95 |
| Qwen2.5 + KTO (Base) | 44.3 | 60.5 | 29.6 | 60.6 | 22.5 | 0.0 | 36.25 |
| + KTO (Iter. n) | 43.1 | 60.6 | 32.3 | 60.2 | 27.5 | 0.0 | 37.28 |
| + KTO (Base+LoRR) | 44.3 | 59.8 | 30.9 | 60.4 | 27.5 | 3.3 | 37.70 |
| Qwen2.5 + IPO (Base) | 43.4 | 60.3 | 31.2 | 61.4 | 22.5 | 3.3 | 37.02 |
| + IPO (Iter. n) | 42.3 | 60.0 | 28.1 | 60.2 | 30.0 | 0.0 | 36.77 |
| + IPO (Base+LoRR) | 44.0 | 60.3 | 31.1 | 61.8 | 27.5 | 6.7 | 38.57 |
| Qwen2.5 + rDPO (Base) | 44.5 | 60.8 | 29.6 | 61.4 | 22.5 | 3.3 | 37.02 |
| + rDPO (Iter. n) | 43.1 | 60.6 | 32.3 | 61.2 | 27.5 | 0.0 | 37.45 |
| + rDPO (Base+LoRR) | 44.9 | 61.1 | 30.3 | 58.0 | 32.5 | 3.3 | 38.35 |
| Qwen2.5 + SimPO (Base) | 41.8 | 60.6 | 29.6 | 61.0 | 27.5 | 3.3 | 37.30 |
| + SimPO (Iter. n) | 37.7 | 57.4 | 28.4 | 58.6 | 32.5 | 6.7 | 36.88 |
| + SimPO (Base+LoRR) | 41.8 | 60.4 | 32.9 | 61.8 | 35.0 | 6.7 | 39.77 |

## Appendix C Ablation Study and Analysis

To further analyze the components of LoRR, we conduct an ablation study to assess the effectiveness of our LoRR. Specifically, we first compare the following three ablation settings for training DPO: (i) replays only with reset data, i.e., we adjust the rollout ratio in LoRR; (ii) replays only with hybrid optimization (adjusting SFT ratio) while the training data is all from the policy rollouts; and (iii) a full LoRR algorithm with both components.
For comparison, we use a pure DPO as a baseline.
As summarized in Figure [4](https://arxiv.org/html/2508.06412v2#A3.F4), the results show that both components more or less enhance the DPO performance.
Specifically, replays with reset data require high-quality model rollouts, so performance depends heavily on the base versus base models.
Hybrid optimization, on the other hand, substantially improves sample efficiency in reset replays, resulting in better performance.

We also explore the effect of a high number of replays, visualizing the performance of LoRR under different $L$ values on the six math benchmarks.
As presented in Figure [5](https://arxiv.org/html/2508.06412v2#A3.F5), we observe that $L\in\{2,3,5\}$ could drastically improve the sample efficiency. This conclusion is similar to traditional RL agents (D’Oro et al., 2023; Xu et al., 2024), but we validate it for the first time in a language task.
However, a larger replay number, such as $L=10$, causes poor performance.
A possible reason for this decline is that overly frequent exploitation of the collected data can lead to overfitting on the current experience, even with the reset.
This, in turn, causes the policy to quickly lose its ability to sample high-quality rollouts.
Thus, considering the training time and sample efficiency, we choose $L=3$ as the benchmark setting for LoRR.

Consequently, these results affirm that current preference learning does not effectively utilize the data it gathers. LoRR, conversely, can activate its latent potential as an efficient replay strategy.

Figure: Figure 4: Comparison of different components of LoRR. The experiments were fine-tuned on each of the three language models with DPO, and tested on GSM8K and MATH tasks.
Refer to caption: x8.png

Figure: Figure 5: The performance of DPO with LoRR on six math tasks under a different replay number. The best results are described in bold, and the average of the six tasks is shown at the end.
Refer to caption: x9.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. The main paper should include more ablation and analysis experiments to isolate their individual contributions. Please also explore hyperparameter sensitivities—e.g., how $\lambda$ and $\alpha$ schedules or different initialization choices affect performance.
2. In Algorithm 1, line 17, the notation $y^′_l$ is unclear.
3. How does the proposed method scale to larger model sizes? It would be useful to include results or discussion on its generalization behavior for models beyond 7B parameters.
soundness: 3
strengths: 1. LoRR can be integrated with multiple preference optimization algorithms and shows particular advantages in low-compute scenarios.
2. The paper comprehensively evaluates LoRR on three model families across both math reasoning and general domains.
summary: The paper proposes LoRR: a plug-in that (1) increases the replay/update-to-data ratio during preference-based finetuning, (2) periodically shrinks & perturbs (interpolates) the model toward its initialization to restore plasticity, and (3) mixes SFT and preference losses to improve data reuse; experiments report substantial gains on math and reasoning benchmarks.
weaknesses: 1. The method feels like a combination of three previously known method, without a unifying theoretical explanation or deeper empirical analysis of why they can work together.
2. The example  in Figure 1 showing “DPO with heavy priming” (training on the same data 200 times) seems unrealistic—no one would actually perform 200 passes over the same dataset. Moreover, the appendix itself reports that L=10 already causes degradation, making the L=200 example appear artificially extreme. A fairer comparison would be: 100% data with 1× training, 50% data with 2× training, 25% data with 4× training, and 12.5% data with 8× training, which would more meaningfully test data-reuse efficiency.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1.  What is the exact reset schedule we should follow in code? Algorithm 1 resets inside each replay loop, the text in Sec. 4 describes the reset as periodic, and the experiments in Sec. 5 say you apply it to downstream projections with ($\alpha=0.5$). It would help if you could specify the frequency (per replay, per batch, per N steps) and the exact parameter groups.
2.  Figure 5 shows a clear pattern. Replays 2, 3 and 5 are good, but 10 is bad. Can you explain the mechanism for the drop at 10? Is it that rollouts get worse because the model keeps reusing slightly off-policy data, or is it interaction with the hybrid loss that becomes too SFT-heavy, or simply that the same batch gets seen too many times? Also, do you expect the optimal replay count to change for tasks that are not math?
3.  Section 5.1 says you also include an iterative training style (Iter. n) for each preference method, with 3 iterations and re-rolling through the reward model, and Tables 4 and 5 report Iter. n numbers for Qwen2.5 and Llama3.2. Could you spell out the exact loop for Iter. n? For example, do you regenerate preferences from the current policy at every iteration, do you keep the same 8k pool and only update scores, and do you also reset between iterations? Right now Iter. n is used as a baseline but it is not described at the same level of detail as LoRR.
4.  In Figure 1 you make the central point that heavy priming on the first batch damages later learning and also affects other MMLU-Pro domains. Do you believe the same conclusion holds for long-thinking models that already generate multi-step chains and may redistribute attention across steps? It would be useful to know whether the primacy effect you saw with DPO on Llama-style models transfers to models that run long, more general reasoning patterns.
5.  The SFT weight increases with the replay index and in the example it tops out at 0.5. Was that chosen to prevent the preference signal from being drowned out, or was it just a convenient value? If you let it reach 1.0 on the last replay, do you get better exploitation, or does the model collapse faster?
6.  Since LoRR explicitly reuses the same preference data more times, how robust is it to slightly noisy or biased preferences, for example if the reward model is a few points worse than ArmoRM? A small noised labels experiment would make the sample efficiency claim stronger.
soundness: 3
strengths: 1.  The problem setup is timely. The paper connects the low sample-efficiency of preference post-training to a familiar RL idea, namely that early offline data can dominate, so the motivation is easy to read.
2.  The method is simple to plug in. It is basically higher replay, a shrink and perturb style reset, and a replay-aware SFT mix, and it is written to work with multiple preference optimizers rather than a single one.
3.  On the intended low-data math setup, the LoRR variants are consistently above the corresponding base preference learners and also above the iterative baselines listed in the appendix, which fits the paper’s story.
4.  The paper explains LoRR using ideas from high replay in RL, so it does not look like a collection of unrelated tricks.
summary: The paper argues that preference-based LLM post-training suffers from a primacy-style effect where very early data dominates the rest of training. It proposes LoRR, which keeps replaying the same batch several times, then interpolates the model back toward the initializer and increases an SFT term so the model does not get stuck. When LoRR is attached to several preference losses, the paper reports steady gains on math and reasoning benchmarks using about 8k preference data.
weaknesses: 1.  The algorithm in Sec. 4 is not tight enough for someone to reimplement without guessing. In Algorithm 1 the condition on the rollout ratio is written in a way that flips the intended Bernoulli test, the triple $(x, y, y'_l)$ is inserted without saying what $y$ is, and the reset is shown inside the replay loop even though the text elsewhere calls it periodic.
2.  Figure 5 does contain a replay ablation and it is good to see that. It shows that replay counts 2, 3 and 5 improve the average over six math tasks, while 10 makes things worse. So the data actually say there is a useful middle range and that pushing replay too far harms rollout quality, which is a familiar RL pattern. The introduction, however, still talks about enabling a replay number that is much higher than before, and this does not really match the curve in Figure 5, because the curve peaks around 3 to 5 and then drops. It would help to explain why the paper fixes on ($L=3$) even though ($L=5$) also looks strong.
3.  Important hyperparameters are kept fixed. In Sec. 5 they simply set ($\alpha=0.5$) and apply the reset to downstream projections, but there is no experiment showing what happens if ($\alpha$) is smaller or if more layers are reset or if the SFT cap is different. Since the purpose of LoRR is to avoid overfitting while replaying, this makes the method look more sensitive than the text suggests.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see Weaknesses above.
soundness: 3
strengths: 1. Targeted solution to key LLM optimization challenges: LoRR effectively addresses two critical pain points in LLM post-training: low sample efficiency of offline data utilization and primacy bias leading to early overfitting. Unlike existing methods that either re-generate data to maintain on-policy characteristics or are limited by SFT's few-epoch training, LoRR achieves efficient data reuse through high-replay training while preserving network plasticity via periodic resets, filling the gap in efficient offline data utilization for LLM optimization.
2. Strong generality and practicality: As a plug-and-play framework, LoRR can be seamlessly integrated into mainstream preference-based optimization methods without requiring major modifications to existing post-training workflows. Experiments on multiple model families (Llama3.2, Qwen2.5) demonstrate its stable effectiveness across different architectures, making it easily applicable to existing LLM optimization pipelines and enhancing its industrial application value.
summary: This paper addresses the low sample efficiency and primacy bias issues in post-training optimization of Large Language Models. It proposes a general plugin framework called LoRR, which integrates three core mechanisms: high-replay training, periodic parameter reset based on the Shrink and Perturb strategy, and hybrid optimization combining Supervised Fine-Tuning and preference-based losses. By maximizing the reuse of offline data batches and mitigating overfitting caused by high-replay training, LoRR aims to enhance the performance of preference-based optimization methods. Extensive experiments on mathematical reasoning and general reasoning benchmarks show that LoRR can significantly improve the performance of various preference optimization methods and enable iterative DPO to achieve performance comparable to complex RL-based algorithms with higher sample and computational efficiency.
weaknesses: 1. Limited generalization across model scales: LoRR’s experiments are restricted to models with 3B-7B parameters. Larger models typically have stronger resistance to overfitting and different network plasticity, this limitation makes it impossible to confirm whether LoRR maintains effectiveness across diverse model sizes.
2. The reset strategy relies on a fixed interpolation factor α , but the paper does not analyze how α affects performance across different datasets or tasks. For example, whether sparse datasets require smaller α to retain more initial parameters, or whether complex reasoning tasks need larger α to accept more new updates. Additionally, the replay number L is fixed at 3, and no adaptive adjustment mechanism is proposed. This one-size-fits-all hyperparameter setting may limit LoRR's adaptability to diverse task scenarios.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: please see the weaknesses.
soundness: 2
strengths: 1. The LoRR method is mechanically simple and well-motivated.
2. It provides a easy-to-implement plugin that delivers substantial performance gains.
summary: This manuscript introduces 'LLM Optimization with Reset Replay' (LoRR), a plugin method designed to enhance preference-based fine-tuning of Large Language Models (LLMs). The core idea is to enable high-intensity training on a single data batch by combining a high replay number with periodic parameter resets and a hybrid optimization objective. While the individual components of LoRR are derived from existing concepts in machine learning. The experimental evidence presented is compelling, demonstrating remarkable computational efficiency compared to standard methods and even complex RL-based algorithms.
weaknesses: 1. The claim of novelty requires careful qualification. The manuscript does not introduce a fundamentally new algorithmic concept. The Reset Replay, iterative DPO was proposed before. The core contribution lies in the  application of established techniques to a specific, high-impact problem domain.
2. The experiment results are not enough to prove its effectiveness. In Table 2, the performance of Qwen2.5-7B-DPO-LoRR is not better than Qwen2.5-7B-DPO-LoRR or those SFR+RFT methods [1,2].  
3.  The use of fixed hyperparameters (L, α) is a clear limitation.  The author should focus on developing adaptive schedulers that can dynamically adjust the replay number and reset intensity based on training state.
4. The comparsion with pure SFT methods using less expert data (such as LIMO，S1) should be given. The efficiency of the proposed method and iterative DPO methods should be compared in detail.