Title: Corrector Sampling in Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language modeling

Sampling
Abstract: Autoregressive language models accumulate errors due to their fixed, irrevocable left-to-right token generation. To address this, we propose a new sampling method called Resample-Previous-Tokens (RPT). RPT mitigates error accumulation by iteratively revisiting and potentially replacing tokens in a window of previously generated text. Fine-tuning a pretrained 8B parameter model with RPT for only 100B resulted in ~10% relative improvements on reasoning and coding benchmarks compared to the standard sampling.


Peer reviews:
comments: SUMMARY:
This paper introduces Resample-Previous-Tokens (RPT), a new sampling method for autoregressive language models. RPT iteratively revisits and replaces tokens in a window to reduce error accumulation. Fine-tuning an 8B model with RPT for 100B tokens improves reasoning and coding benchmarks by 10%. RPT preserves next-token-prediction quality and speed, outperforming standard sampling.

STRENGTHS AND WEAKNESSES:
#### Strengths
1. The paper proposes a good solution to save the token consumption of large models
2. RPT maintains next-token prediction quality.
3. Both theory and experiment have proved that the upper limit of RPT is higher than that of NTP.

#### Weaknesses
1. The processing of position coding is relatively rough, the sequence order is modified, but no ablation experiment is performed on position coding.
2. The paper does not provide a larger scaling. Can the mode modify the previous 512 tokens?

QUESTIONS:
1. What does the parameter n in Table 1 represent?
2. Llama-3 uses RoPE, which already has relative position information. Why still need to add a new positional encoding? Can you directly set the position to $\sigma$?
3. Why do the authors claim "preserving their next-token-prediction speed"? During inference, each token needs to be predicted at least twice. Does this mean that the decoding speed is twice as fast?
4. In the Appendix. B, $\omega=3$ is used for training, but $\omega=2$ works better for inference. Is this in line with expectations? Does this mean that the scaling limit of this method is very low? After all, in code and mathematical reasoning scenarios, a large number of tokens often need to be modified.

LIMITATIONS:
see weaknesses
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new sampling method, resample previous token, to allow the LLM to correct the previous tokens by themselves. The authors design a new training objective by modifying the positions/orders of tokens in a sequence to make it compatible with existing training infrastructure for next token prediction in autoregressive model. The authors provide theoretical analysis on why the correction prediction (conditioned on the future token) is easier than standard next token prediction. The experiments on coding benchmark show that the proposed method improves over the next-token-prediction baseline in pretraining.\n\nSTRENGTHS AND WEAKNESSES:\n**strengths**\n- Overall, the paper is well written and the illustrations such as fig.1 and 2 are clear.\n- The idea of RPT sampling to allow the LLM to self-correct the generated token is novel. Moreover, the authors also explain the effectiveness by an asymptotic analysis on approximation error.\n\n**weaknesses**\n- My major concern on this paper is on the training paradigm of RPT: In inference, RPT corrects token in the middle (e.g., when predicting $x_3$ by $x_1,x_2, x_4$, the $x_1,x_2,x_4$ are at the positions $1,2,4$). However, in training, RPT uses different positions of tokens after the corrected token (e.g., the $x_1,x_2,x_4$ are at the positions $1,2,3$). Meanwhile, the training objective of $x_5$ also changes in fig.2: the input becomes $x_1,x_2,x_4,x_3$ instead of $x_1,x_2,x_3,x_4$. Although the authors introduce new position embedding, I still feel there is large gap between training and inference.\n- In experiment, the authors train the model by perturbing data with window size=3. However, the table in Appendix B.1 shows that window size =3 achieves similar or even worse performances than window size =2, which also suggests the training inference gap.\n\nQUESTIONS:\n- What is the definition of $\\hat{p}$ and $p$ in Eq.(4)? Is the former an approximation and the latter the ground-truth distribution? It will be more clear if there are some explanations.\n- Most evaluation comparisons are on coding and math (only GSM8K). What's the performance of proposed method on other domains? E.g., general QA, science.\n- Why do you use 90% data for normal next-token prediction training and only 10% for RPT? Do you have any ablation of data proportion on final performances? For example, what if you use the full dataset for RPT training?\n- In general, how long will it take in RPT inference compared to NTP for window size=2?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCorrector sampling proposes Resample Previous Tokens (RPT): a novel language modelling task and sampling strategy that facilitates revision of previously generated tokens within a fixed sized lookback window conditioned both on the prior context and subsequently generated tokens. The authors formally introduce their method and provide a robust theoretical analysis of RPT and Next Token Prediction (NTP) asymptotic errors. The error analysis is empirically validated with a synthetic experiment which agrees with their analysis that RPT exhibits a lower error than NTP. Finally, the authors pre-train a Llama-3-8B model from scratch for 900B tokens with the classic NTP task and use this checkpoint to further fine-tune two variants: 1) AR-F which is fine-tuned on an additional 100B tokens in the pure NTP task; and, 2) a model trained for RPT by randomly swapping target indices and carefully manipulating positional encoding. The results demonstrate that even in the purely auto-regressive sampling setting, RPT improves model quality on a variety of reasoning and coding benchmarks. Further, when using iterative RPT sampling, the model quality is further improved.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The proposed methodology appears to be novel. \n* The submission includes both high-quality theoretical analysis and compelling empirical evidence that the proposed technique materially benefits the RPT trained models compared to pure NTP models. \n* The submission is relatively clear; however, some notation could be made more consistent and a few statements clarified as outlined below. \n* The results are very promising and suggest a potential improvement to current the pre-training paradigm. In particular, it is very significant that RPT trained models improve in quality *even when auto-regressively sampled* (i.e., with no RPT iterations to actually revise past tokens). This suggests that the representations learned by the model during RPT are beneficially even for NTP. This is a very important outcome since, if I understand the sampling procedure correctly, RPT requires significantly more computation than pure auto-regressive decoding. \n\n## Weaknesses\n* The main weaknesses are some unexpected results in the empirical results where AR-C out performs AR-F and/or the RPT variants; for instance, the C# and PHP tasks for AR-C outperforming AR-F and RPT, respectively. \n* Some additional information on the empirical experiments would benefit the reader and improve reproducibility. \n* While the authors demonstrate the benefit of RPT training using only 10% of the pre-training budget; this still represents a very significant amount of training (100B tokens). It could be particularly impactful if RPT training could be extended to a more lightweight fine-tuning setting of instruct tuned model checkpoints. \n* Further discussion on the computational requirements of RPT sampling would improve the work and help clearly establish some of the limitations of RPT sampling. \n* Additionally to the above, since RPT requires additional computation, an interesting ablation would be to compare RPT with compute-matched test-time-scaling such as beam search or Best-of-N parallel sampling with the NPT trained checkpoint. Does RPT sampling provide a benefit in a strict compute-matched setting?\n* Some potentially imprecise notation:\n\t* L72: Shouldn’t $w=1$ represent the NTP sampling procedure as defined? The most recent token generated in equation 6 is $x_{i+w-1}$. This seems to be the case in algorithm 1 where w is explicitly restricted to $\\geq 2$. \n\t* L80: It appears that $k$ should be restricted to the domain $k \\in \\{1, …, n-w+1\\}$ since we can’t permute tokens within a window size of the last token in a sequence; there would be no future tokens to condition RPT sampling with. \n* In Table 1, $n$ is used to signify the number of RPT iterations. However, elsewhere in the paper such as section 5.3 $k$ is used. Unifying the notation between these sections would improve readability. Further, it’s unclear what n=0.5 or 1.5 refer to; does 0.5 represent a sliding window of 2 with a stride of 2 such that only every second token is resampled?\n* In Appendix B.2, the authors refer to “naive” sampling. It seems like this is simply nucleus samping. If so, please simply state nucleus sampling as it is less confusing to the reader than introducing a second, less common term. \n* L263 states that “greedy decoding is always used in the main paper”, but based on section B.2 it seems like what was actually reported in table 1 is greedy decoding **+ confidence**. Please clarify.\n\nQUESTIONS:\n## Key questions\n* Could the authors speculate on the cause of surprising results in Table 1. I would have naively expected AR-F to outperform AR-C in all cases and similarly was surprised to see AR-C significantly outperform in the PHP category. \n* Was only pre-training data used? Any SFT or RL training? Please discuss the high-level content of the corpus used; was it curated for the benchmark tasks specifically or drawn from a larger, more general dataset?\n* Can RPT training be extended to previously instruct-tuned checkpoints? What are some potential challenges to extending RPT training to this paradigm? \n* What is the computational overhead of RPT samping? Am I correct in my understanding that for a single RPT iteration (k=1) and a window size w, each output token requires w forward passes? \n* Please clarify the imprecise or misunderstood notation as requested in the weaknesses section. \n\n## Minor/Typos:\n* L129: “next-token-**rediction** (**NPT**)”\n* L132: NPT used in lieu of NTP twice. \n* L193: Should be revised as “left” and “right” instead of “top” and “bottom”, respectively. \n* L304: “**fo**”\n\nLIMITATIONS:\n* **No**. The authors stated in the checklist that limitations were discussed; however, I could not find any related discussion. I encourage the authors to consider the limitations of their method in particular from the perspectives of computational efficiency, performance (latency/throughput), and extending RPT training to previously instruct-tuned models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Resample-Previous-Tokens (RPT) for autoregressive (AR) LLMs that aims to mitigate the issue of error accumulation during token generation. Traditional AR models suffer from irrevocable errors in token generation, as once a token is produced, it cannot be revised. RPT iteratively revisits previously generated tokens within a fixed-size window and replacing them based on context. This method is designed to be easily integrated into existing AR models. The paper demonstrates that fine-tuning a pretrained 8B parameter model with RPT results in a 5-10% improvement in reasoning and coding tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros**:\n\n1. The paper addresses a significant issue in AR: error accumulation due to the fixed left-to-right generation process. RPT provides a novel and practical solution to enhance the model performance.\n\n2. RPT is straightforward and easy to implement in AR training. It adds minimal computational overhead.\n\n3. The paper provides empirical validation of RPT via reasoning and coding tasks, e.g., HumanEval+, GSM8K, MBPP. \n\n4. The authors give a solid theoretical foundation for the method, which provides an asymptotic error analysis comparing RPT with NTP sampling.\n\n**Cons**:\n\n1. The paper does not provide sufficient detail on how RPT training influences the model during inference.\n\n2. In Table 1, the meaning of $n=1.5, 2.5$ is not clearly explained. Further discussion is needed to clarify this term and provide more context. Does it refer to different values of the window size or some other hyperparameter? Also, the results would benefit from a comprehensive analysis of how varying $n$ impacts the model’s performance.\n\n3. It lacks comprehensive analysis on the introduced extra hyperparameters, such as $s$, $q$, $n$, $w$. The choices and selection strategy are required to be carefully compared.\n\n4. RPT introduces extra relative position embedding layers, which requires further discussion about its potential drawbacks. It introduces extra parameters to optimize. How are these embeddings injected into the model’s layer inputs (added on each layer or on the token embedding layer), and what are the implications of this additional layer in terms of optimization and model efficiency?\n\nQUESTIONS:\n1. What is the ratio of NTP and RPT? How did the ratio of RPT have the effect on the model performance?\n\n2. The window size w is set to 3 during training while w=2 during sampling, why did not keep them the same?\n\n3. The RPT shares the similar format to FIM (Fill-in-the-Middle). The discussion of relation to FIM could be helpful.\n\n4. In Table 1, what is the meaning and choices of $n$? A further discussion and analysis is required.\n\n5. The paper continue-trained LLM on RPT for 100 tokens after AR training for 900B tokens. How is the scaling trend on pure RPT?\n\n6. How about mixing the NTP and RPT? What is the performance of it?\n\n7. How did the author conduct RPT during inference on the evaluated experiments? A comparison results among varying sampling hyperparameters such as $w$ is required.\n\n7. What is the weakness of RPT brought during pretraining?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}