Reflection-Window Decoding: Text Generation with Selective Refinement
Authors: Zeyu Tang
Zhenhao Chen
Xiangchen Song
Loka Li
Yunlong Deng
Yifan Shen
Guangyi Chen
Peter Spirtes
Kun Zhang
Venue: ICML 2025 poster (2025)
Keywords: Autogressive Decoding
Text Generation
Reflection Window
Selective Refinement
Large Language Model
Primary area: deep_learning->large_language_models
TL;DR: We propose reflection-window decoding for text generation to address the inherent shortcoming of the purely autoregressive decoding approach.

Abstract: The autoregressive decoding for text generation in large language models (LLMs), while widely used, is inherently suboptimal due to the lack of a built-in mechanism to perform refinement and/or correction of the generated content. In this paper, we consider optimality in terms of the joint probability over the generated response, when jointly considering all tokens at the same time. We theoretically characterize the potential deviation of the autoregressively generated response from its globally optimal counterpart that is of the same length. Our analysis suggests that we need to be cautious when noticeable uncertainty arises during text generation, which may signal the sub-optimality of the generation history. To address the pitfall of autoregressive decoding for text generation, we propose an approach that incorporates a sliding reflection window and a pausing criterion, such that refinement and generation can be carried out interchangeably as the decoding proceeds. Our selective refinement framework strikes a balance between efficiency and optimality, and our extensive experimental results demonstrate the effectiveness of our approach.

Reflection-Window Decoding: Text Generation with Selective Refinement
Zeyu Tang"
Zhenhao Chen
Xiangehen Song'
Loka Li Yunlong Dengà Yifan Shen Guangyi Chen
Peter Spirtes'
Kun Zhang!
Abstract
purely autoregressive decoding, may be overly optimistie. Despite significant recent progress (Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020; OpenAI, 2023; Tou- vron et al., 2023; Gemini Team, 2023; Llama Team, 2024; Gemma Team, 2024; Abdin et al., 2024a; DeepSeck-Al. 2025), how to approach the optimal text that one can pos- sibly sample from model still remains an open question.
The autoregressive decoding for text generation in large language models (LLMs), while widely used, is inherently suboptimal due to the lack of a built-in mechanism to perform refinement and/or correction of the generated content. In this paper, we consider optimality in terms of the joint prob- ability over the generated response, when jointly considering all tokens at the same time. We theo- retically characterize the potential deviation of the autoregressively generated response from its glob- ally optimal counterpart that is of the same length. Our analysis suggests that we need to be cautious when noticeable uncertainty arises during text generation, which may signal the sub-optimality of the generation history. To address the pitfall of autoregressive decoding for text generation, we propose an approach that incorporates a sliding reflection window and a pausing criterion, such that refinement and generation can be carried out interchangeably as the decoding proceeds. Our selective refinement framework strikes a balance between efficiency and optimality, and our extensive experimental results demonstrate the effectiienees of our approach.
Previous works have demonstrated challenges faced by autoregressive decoding in terms of the handling of long sequences (Wu et al., 2021) and the inefficient inference (Lin et al., 2021; Li et al., 2022). At the level of decoding, other than the traditional greedy search, Holtzman al. (2020) proposed Top-p sampling (also known as Nucleus Sampling), stochastic method that adjusts the next-token set based the shape of conditional probability mass function. Alternatively. different from the cumulated probability mass, Top-k sampling limits the number of available options when sampling for the next token (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019). Another empirical technique involves adjusting the shape of probability distribution with the temperature hyperparameter (Fan et al., 2018; Holtzman et al., 2018; Shih et al., 2023; Peeperkorn et al., 2024), or decoding as optimization with more than one models (Li et al., 2023).
In order to improve the sampling efficiency. speculative decoding approaches have been proposed, where multiple tokens are predicted in parallel as one were sampling from the model (or its lighter counterpart) repetitively (Leviathan et al., 2023; Chen et al., 2023a; Xia al., 2023; Kim et al., 2024; Sun al., 2024). Efficient inference with beam search (Xie et al., 2024; Zhu et al., 2024; Wei et al., 2024; Yang et al., 2024b) and probabilistic programs (Lew et al., 2023) have been explored in the recent literature. Improving gen- erated content through high-level behaviors, e.g., through instructing self-correction or conducting self-improvement with external or internal feedback (Yao et al., 2022; Bai et al., 2022; Pan et al., 2023; Shinn et al., 2023; Ganguli et al., 2023; Chen et al., 2023b; Kim et al., 2023; Tyen et al., 2023; Madaan et al., 2024), has also been studied.
1. Introduction
It is a well-known insight in optimization theory that coordinate-wise optimization, conditioned on the previously optimized arguments along each axis sequentially, gener- ally does not guarantee finding the global optimum (Törn & Zilinskas, 1989; Nocedal & Wright, 1999). Similarly, in the context of decoding in large language models (LLMs), expecting to achieve a globally optimal response by sequen- tially accumulating per-token optimal decisions, as done in
'Equal contribution 'Carnegie Mellon University 'Mohamed bin Zayed University of Artificial Intelligence. Correspon- dence Zeyu Tang <zcyutangecnu, edu>, Zhenhao Chen <zhenhao. chen@mbzudl w.a.aaa...... .ac.a>>
While previous literature has explored various methods to enhance generated content, the fundamental limitation of autoregressive decoding for text generation remains
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Snllection-iindor Decoding: Text Generation with Selective Refinement
under-explored. This gap represents a distinet perspective, different from high-level model behaviors (e.g., self- refinement studied in Madaan et al. 2024) or inference efficiency (e.g,, various speculative decoding methods surveyed in Xia et al. 2024). In this paper, we theoretically characterize the inherent shortcoming of autoregressive decoding and propose an empirical method to mitigate this issue. Our contributions can be summarized as follows:
Xi
X5
81
Xs
(a) (Sub-)goals jointly satisfied by the optimal sequence
(b) Purely autoregressive way of decoding for text generation
Illustrative diagrams of different dependence patterns among variables representing tokens or phrases in text generation. Panel (a): the dependence pattern among variables in the optimal sequence where there are (sub-)goals S:'s to achieve, specifying conditions or constraints that should jointly be satisfied by X;'s. Panel (b): the autoregressive way of text generation, where X; is only allowed depend on X, jf < i.
Figure 1:
We theoretically characterize the sub-optimality of autore- gressive decoding for text generation, demonstrating its lack of a built-in mechanism to perform refinement/cor- rection of generated content at the decoding level.
We propose a framework involving a sliding reflection window and a corresponding pause criterion, enabling an interchangeable process of refinement and generation.
sub-goals within a single scene altogether serve the purpose of furthering the development of the story. We model sub-goals in terms of selection variables S, since they represent constraints or objectives to achieve, which involve certain criteria be satísfied over the variables that they operate upon.' 1 As we can see from Figure I(a), the variables in optimal sequence (the novel in this example) XT's jointly satisfy criteria, or optimize objectives, specified by sub-goals S This indicates that the best X; in the optimal sequence depends on best values of all other Xj's. However, with an autoregressive way of text generation, as illustrated in Figure 1(b), we only allow X; to depend on Xj' 's if < i, which is clearly sub-optimal.
Through extensive empirical evaluations, our approach demonstrates significant improvement over existing de- coding approaches, and maintains performance compara- ble or superior to beam search while being more efficient.
2. Motivation and High-Level Illustration
In this section, we first present our motivations behind ad- dressing the inherent shortcoming of purely autoregressive decoding for text generation (Section 2.1). Then in Sec- tion 2.2, we present high-level summary of our proposed approach involving interchangeably switching between re- finement (upon reflection on previously generated content) and generation (of the additional new content).
2.2. Selective Refinement Through Reflection Window
As illustrated in Section 2.1, one outcome of limiting the dependence pattern to the autoregressive structure is the lack of a built-in mechanism to correct or refine previously gener- ated content at the decoding level, since what was generated is not subject to further edit (if without further processing). In this subsection, we present a high-level summary of our approach that attempts to address this issue.? ? Specifically, we propose a selective refinement framework that facilitates interchangeable process of refinement and generation, so that the overall response satisfies requirements or objectives that operate jointly over all involved tokens.
2.1. Inherent Shortcoming of Autoregressive Decoding
Insight from the optimization theory highlights the gap between coordinate-wise accumulation of optimum and the global optimum (Törn& & Zilinskas, 1989; Nocedal & Wright, 1999). Recent research advances in cognitive linguisties have also argued that language is primarily a tool for communication (for humans) rather than thought (Fedorenko et al., 2024). While language naturally unfolds in one-dimensional sequence, its underlying dependence pattern extends beyond a purely autoregressive structure.
As illustrated in Figures 2 and 3, we can use fast and slow pointers on the generated content to form segments of a certain length, namely, the sliding reflection window, and perform potential refinements within this sliding window as the text generation proceeds. Our reflection-winde decoding framework allows for revision before the entire output is completed, which offers several advantages.
Let us consider an example of writing novel. For a long- format writing like novels, outlining (also referred to as plotting) is essential for structuring ideas, planning narra- tives, and crafting engaging drafts (King, 2000; Serravallo, 2017). Sub-goals refer to relatively small and achievable tasks that guide the author through each stage of the story, for instance, the setting of the circumstance, the element of tension and emotion, the sensory imagination of the scene.
'This modeling choice is consistent with the modeling of causal relations among variables of interest through selection variables in directed acyelic graph (DAG) (Spirtes et al., 1993; Pearl, 2009). "We present in detail our technical approach in Section 5.1. "The naming of pointers is motivated by Kahneman (201 I).
As we illustrate in Figure 1(a), X²'s represent words or phrases in the novel, and S,'s represent sub-goals, which may be related in a hierarchical way. For instance,
2

---

Reflection- Decoding: Text Generation with Selective Refinement
Reflection Window
After Selective Refinement
Reflect (& Pause)
(Refine &)
Generate
Slow
Fast
Slow
Slow Fast
Figure 2: Overview of our approach to address the inherent shortcoming of autoregressive decoding for text generation, where the LLM interchangeably switches between refinement and generation. The fast pointer pauses if the pausing criterion is triggered, indicating the existence of a potential issue in the generated history. Then, the model refines/corrects the content between the fast and slow pointers before continuing generation, so that the slow pointer catches up with the fast pointer before the latter can move on. Reflection window refers to the content between the fast and slow pointers.
First, we can improve the generated content in a timely man- ner. If there are multiple potential issues in the generation history, the revision after finishing the generation can be in- efficient since we allow errors to accumulate. In other words, without built-in mechanism for refinement or correction at the decoding level, we are forced to rely on high-level model behaviors and operate at a coarser granularity. This often involves regenerating entire sentences (rather than refining words or phrases), and/or editing through multiple iterations (rather than interchangeably generate and refine in single run), as in self-correction and seff-improvement approaches (Yao et al., 2022; Bai et al., 2022; Pan et al., 2023; Shinn et al., 2023; Ganguli et al., 2023; Chen et al., 2023b; Kim et al., 2023; Tyen et al., 2023; Madaan et al., 2024).
sufficiently trained and can perfectly capture any autoregres- sive decomposition of the joint density, the autoregressive way of text generation can still deviate from the globally optimal response, even for the well-defined objective of max- imizing output probability given a fixed length (setting aside whether this objective fully aligns with the ultimate goal).
Let us denote a token from the vocabulary as wi V. whose index in the vocabulary is E V We use "i: j" to denote the increasing integer sequence from to j if e.g., 1 t == 1,2,..... t if 1, otherwise, i Ø.
Definition 3.1 (Globally Optimal Length-T Response). We say sequence Wvv [1]"";; ... [v] is globally optimal among all possible length-? responses following the prompt Xoo if it has the highest ground-trute conditional probabil- ity, denoted by f(X
Second, our focus on selective refinement during decoding is not solely driven by inference efficiency considerations. The primary goal of previous approaches, e.g., speculative decoding (Leviathan et al., 2023; Chen et al., 2023a; Xia et al., 2023; Kim et al., 2024; Sun et al., 2024; Xia et al., 2024), is to accelerate sampling from (a lighter version of) the original model, while the underlying decoding mecha- nism remains purely autoregressive.
X where t E [1,7]:
vi = (vi v (2] (2)....,
= argmax
f(XIT = Way Woz
Way
Xgo). (1)
Definition 3.2 (Stepwise Optimal Length-T Response). We say sequence Wer [1] Way ... Wer(7] is stepwise optimal for prompt X if the sequence consists of tokens that cor- respond to highest token-by-token conditional probabílities, denoted by g(X, X1.t-1.X.. xo) where E [1,
Third, due to the one-dimensional progression of text gen- eration, our sliding reflection window mechanism, given pausing criterion, enables timely and assured detection of is- sues in the generated text. Our framework complements pre- vious approaches, and furthermore, offers versatility. One can incorporate pausing criteria and refinement//orrection methods at the decoding level, while preserving the ability to further leverage strategies that rely on high-level behaviors.
V [1] = argmax g(x1 = Ww, Xggo,
vr [2] = argmax g(X; wing Xi ( Xgo):
(2)
[T[ = argmax 9 (Xr = Wer X and
The empirical pausing criteria we use (detailed in Section 5) are guided by our theoretical characterization of the sub-optimality of autoregressive text generation, and to this theoretical analysis we now turn.
-y X1.T = Werl wxr(r-11)
and रT (Vr ...., (.....
In general, the longer the sequence, the lower the joint prob- ability tends to be. The fair comparison of optimality should be length- -eecif,, and the optimal response of a shorter
3. Theoretical Characterization of the Sub-Optimality of Autoregressive Decoding
'For notional clarity, instead of g(X, |
X1.-11
We theoretically characterize the sub- optimaailyl of autore- gressive text generation. We show that even if L.M is
Xitt-1, Xzo = x<o). we use shorthand notations in the condi- tioning set, i.c., g(X, XIa-1. Xgo). We will adopt this simplifi- cation throughout the paper, as long as it remains unambiguous.
3

---

Decoding: Text Generation with Selective Refinement
Prompt
Model Response
The word that does not belong with the others is "tyre." " The other three terms "steering wheel," "car," and "engine" are all related to a vehicle, specifically a car. A "tyre." on the other hand, is component of vehicle, but it's a part of the car rather than
repluced specifie part, whereus
part of the car
Which word does not belong with the others, among tyre, steering wheel, car, engine?
-ure
vehicle, but it's specifii part, whereas the other three
Pausing criterion triggered
words are more general terms related to the vehicle as whole ... but one word that is the least related to the others, would say "car."
Slow
Fast
Continue to generate
Figure 3: A concrete example demonstrating our reflection- decoding.
length is not necessarily identical to the prefix of the optimal response that is longer in length. For instance, if we were to use 10 words to distinguish between joint and conditional densities, one might say "joint density combines all vari- ables; conditional adjusts for known variables." However, we can use 15 words, one might say "joint density reflects combined probabilities of all variables; conditional density adjusts probabilities given known variables." Therefore, we explicitly keep the length T in the notation of vocabulary indices of tokens that constitute the length-T responses.
low the prompt Xço, i.e., the ground-truth probability mass function f(X Xxoo) is injective for any given T 0. Assumption 3.5 (Irreversible Advantage Once Mani- fested). When stepwise optimal length-T response from an oracle (Assumption 3.3) autoregressive LLM Wez [ [[]] ... ".. is not the globally optimal length- T response Wy; [1]W<; ... (.. then if the deviation manifests itself at the length-L (1 < L S 2) prefix- sequences, the advantage of the globally optimal length-T response will not be reversed afterwards:
Assumption 3.3 (Oracle LLM). We say an autoregressive LLM is an oracle LLM, if the following relation holds for any response of a length T 2 1:
ir E (L,2), f(XI.. = [1r(1] [ (c] Xgo)
< <(((X. = (1g(1) ... ; [t] Xgul,
then VM E JL,T),((KLN) = werll ... [er[ X
[(Xur Xgo) = 1XX, X
(3)
f(XI:M = -Wv--) ............... .. Xga).
Assumption 3.3 specifies that, after given the prompt or generated text history Xso. an oracle (or very well-trained) LLM can recover the ground-truth probability of X1:1 whole follows Xgo, by multiplying token-by-token generat- ing probabilities in an autoregressive way. We would like to note that Assumption 3.3 only states that an oracle LLM can perfectly capture the autoregressive way of probability partitioning of text sequences, and this itself does not guar- antee the equivalence between stepwise optimal response and the same-length globally optimal response for T >>>>> 1.6
Assumption 3.5 specifies that if the advantage (in terms of a higher ground-truth conditional probability) of the globally optimal length-?' sequence can be observed at the length-L prefi--squuene sequence, such advantage will not be reversed when considering longer prefix-sequences.
Theorem 3.6 (Indication of Deviation from the Globally Op- timal Length-T Response). Given the prompt X when an oracle LLM (Assumption 3.3) generates a stepwise op- timal length-T response which is not the globally optimal response with the same length, let T, S T denote the min- imum length of prefix-sequence needed in order for such deviation manifest itself (Assumptions 3.4 and 3.5). Then, the deviation from the globally optimal response happens at some step K- < L. Furthermore, the conditional probability when generating the token Wez E V is strictly smaller than a positive number, which itself is strictly smaller than 1, iee,
Assumption 3.4 (Strict Preference Among Same-Length Sequences). For any two length-TT different sequences fol- lowing the prompt X there is a strict preference between them in terms of the ground-trum truth conditional probability f(Xır Xca). In other words, the ground-truth condi- tional probabilities of two length-T' sequences equal to each other if and only if the sequences are identical.
max g(Xr = w. XI:L-1 Woy 1h 11, Xgo) EL
Assumption 3.4 specifies that from the ground-trufe -trtt con- ditional probability perspective, there is strict preference between how well two different same-length responses fol-
and et. 1(XI. [vz(1)) ... Win (1- 1y [L]
X L.
f(XILL-1 (ey Wey IL-1)
X
Theorem 3.6 provides a necessary (but not sufficient) con- dition for the deviation of the stepwise optimal length-T response from the same-length globally optimal response. The uncertainty (i.c., low conditional probabilities) in gen- erating the next token can result from different factors. For
'Here, we implicitly assume that the context length LLM is sufficiently large to allow for meaningful discussion.
'When T = i.e.. if the response is of length 1, the stepwise optimal is just the globally optimal for oracle I.IM, since there only one step in total, and f(X, Xgo)= = g(X, Xgo).
4

---

Dellection-Window Decoding: Text Generation with Selective Refinement
Optimal rate (greedl))
Optimal rate (greedy)
Optimal rate (gready)
Optimal rate (gready)
reasoning
2.0
wriling
humenities
0.6
math
84-
stem
roleplay
22
extraction
15
10
5
15
coding
Num tokens
new
of tokens
all
(a) O'th token
(b) 20'th token
(c) 50 th token
(d) 200'th token
Figure 4: The probability that greedy decoding can attain globally optimal response, with respect the number of newly generated tokens, and with different starting positions in the generation history. The legend is shared across sub-figures.
instance, previous mistake or detour makes it challenging to continue in any way that could possibly satisfy the goal specified by the prompt. Such uncertainty can also result from multiple valid ways to proceed in order to achieve the goal. Although we do not have access the ground-truth conditional probability f(X Xgo), Theorem 3.6 sug- gests that when noticeable uncertainty arises, one should to be cautious of a potential deviation from the globally optimal response in the generated text.
strategy of obtaining globally optimal sequence. Since we measure the chance that greedy decoding can attain the global optimum with the stepwise optimal response, this approximation serves as an upper bound on achievable performance, indicating the discrepancy between greedy decoding and the true globally optimal response.
Approximating Natural Language Scenarios Since the prompt or context of the generation influence model be- havior, we align experimental setting with common human-LLM interactions. Specifically, we utilize MT- Bench (Zheng et al., 2023) questions as curated prompts, which are designed to evaluate conversational chat models. These samples serve as an approximation of real-world natu- language context distributions, ensuring that our findings are grounded in practical scenarios.
Remark: Implication of Our Theoretical Analysis Our Theorem 3.6 indicate that, even if with an oracle LLM (Assumption 3.3), there is still suboptimality in the purely autogressive way of decoding. In other words, even if the I.LM itself perfectly decomposes the (conditional) probabilities (which is a far-fetched benefit assume in practice), there is still no guarantee obtaining the globally optimal sequence with purely autoregressive decoding. Our empirical evaluations (Sections 4-5) do not rely on or employ this assumption.
Findings We use Llama-3 1.1-8B-Instruct (Llama Team, 2024), and for each prompt, together with a certain length of generation history (0 means only the prompt is given), evaluate whether the joint probability of the sequence generated with greedy decoding is greater than or equal to that produced by beam search (with beam width set to 10, as proxy of the global optimum). This comparison indicates the extent to which greedy decoding deviates from the glob- ally optimal response. As illustrated in Figure 4(a), greedy decoding consistently results in suboptimal sequences, and the phenomenon can be observed with small number of newly generated tokens.
4. Sanity Check: Semi-Synthetic Settings
The implication of theoretical analysis is straightfor- ward. However, it is natural to ask whether the phenomenon actually occurs in real-world LLM decoding scenarios. To provide clear empirical evidence accompanying our theo- retical analysis, in this section, we present semi-synthetic experiments that serves as a sanity check. In particular, in moderately realistic settings, we show that greedy decod- ing for text generation with stepwise optimization results in suboptimal responses. We first outline the semi-synthetic setting, and then present the empirical findings.
In addition, the potential deviation may behave differently across various positions in the generated text. For instance, when openings of response diverge, is hard for greedy decoding to achieve optimal results afterwards. To reduce potential inductive bias resulting from the diversity at early evaluate generations starting/con-
Illustrative Approximation For any modern LLM with vocabulary size |V] (typically the order of 104 to 105), identifying the globally optimal sequence across multiple steps becomes computationally intractable, even for relatively short sequence lengths (< 100). To ensure the validity of our claim while providing a clear and accessible illustration, we adopt beam search as an approximation
stages of generation, tinuing from various positions throughout generation history, as presented in Figures 4(b)-4(d). We can observe that the deviation persists across different positions, which empiri- cally demonstrate the common existence of sub-optimality in autoregressive decoding for text generation.
5

---

Reflection-indoww Decoding: Text Generation with Selective Refinement
Algorithm 1: Reflection-We Winddwww Decoding with Selec- tive Refinement for Text Generation
additional modules: (1) a pausing criterion that spec- ifies whether should pause the generation upon reflect- ing generated content, and (2) refinemenn//orrreciion method that facilitates revision at the decoding level (if the pausing criterion is triggered). We present the pseudocode of our reflection-window decoding approach in Algorithm 1.
Input :The prompt Xgo the reflection window size d, as well as hyperparrameter-eeelooedd functions for the pause criterion IfP (Puu(s and the refine/correct method ReGenerate(.). Output: generated with selective refinement / ((slow) +- 0, (ffstt) < 1; // slow/ fast pointer response <- empty string;
Pausing Criterion Guided by our theoretical character- ization (Theorem 3.6), the reflection the decoding level needs to capture the (increasing trend of) uncertainty text generation proceeds. For empirical pausing criterion, use the conditional entropy (-) based on the next-token logits across the vocabulary. Specifically, given an LLM which models the conditional distribution g(X, (1=-1) of the token at t-th step given all the observed history X Tit-1 X1 is we use the pausing criterion o(t;o,d):
While not stopped, or response not completed Do regular decoding for the next token [y((o)));
If (fast) - ((slow) Then
x (x (x1z((nos) -1: Wg(Gmm)))
Else
concatenate and get temporary sequence
X + (x1:00000))11 Wy(tass)):
If True = ItPause IfPause(X{tto)) dd 1+((t)))) Then
True
if H - X1s(-i-1)
& Refine upon reflection
h(t;ord)
E [0, 0 1}, Else,
obtain a length-d replacement text
False
X Xittee: x -d+1:
ReGenerate (xtaaae(((xstsss (11((moss_
where a denotes the hyperparameter for the threshold of con- ditional entropy, and d denotes that for the window size (how far we look back in history, in terms of the token counts).
11
x < (x1:e(ea)) -d: di (((mm)) - :41:0(0m)))
(sloww t- {(fast); // update slow ptr
13
Else
When ((fsst) d) is True, the pausing criterion (denoted by lfPPause()) in Algorithm 1, with hyperparameter en- closed) is triggered upon reflecting on the most recent d gen- crated tokens, i.c., the length- d reflection window when the fast pointer is at {(fast) The two parameters, a and d. jointly decide the sensitivity and effective region of the pausing criterion, and present more discussions in Section 5.4.
Continue to generate
x-x; X;
(fast)
(last) + // update fast ptr
15
16
Return (x).
5. Empirical Approach and Experiments
Refinement/Correcton Method When the pausing cri- terion is triggered, tokens within the current sliding reflec- tion window need to be refined or corrected. Since beam search can approximate the global optimum relatively well, empirically we introduce a fixed-length beam search to generate a new segment with a length d (denoted by the hyperparameter-enclloed function ReGenerate(() in Algo- rithm 1), to replace the content within the reflection window. After the refinement, the slow pointer ((loow) catches up with the fast one {fastt and the model proceeds with generation while maintaining the sliding reflection window.
In Sections 5.1-5.2, we provide technical details about our empirical approach and settings of our experiments. Then in Sections 5.3-5.4, we present experimental results to demon- strate both the effectiveness and efficiency of our method.
5.1. Sifeetion-Window Decoding: Technical Details
Our findings through both the theoretical characterization of sub-optimality in autoregressive decoding for text gen- eration (Section 3), and the sanity check with empirical verifications in cemi-synthetii settings (Section 4), suggest the necessity of a built-in reflection-and-refine mechanism at the decoding level. To empirically address this issue, we propose a selective refinement framework that interchange- ably refine and generate as the response unfolds.
Versatile Decoding Framework We would like to note that our reflection-window decoding approach is highly versatile. While our empirical approach em- ploys specific pausing criterion and refinement/correction method, practitioners can customize these components by incorporating different functions, namely, I/Pause(.) and ReGenerate() :) in Algorithm 1, to meet their needs. Our selective refinement framework integrates the sliding reflec- tion window mechanism with these components, enabling simultaneous refinement and generation at the decoding
Remark:
Text typically unfolds in single direction, i.e., from the start to the end, with words, phrases, and sentences. This differentiates text from other forms of objects that occupy multiple dimensional spaces, e.g., images or videos. Tak- ing advantage of this -ne-dimensional nature, our decoding framework integrates sliding reflection window along with
6

---

Dellection-Windwr Decoding: Text Generation with Selective Refinement
level while retaining the flexibility to incorporate additional strategies, such those based high-level model behav- iors and/or speculative decoding (Section 2.2).
Win Rate (%)
80
Beam Search
Ours
60
40
5.2. Experiment Settings
We provide technical details about settings of our experi- ments, including models, benchmarks, evaluation metrics, and baseline methods.
coding
extraction
humanities
math
reasoning
releplay
STEM writing
Figure 5: Comparison of win rates between beam search and our reflection-window decoding (both against greedy decoding) on MTT-Bench across categories
LLM Models We conduct experiments using multiple models across different families/heerss. Specifically, we use 1ertaa...----IInseee
(denoted as Llama3. 1-8B), which belongs to the Llama 3.1 herds (Llama Team, 2024), Phi- 3-Mediun-128K- Instructt (Abdin et al., 2024b) (denoted as Phi3-Medium) with 14 billion parameters, Qwen2.5-14p 14B- Instruct (Yang et al., 2024a) (denoted as Qwen2.5-14B) with 14 billion parameters. (owen..5-7B-Instreee Qwen2.5-7B) with 7 billion parameters, and Mistral-Nemo- Instruct-2407 (MistralAI, 2024) (denoted as Mistral-Nemo) with 12 billion parameters.
sampling (Fan et al., 2018; Holtzman et al., 2020).? We consider greedy search one baseline method since it corresponds closely to our theoretical analysis. However, since the widely usage of Top-p and Top-k sampling (and often used in combination), we also include it as a baseline approach. Reflection-window decoding only leverages beam search when necessary, i.e., when I(Pause(-) in Algorithm I is triggered. For fair comparison, we set the beam size to in both reflection-windor decoding and the vanilla beam search.
(denoted as
Benchmarks and Evaluation Metrics Our experiments conducted on MMLU (Hendrycks et al., 2020) and MT- Bench (Zheng et al., 2023). MMLU tests model's gen- eral knowledge across 57 diverse subjects, e.g., humanities, STEM (Science, Technology, Engineering, and Mathemat- ics), and social sciences, at varying difficulty levels, making it a comprehensive evaluation of model's reasoning perfor- mance and factual knowledge. MTT-Bench, on the other hand, provides fine-grained evaluation through multi-turn conversational tasks, evaluating not just correctness, but also coherence and fluency.
5.3. Experiment Results
We compare the performance among greedy decoding, beam search, Top-k/Top-p sampling, and our reflection-window decoding. We use an entropy threshold of = 0.5 and window size of d = 4 in reflection-window decoding.
MMLU Table presents the comparison among greedy decoding, beam search, and infection-windowy decoding (with greedy search as the "regular decoding" in Algo- rithm 1) on four question categories in MMLU (STEM, humanities, social sciences, and others), using Phi3-Medium as the base LLM. Our approach demonstrates comparable or superior performance across all categories, achieving the highest average accuracy of 78.15% The results containing all MMLU subjects can be found in Table 11 in Appendix.
For MMLU, we adopt the macro averaged accuracy met- ric because the number of questions varies across subjects. For MT-Bench, for each pair of responses, we prompt the LLM judge (for which we use GPT-4o, OpenAl 2024) with two responses following the prompting method outlined in Zheng et al. (2023). The LLM judge should return deci- sion from three options: win, lose or t tie. To avoid the influence from the preference bias, for each pair we prompt the LLM judge twice with responses placed in different se- quences. When a response gets two win's (lose's) 's) or one win (Lose) plus one tie, we record the response as (not) prevailing. The rest situations treated tie. We use win rate as the measurement of performance on MT-BBenh, calculated by win rate := number number of wins number losses'
MT-Bench
We choose Llama3. 1-B as the base model in this experiment. On MT-Bench, the Reflection Window method clearly outperformed both greedy decoding and beam search. In comparisons with greedy decoding, according to assessments by GPT-4o, reflection window prevails in 66.67% of cases, a win rate significantly higher than that of beam search, which only outperformed greedy decoding in 46.3% of cases.
"If there is no limit on the computation and storage, beam search with an unconstrained beam width could yield the globally optimal output through brute force. However, in practice, maintain- ing a full frontier quickly becomes intractable, and a finite beam width is often introduced as a hyperparameter.
Baseline Methods We compare dnfecciio-windowe decod- ing with three baseline methods: greedy decoding, vanilla beam search (with finite beam width), and Top-k/Top-p
7

---

Snllettio-Windowe Decoding: Text Generation with Selective Refinement
Table I: Comparison of macro averaged accuracy on MMLU across subject categories with Phi3-Medium
Decoding Method
STEM
Humanities
Social Sciences
Others
Macro Average
Greedy Decoding
78.40%
71.92%
83.91 %
81.59%
78.14%
Beam Search
78.62%
68.65%
82.93%
79.59%
76.44%
Reflection- Windoww (Greedy)
79.06%
71.98%
83.65%
81.10%
78.15%
Table 2: Summary of comparison between Top-k/Top-p and reflection-window decoding on MMLU (multiple models)
Model
Decoding Method
STEM
Humanities
Social Sciences
Others
Average
Top-k Top-p 65.94%
58.43%
72.60%
73.06%
66.46%
1-8m
Reflection- Window
67.21 %
59.43 %
72.86%
73.41 %
67.21 %
Top-k Top-p Reflection- Window
71.36%
65.76%
79.10%
74.93%
71.97%
Phi3-Medium
73.07 %
71.31% %
79.85%
74.86%
74.73%
Top-k Top-p
83.00%
69.86%
82.74%
79.82%
77.84%
Qwen2.5-14B
Reflection- Window
83.30 %
70.33%
83.85%
81.41
78.48 %
Top-k Top-p
60.74%
50.16%
67.79% 69.22% %
65.66
59.83%
Mistral-Nemo
Reflection-Wiin Window
61.78 %
51.41 %
65.27%
60.71 %
Table 3: Regeneration metrics on MMLU with Llama3. 1lama3.1-8B 1-8B
Table 5: Acc. by d (MMLU Social Sciences, Qwen2.5-7B)
Category
ReGen. Ratio (%)
ReGen. Call
d 2
3
4
5
6
STEM
3.50
3.15
Acc. (%)
79.66
79.75
79.82
79.62
79.88
Humanities
5.04
4.27
Social Sciences
4.82
3.84
Others
5.54
4.31
improvements range from 0.88 percentage points (Mistral- Nemo) to 2.76 percentage points (Phi3-Medium), highlight- ing the effectiveness of our approach even when stochastic sampling is introduced. Notably, the largest performance gains are observed STEM and humanities categories, suggesting that reflection-window decoding particularly beneficial for reasoning -eeeonnnnn--vvyyyyyy tasks. This aligns with obser- vations from the MTT-Bench experiments, which also demon- strate that our approach excels in tasks demanding structured logical thinking and complex problem-solving. We present further results from Table 12 to Table 15 in Appendix.
Table 4: Acc. by (MMLU Social Sciences, Qwen2.5-7B)
0.1
0.25
0.5
0.75
1.0
Aec. (%)
79.40
80.31
79.82
79.88
79.69
Figure shows that our reflection- window decoding signifi- cantly outperforms beam search in Roleplay, STEM, Math, and Reasoning tasks. These tasks require strong logical consistency, and greedy or fixed search strategies often lead to early sub-optimal choices that degrade output quality. Our approach mitigates this by enabling refinement during generation, making generated contents more coherent.
5.4. Further Discussions and Analyses
Efficiency of Reflection- Winddww Decoding In Table 3, we aggregate the regeneration statistics on MMLU with Ilama3.1-8B We record two metrics: (1) the regeneration ratio, which calculates the overall ratio of refined/correeted tokens in the completed response, and (2) the regeneration call, which counts the number of times the pausing criterion is triggered and refinement/corretion is needed before fin- ishing any particular response. We find that regeneration ratio ranges from 3.5% to 5.5% across all categories with a relatively mild pausing criterion. This suggests that the refinements are usually needed during decoding. While beam search always maintains a complete frontier of candi-
Compatibility with Top-k/Top-p Sampling Reflection- window decoding generates tokens autogressively when the pausing criterion is not triggered and only performs selective refinements. The framework compatible with Top-k/Top p sampling except in the refinement/correction mode. In these experiments, we set k = 10, p 0.9, and temperature as 1.0 for both our approach and the baseline Top-k/Top-p sampling. As shown in Table 2, our decoding approach con- sistently outperforms the standard Top-k/Top-p approach across all four models. particular, the average accuracy
8

---

Reflection-indoww Decoding: Text Generation with Selective Refinement
date sequences, our reflection-window decoding approach only activates beam search when necessary, and at the sub- sequence level.
Program. ZT is supported by the National Institute of Justice (NIJ) Graduate Research Fellowship, Award No. 15PND-24-G6--1156-RBBS.
Impact Statement
Entropy Threshold and Window Size df We investigate the impact of threshold a on MMLU Social Sciences with Qwen2.5-7B with fixed window size d = 4. The results in Table 4 demonstrate that our method performs robustly across values ranging from 0.25 to 0.75, with a = 0.25 achieving the highest macro average of 80.31%. While our default setting of a 0.5 is not the best in this specific experiment, it maintains strong performance and shows consistent improvements on most subjects, suggesting that it serves as a reliable default configuration for general usage.
In this paper, we theoretically characterize and empirically address the sub-optimality of the autoregressive decoding for text generation. In particular, we propose selective refinement framework and implement it with a sliding re- flection window mechanism, enabling interchangeable re- finement and generation as the decoding proceeds. Our ap- proach strikes a balance between efficiency and optimality. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
In terms of window size d, we also choose social science subjects using Qwen2.5-7B. The results in Table show that our method maintains strong performance across var- ious window sizes (d = 2 d = 6), with overall macro averages consistently around 79.7%. While d = 6 achieves the highest macro average, d = 4 demonstrates comparable performance and maintains better computational efficiency. These results further support our choice of d = as a robust default setting, offering a good balance between per- formance and efficiency across different models and tasks. We provide additional results MTT-Bech in Table 6 and Table 7, and on MMLU in Table 8 and Table 9 in Appendix.
References
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi. M., Kauffmann, P., al. Phi-4 technical report. arXiv preprint 24Xxx:2412.08900 2024a.
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 1.Xiv:2404.142199 2024b.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077, 2022.
6. Conclusion
In this paper, we theoretically characterize one inherent shortcoming, among others, of the autoregressive decoding for text generation in LLMs. In particular, we show that even when the optimality is defined in terms of the joint probabil- ity over all generated tokens, an oracle LLM can still poten- tially deviate from the globally optimal response of the same length. To mitigate the sub-optimality of the autoregressive way of text generation, we propose an empirical approach guided by our theoretical characterization. We incorporate a sliding reflection window and a pausing criterion so that re- finement and generation can be performed interchangeably. Our experimental results demonstrate that our reflection- window decoding strategy achieves significant improvement over regular decoding strategies in inference-intensive set- tings and maintains performance that is comparable, or even superior to, beam search while being more efficient.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A.. Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877-1901, 2020.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, I.,, and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint a021v:2302.01331,, 2023a.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05/28, 2023b.
DeepSeek-AI. DeepSeek-R1:
Incentivizing reasoning capa- bility in LLMS via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Acknowledgments
We would also like to acknowledge the support from NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R01HL159805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIT Joint
Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 889-898,778888888888888888888818 2018

---

Reflection-Window Decoding: Text Generation with Selective Refinement
Fedorenko, E., Piantadosi, S. T., and Gibson, E. A. Lan- guage is primarily tool for communication rather than thought. Nature, 630(8017):575-586, 2024.
of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L.,, and Lewis, M. Con- trastive decoding: Open-ended text generation as opti- mization. The 61st Anmual Meeting of the Association for Computational Linguistics, 2023.
Ganguli, D., Askell, A., Schiefer, N., Liao, T., Lukošiūté, K., Chen, A., Goldie, A., Mirhoseini, A., Olsson, C., Hernan- dez, D., et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459 2023.
Lin, C.-C., Jaech, A., Li, X., Gormley, R., and Eis- ner, J. Limitations of autoregressive models and their alternatives. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (NAACL-HLT), 2021.
Gemini Team. Gemini: A family of highly capable multi- modal models. arXiv preprint arXiv:2312.1 2tl::2331.//0000 1805, 2023.
Gemma Team, Gemma: Open models based on gemini re- search and technology. arXiv preprini wrXiv:2403.08295 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Mcasuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.60 03300, 2020.
Llama Team. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Madaan. A., Tandon, N., Gupta, P., Hallinan, S., Gao, L.,, Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., al. Self-refine: Iterative refinement with self- feedback. In Advances in Neural Information Processing Systems, volume 36, 2024.
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., and Choi, Y. Learning write with cooperative discriminators. arXiv preprint vrXiv:1805.066 2018.
Holtzman, A., Buys, J., Du, I.., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference Learning Representations, 2020.
MistralAI. Mistral nemo, 2024. URL 1 ai waa.//ww/mmistte -oeoll.nemo....mmm..m
Nocedal, J. and Wright, S. J. Numerical Optimization. Springer, 1999.
Kahneman, D. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.
OpenAL. GPT-4 technical report. arXiv:2303.08774.1 2023.
arXiv preprint
Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. arXiv preprint 1rX1 2023.
OpenAL. GPT-4o system card. arXiv preprint a4X1::2410.21276, 2024.
Kim, S., Mangalam, K., Moon. S., Malik, J., Mahoney, M. W., Gholami, A., and Keutzer, K. Speculative decod- ing with big little decoder. Advances in Neural Informa- tion Processing Systems, 36. 2024.
Pan, I.., Saxon, M., Xu, W., Nathani, D., Wang, X., and Wang, W. Y. Automatically correcting large lan- guage models: Surveying the landscape of diverse self- correction strategies. arXiv preprint 2308::230..03188 2023.
King, S. On writing: A memoir of the craft. Pocket Books, 2000.
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Inter- national Conference on Machine Learning, pp. 19274- 19286. PMLR, 2023.
Pearl, J. Causality. Cambridge University Press, 2009.
Peeperkorn, M., Kouwenhoven, T., Brown, D., and Jor- danous, A. Is temperature the creativity parameter of large language models? In Proceedings of the 15th Inter- national Conference Computational Creativity, 2024.
Lew, A. K., Matheos, G., Zua-Xuan, T., Ghavamizadeh, M., Gothoskar, N., Russell, S., and Mansinghka, V. K. SMCP3: Sequential monte carlo with probabilistic pro- gram proposals. International conference on artificial intelligence and statistics, pp. 7061-7088. PMLR, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. OpenAl Blog, 188)99 2019.
Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., and Wen, J.RR. Elmer: A aooreeutreeresssvve pre-trained language model for efficient and effective text generation. In Proceedings
Serravallo, The writing strategies book: Your every- thing guide to developing skilled writers. Heinemann Portsmouth, NH, 2017.
10

---

Reflection- Decoding: Text Generation with Selective Refinement
Shih, A., Sadigh, D., and Ermon, S. Long horizon temper- ature scaling. In International Conference on Machine Learning, pp. 31422-31434, PMLR, 2023.
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.- Y., He, J., and Xie, M. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36, 2024.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. In T'iirty-eeeenth Conference on Neural Information Processing Systems, 2023.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang. F,, al. Qwen2 technical report. arXiv preprint XXiv:2407.10677 2024a.
Yang, Y., Lee, S., and Tadepalli, P. Language-innone innorage- beam search decoding for multilingual machine transla- tion. arXiv preprint arXiv: 05738, 2024b.
Spirtes, P., Glymour, C., and Scheines, R. Causation, Pre- diction, and Search. Springer New York, 1993.
Sun, Z., Suresh, A. T., Ro, J. H., Beirami, A., Jain, H., and Yu, F. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Töm, A. and Zilinskas, A. Global Optimization, volume 350. Springer, 1989.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judg- ing LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pp. 46595 496953,463, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint MXXv:23077.0988, 2023.
Zhu, T., Zhang, K., Xie, J., and Su, Y. Deductive beam search: Decoding deducible rationale for chain- of-thought reasoning, arXiv preprint 27X1v2240..17666 2024.
Tyen, G., Mansoor, H., Chen, P., Mak, T., and Cärbune, V. Lims cannot find reasoning errors, but correct them! arXiv preprint arXiv:2311.085/1 2023.
Vaswani, A., Shazeer, N., Parmar, N.. Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, I.,, and Polosukhin, I. Atten- tion is all you need. Advances in Neural Information Processing Systems, 2017.
Wei, C.. Koo, K. K., Tavanaci, A., and Bouyarmane, K. Confidence-aware sub-structure beam search (cabs): Mit- igating hallucination in structured data generation with large language models. arXiv preprint arXiv:2406.0M0 00069, 2024.
Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: Decom- position transformers with curo-correlation for long-term series forecasting. Advances in neural information pro- cessing systems, 34:22419-224300 2021.
Xia, H., Ge, T., Wang, P., Chen, S.-Q.. Wei, F,, and Sui, Z. Speculative decoding: Exploiting speculative execu- tion for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909-3925, 2023.
Xia, H., Yang, Z. Dong, Q., Wang, Li, Y., Ge, Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprchensive survey of speculative decoding. arXiv preprint arXiv:2401.07851. 2024.

---

Supplement to "Refleciio-Window Decoding: Text Generation with Selective Refinement"
Table of Contents: Appendix
The Proof of Theoretical Result
13
B Additional Results and Analyses
14
B.1 Analysis on Window Size
14
B.2 Analysis on Threshold
16
B.3 Analysis on Regeneration Ratio
16
B.4 Demonstrative Examples MT-BBecch
17
C Performance Across All MMLU Subjects
21
List of Tables
Comparison macro averaged accuracy on MMLU across subject categories with Phi3-Medium
2 Summary of comparison between Top-k/Top-p and reflection-window decoding on MMLU (multiple models)
3. Regeneration metrics MMLU with Llama3.1-88
4. Acc. by (MMLU Social Sciences, Qwen2.5-7B)
8
5 Acc. by (MMLU Social Sciences, Qwen2.5-7BB)
6 Performance across different window sizes on MT-Bench with Mistral-Nemo
15
7
Performance across different window sizes on MT-Bench with Llama3.
15
8
Accuracy (%) across different window sizes d on MMLU Social Sciences with Qwen2.5-7B
15
9
Accuracy (%) across different entropy thresholds on MMLU Social Sciences with Qwen2.5-7B
16
10
Average regeneration ratio by window sizes d on MMLU college-level subjects with Llama3. I-BB
16
11
Comparison among greedy decoding, beam search, reflection-windor decoding on MMLU with Phi3-Medium
22
12
Comparison between Top-k/Top-p and reflection-window decoding MMLU with Llama3.1-8B
23
13
Comparison between Top-k/Top-p and reflection-windor decoding on MMLU with Mistral-Nemo
24
14
Comparison between Top-k/Top-p and reflection-window decoding on MMLU with Phi3-Medium
25
15
Comparison between Top-k/Top-p and reflection sellectio-wwindow decoding on MMLU with Qwen2.5-14B
26
12

---

Reflection-Widdon Decoding: Text Generation with Selective Refinement
A. The Proof of Theoretical Result
Theorem A.1 (Indication Deviation from the Globally Optimal Length-T Response). Given the prompt X when an oracle LLM (Assumption 3.3) generates a stepwise optimal length-T response which is not the globally optimal response with the same length, let L S T denote the minimum length of prefix-sequence needed in order for such deviation to manifest itself (Assumptions 3.4 and 3.5). Then, the deviation from the globally optimal response happens some step K < L. F Furthermore, the conditional probability when generating the token Wwz E V is strictly smaller than positive number, which itself is strictly smaller than 1, i.e.,
u 1> EL > max 9(x,, = w X1:L-1 = uV (1) (1 XgD), WEV
f(X1LL W ww; 10
Xxo)
(5)
where
S(XI:L-1 Wor (br(L-11 Xgo)
Proof. We first show that the deviation from the globally optimal response happens before step L. Then, we show that the conditional probability when generating the token Wer is bounded away from 1.
By definition of oracle LLM (Assumption 3.3), the advantage of the globally optimal response cannot manifest itself at L= 1 (even if the deviation happens at step 1), i.e., L > 1. Since the minimum length of prefix-sequence needed in order for the deviation of stepwise optimal response from the same-length globally optimal response to manifest is L, then the advantage of the globally optimal response is not manifested until step L. Until step L - 1, in terms of the ground-truth conditional probability following the prompt X so, -equensee the globally optimal response not strictly preferred compared their same-length counterparts of the stepwise optimal response:
f(Xi (vr
Xgo) f(X; "/ynl Xgo),
f(X12 = [e,11
Xgo) 2 f(X1:2 = "V3 Xgo),
(6)
1(X 11 "0
Wey[L-1] Xgo) 2 f(X = ",3[1]W,,11 Wozlt-11 Xgo).
Starting from step and onwards (Assumption 3.5), prefix-sequences of the globally optimal response are strictly preferred compared to their counterparts of the stepwise optimal response:
f(X..1 wv,(1) We Wv [t]
Xgo) f(Xx.L (vz(l)) W. 1lW (t) Xgv),
(7)
f(XIr Worl1 ... "..../..............
X < f(X1. w. ..:. Xgo)-
Assumption 3.4 specifies that for any two same-length but different sequences following the prompt X there is strict ordering between them. Then, in order for the advantage of the globally optimal length-T response to manifest, in terms of strict preferences staring from the length- I prefix- meeix-seeuence (Fquation (7)), there at least one strict preference of the of stepwise optimal response over its globally optimal counterpart before step L. In other words, there is at least one step K [1, 1] such that a strict preference (">" instead of ">") is present in Equation (6):
f(X:k we, Wo ... [e) Xgo) f(XI:K (1) ... Wey (KC
Xgo).
(8)
In order to see why this is the case, consider the opposite scenario where there is no strict preference in Equation (6). Under Assumption 3.4, the comparison between prefix-sequences is either strict preference (they are different) or exactly the same (identical sequences). If there is no strict preference in Equation (6), then for all E [1, L- - 1). W = ww; i.e., the first 1 tokens in the stepwise optimal response are the length- (L - 1) prefix of the globally optimal response. If this is the case, the token generated at step T. has to deviate from the globally optimal response (since the minimum length for
13

---

Snllection-iiddor Decoding: Text Generation with Selective Refinement
the deviation to manifest) 1 # Wvy [2]
f(X11 Werl1] ... Mer X
9(X, [v XI:L-1 = Wvr/1 1.p11-.--011--111
"00l1-1111 Xgo) 1(X "NrD. ... 00-1011111 Xgo)
9Xrr wvr(4 [4]
X1: = [vilt] Wv [ Xg) f(X "v .. ",: Xgo)
(9)
(00)
> 9(x== = wvy [0]
XI:L-1 = - 5o) [(X1L-1 = w -.. Xço)
@ f(XLL = Wvgill -y.. X
where (1) and (iv) follow Assumption 3.3, (di) corresponds to the setting in this opposite scenario, and (iti) follows Definition 3.2 and that (Vv(()] [4] # Wv]](2)- This preference relation Equation (9) contradicts with that in Equation (7), and therefore, Equation (8) has to hold true.
Therefore, when the advantage of the globally optimal response does not manifest itself until step L, the stepwise optimal response deviates from the globally optimal counterpart at some step K < I,, and that under Assumption 3.4, the following strict preference relations hold true:
f(X1.K = wey (1) (1) ... (
Xgo) > f(X1.K = (v3(1) ... W.y (K) Xgo),
(10)
f(X D, ... "V-[L-1] Xgo) f(XI:-1 Wog, M. X
This, together with Equation (7) and Assumption 3.3, indicates that:
9(X, = 1 X1.2-1 = (0,(1 ... 10 1L-11-111 Xgo)
(0 IX.... = 1 ... -,,, [4] X
f(X1LL-1
... -. Xgo)
(40 fX. = -W -,,
...
X
f(X1:t-1 = W8 ... 00,,111...0,,,22-- Xgo)
=
(11)
(00) f(X1.1 W. [1] ... -. Xgo)
f(X1.L-1 = W.
11 X
g(x, 1.v X = = 11 Xgo) S 1,
where (i) and (iv) follow Assumption 3.3, (ii) follows Equation (7), and (iii) follows Equation (10).
Therefore, the conditional probability of generating any wez is strictly smaller than positive number ez, which is further strictly smaller than positive number upper- bounded by 1.
B. Additional Results and Analyses
In this section, we present additional results and further discussions on influences from hyperparameters. We also provide concrete examples demonstrating the process and overall performance of our reflection-window decoding.
B.1. Analysis on Window Size dl
We conduct extensive experiments on MT-Bencch to analyze the impact of window size using both Mistral-Nemo (Table 6) and Llama3. 1-8B (Table 7), with fixed pausing criterion with a = 0.5. These GPT-4o evaluator scores on MT-Bench provide additional evidence that our approach consistently outperforms traditional decoding methods.
For Mistral-Nemo, the optimal performance is achieved d with overall score of 7.93, surpassing both greedy decoding and beam search. For Llama3.1-88 our method consistently outperforms both greedy decoding and beam search across different window sizes, with d = 5 achieving the best overall performance. While d 4 may not always yield the best result, it demonstrates robust performance across both models and serves as reliable default setting.
We further evaluate how different window sizes affect the performance on MMLU social science tasks using Qwen2.5-7B
The results are presented Table
14

---

Sellection-iindow Decoding: Text Generation with Selective Refinement
Table 6: Performance across different window sizes d on MT-Bench with Mistral-Nemo
Decoding Method
Rating
Rating 2
Overall Mean
Reflection Sellettio-Wiiddow (d = 2)
8.38
7.28
7.82
Reflection Winddww
8.44
7.42
7.93
(d
8.28
7.41
7.84
Greedy Decoding
8.38
7.29
7.83
Beam Search
8.32
7.49
7.91
Table 7: Performance across different window sizes d MT-Bench with Llama3.1-8B
Decoding Method
Rating
Rating 2
Overall Mean
Reflection-Winn Window (d 2)
8.29
7.09
7.69
Reflection- Window (d = 3)
8.35
7.51
7.93
Reflection- Window (d = 4)
8.36
7.42
7.89
Reflection- Window (d 5) 5)
8.31
7.62
7.97
Greedy Decoding
8.28
7.49
7.88
Beam Search
8.07
7.19
7.63
Table 8: Accuracy (%) across different window sizes on MMLU Social Sciences with Qwen2.5-7B
Subject
d = 2
d
=
3
d
d=
4
d
=
5
d = 6
Econometrics
62.28
62.28
64.91
64.04
64.91
High School Geography
85.86
84.34
86.36
87.37
85.86
High School Government and Politics
93.26
93.26
92.23
91.19
92.23
High School Macroeconomics
75.90
76.15
75.13
76.15
75.64
High School Microeconomics
83.61
84.03
83.61
82.77
83.19
High School Psychology
87.89
88.07
88.07
88.07
88.26
Human Sexuality
77.86
75.57
78.63
79.39
77.86
Professional Psychology
73.86
73.37
73.20
72.88
73.37
Public Relations
68.18
70.00
70.00
65.45
68.18
Sociology
71.02
73.06
73.47
72.24
73.06
Security Studies
83.08
83.58
83.08
84.08
84.58
US Foreign Policy
86.00
86.00
86.00
86.00
88.00
Macro Average
79.66
79.75
79.82
79.62
79.88
15

---

Sollection-iidoor Decoding: Text Generation with Selective Refinement
B.2. Analysis Threshold
We investigate the impact of threshold on MMLU social science subjects using Qwen2.5-7B with fixed window size d 4. The detailed results are presented in Table 9.
Table Accuracy (%) across different entropy thresholds a on MMLUU Social Sciences with Qwen2.5-7B
Subject
0 =
0.1
0
0.25
0 = 0.5
== 0.75
0 1.0
Econometrics
62.28
64.91
64.91
64.91
64.91
High School Geography
92.23
91.71
92.23
92.23
91.19
High School Govemment and Politics
92.23
91.71
92.23
92.23
91.19
High School Macroeconomics
75.13
76.67
75.13
75.90
75.90
High School Microeconomics
84.45
84.45
83.61
83.19
83.61
High School Psychology
87.52
88.44
88.07
88.26
88.07
Human Sexuality
74.05
77.86
78.63
77.10
77.10
Professional Psychology
73.20
74.35
73.20
73.20
73.04
Public Relations
69.09
70.00
70.00
67.27
67.27
Sociology
84.58
85.07
83.08
84.58
84.58
Security Studies
72.24
71.43
73.47
72.24
72.24
US Foreign Policy
86.00
85.00
85.00
85.00
87.00
Macro Average
79.40
80.31
79.82
79.88
79.69
B.3. Analysis on Regeneration Ratio
To further understand the computational efficiency of our method, we analyze the regeneration ratio under different window size settings. We select six college-level subject categories from the MMLU seience, mathematics, medicine, and physics) for analysis, and conduct experiments with the Llama3.1-8P model with threshold of a 0.5. We consider the window size d as the key hyperparameter, because it directly influences the regeneration ratio, which is calculate by the product of the times criterion get triggered (the regeneration call) and the window size (d), divided by the total length of final response.
(including biology, chemistry, computer
As shown in Table 10 and Figure 6, as the window size increases from 2 to 4, the average regeneration ratio shows a clear downward trend, decreasing from 9.60% to 3. 3.70%. The trend indicates that larger window sizes lead to a faster decrease in the number of modifications needed. Notably, across all settings, the regeneration ratio remains below 15%, suggesting that method maintains comparable computational workload greedy decoding for the majority of the time. These results demonstrate the efficiency of our approach, since it only invoke beam search to find optimal approximations for sub-sequences when necessary, while maintaining the overall efficiency comparable to greedy decoding.
Table 10: Average regeneration ratio by window sizes d on MMLU college-level subjects with Llama3.1-88
Window Size
Average Regeneration Ratio (%)
9.60
6.02
3.70
16

---

Reflection- Window Decoding: Text Generation with Selective Refinement
15.00%
gollege_biologe
college_chemisiee chemistry
caarte___opute____ee science
10.00%
college_mae mathematics
college_medicine
medicine
college__hysiien
5.00%
0.00%
2
Figure 6: Regeneration ratio on MMLU college-level subjects with 1--m
B.4. Demonstrative Examples on MT-Bench
In this subsection, we present three examples selected from MT-Bench to illustrate the generation process our reflection- window decoding. The first example is the 28th test sample from MT-Bench. As is shown in Figure 7, the pausing criterion gets triggered when "part of the car" was generated. The part gets refined and replaced by "specific part, whereas" " which leads more flexible and natural answer. Figure and Figure selected from the 29th and 7th test sample of MT MT-BBench, respectively.
17

---

Sellectio--iiddor Decoding: Text Generation with Selective Refinement
Question
Which word does not belong with the others? tyre, steering wheel, car, engine
Reflection Window
The word that does not belong with the others is "tyre".
The other three words-"steering wheel", "car", and "engine" are all related to vehicle, specifically a car.
A "tyre", on the other hand, component of vehicle, but it's a part the specific part, whereas the other three words are more general terms related the vehicle whole. However, if I had to choose word that stands out not belonging, would say "steering wheel". The other three words-"'car", "engine", and "tyre"-are all nouns that refer to parts aspects of a vehicle, whereas "steering wheel" is a specific component of a vehicle.
But if had to choose one word that the least related to the others, I would say "car". The other three wores-"steering wheel", "engine", and "tyre" all components or parts of vehicle, whereas "car" is the vehicle itself.
So, it really depends how you look at it!
Greedy Decoding
The word that does not belong with the others is "tyre". The other three words-"stcering wheel", "car", and "engine"-arr all related to vehicle, specifically car.
A "tyre", on the other hand, is component of a vehicle, but it's part of the car, rather than the itself.
Beam Search
The word that does not belong with the others is "tyre". The other three words-"steering wheel", "car", and "engine"-are all related to vehicle, specifically a car.
"tyre", on the other hand, component vehicle, but it's not a vehicle itself.
Figure 7: Test Sample #28
18

---

Seflection-Windor Decoding: Text Generation with Selective Refinement
Question
Could you write a captivating short story beginning with the sentence: The old abandoned house at the end of the street held secret that no one had ever discovered.
Reflection Window
Here's a captivating short story for you:
The old abandoned house at the end of the street held secret that no one had ever discovered... (.mittet))
Suddenly, Emily heard a faint whispering in her ear. was a soft, raspy voice that seemed to be speaking an ancient coming from all around her. "Welcome, Emily," said. "I've been waiting for you."
(omitted)
From that day Emily returned to the old house again and again, learning the secrets of the nexus point and the magic that lay within. And she did, she discovered that the house was not just a place it was a doorway to a whole new world of wonder and possibility.
Greedy Decoding
Here's captivating short story for you:
The old abandoned house at the end of the street held secret that no one had ever discovered...
Suddenly, Emily heard a faint whispering in her ear. was a soft, raspy voice that seemed to be speaking an ancient language she couldn't 't quite understand. But as she listened more closely, she began to pick out words words that seemed be calling to her, drawing her deeper into the heart of the house. ... (omitted)
But as she looked up. she saw that she was no longer alone. A figure stood in the doorway, watching
her with eyes that seemed to hold a deep sadness and a hint of longing. And as Emily looked into those eyes, she knew that she had stumbled upon secret that was not just about the house, but about herself a secret that would change her life forever.
Beam Search
Here's captivating short story for you:
The old abandoned house at the end of the street held secret that one had ever discovered. For as long as anyone could remember, the house had stood vacant, its grandeur and beauty slowly being consumed by the passing of time. ... (omitted)
was then that Emily realized that she was not just curious adventurer, but key player in a much larger game. The old abandoned house at the end of the street was not just secret, but a doorway to world of wonder and discovery, and Emily was about to embark journey that would change her life forever.
Figure 8: Test Sample #29
19

---

Snleettion-iidonr Decoding: Text Generation with Selective Refinement
Question
One morning after sunrise, Suresh standing facing pole. The shadow of the pole fell exactly to his right. Can you tell me the direction towards which the shadow pointing-cast, south, west, north?
To which direction was Suresh facing? How do you solve this?
Reflection Window
To determine the direction Suresh was facing, we need to consider the position of the sun and the direction of the shadow.
Since the shadow-of of the pole sun is in the east and its rays cast shadows to the west, and the shadow of the pole fell to Suresh's 's right, we can infer that Suresh was facing south. This is because, when the sun in the east, the shadows fall to the west, and if the shadow falls to the right, it means Suresh was facing south.
Greedy Decoding
To determine the direction in which Suresh was facing, we need to consider the position of the sun and the direction of the shadow.
Since the shadow of the pole fell to Suresh's s right, we can infer that the sun is in the east. This because the sun's rays are coming from the east, and the shadows are cast in the opposite direction. Now, since the shadow fell to Suresh's 's right, we can conclude that Suresh was facing the west. This is because the shadow fell to his right, which means he facing the opposite direction, i.e., the west.
Therefore, the direction in which Suresh was facing is west.
Beam Search
Since Suresh was standing facing a pole and the shadow of the pole fell exactly his right, we can deduce the direction the shadow based on the position of the sun sunrise.
1. At sunrise, the rises in the east.
2. Since Suresh was facing the pole and the shadow of the pole fell to his right, it means the shadow was pointing in direction perpendicular to his line of sight.
3. Given that the sun rises in the east, the shadow of the pole would fall to the right of Suresh's line sight, which means the shadow would be pointing west. Therefore, the direction towards which the shadow of the pole was pointing is west.
Figure 9: Test Sample #7
20

---

Reflection-Windo Decoding: Text Generation with Selective Refinement
C. Performance Across All MMLU Subjects
Due to the size of the table, the material is arranged in the mne-table-per-page manner (starting from the next page).
21

---

Redlection-Window Decoding: Text Generation with Selective Refinement
Table 11: Comparison among greedy decoding, beam search, and reflection-window decoding MMLU with Phi3-Medium
Greedy Decoding
Reflection- Window
Subject
Beam Search
abstract.algebra
58.00
56.00
59.00
anatomy
73.33
74.81
72.59
astronomy
87.50
84.21
88.82
business_ethice
80.00
74.00
79.00
clinical knowledge
84.91
83.40
84.53
college biology
86.81
88.19
88.19
college_chemise
55.00
59.00
58.00
68.00
68.00
70.00
college_mathematic,
62.00
58.00
57.00
Dollege_medieine
78.61
75.14
75.72
college physics
77.45
73.53
80.39
computer security
80.00
75.00
79.00
conseptual.physics
80,85
82.98
82.55
econometrics electrical electrical.engineering Lengineriinggggn..
59.65
57.89
61.40
70.34
68.97
72.41
elementary elementary.mathemaaiss .mathematics
94.44
93,92
93,92
formal
61.11
53.17
61.11
global facts
62.00
64.00
63.00
high_schooll _booogy
88.71
91.29
90.32
high school chemistry
78.33
75.37
77.34
high school computer science
87.00
87.00
88.00
high school european history
80.00
71.52
81.21
88.89
88.89
88,38
high_school wogg..chhoogooeeeeeet.mmmoom
95.34
93.78
94.82
high_schoo aech_school_maace .macrocconomics
84.10
82.56
86.15
high. school mathematics
74.07
75.93
74.07
high_schon microcconomics
89.92
89.92
89.08
high. school _hyyicss
66.89
73.51
69.54
sigh_schhol_pps psychology
91.56
91.01
91.74
seghoschhol.se schoolssttisstiicc..........
78.24
77.78
78.70
high. schosschol.uuc us. history
85.78
77.94
85,78
seahsscho_wwrrll wist
85.23
75.95
83.12
human
74.44
73.54
71.75
numan_scxualee
83.21
81.68
79.39
international law
85.12
85.95
81.82
jurisprudence
85.19
84.26
85.19
logical fallacies
84,66
85,89
85,89
machine
65.18
66.96
66.07
management
85.44
82.52
83,50
marketing
89.32
87.18
88.46
medical.genetics
86.00
90.00
87.00
miscellancous
91.57
91.95
91.19
moral disputes
78.03
75.72
78.32
moral scenarios
74.53
75.53
73.85
nutrition
82.68
81.70
83.01
philosophy
75,88
76,21
77.17
prehistory
84.57
84.88
83.95
professional accouuttinn
75.89
74.11
75.18
professional
56,98
50.72
57.43
professional medicine
77.94
63.60
79.04
professional psychology
80.07
78.27
79.08
public relations
68.18
67.27
66.36
-ttuuiysstudies
73.06
73.47
73.47
sociology
87.06
85,57
86.07
us.forcign_policy
85.00
85,00
84.00
virology
52.41
53,61
53,01
werld.relsgions
84.21
83.63
86.55
22

---

Reflection-Windee Decoding: Text Generation with Selective Refinement
Table 12: Comparison between Top-k/Top-p and reflection-windor decoding on MMLU with Llama3.1-8
Subject
Top-k/Top-p
Sellection-iindwn
abstract.alge algebra
43.00
50.00
anatomy
70.37
68.15
astronomy
71.05
75.00
bebiness_ethice
69.00
71.00
clinical knowledgee
75.09
74.72
college_biologg
80,56
81,9
bollege_chemistry
54.00
50.00
college computer science
58.00
47.00
college mathematics
38.00
45.00
college medicine
68.79
70.52
college.physics
49.02
65.69
computer_securree
69.00
73.00
amncptuul_physiee
64.68
66.81
econometrics
47.37
48.25
electrical eeeerical_eggineering
57.24
60.00
elementary mathematics
88.89
85.71
formal_logic
53.17
46.03
global.fact
48.00
41.00
sugh_school.be bioooo.boooooy
78.71
79.03
high_schoo-cchemie science
61.08
62.07
71.00
72.00
high_sshhol__uuuppnnhhiiie _history high school geography
72.73
73.94
78.79
80.81
high school government politics
84.97
83.42
high school macroeconomics
68.97
73.33
highhscchoommaatenatiiss high_school _mathematics
60.74
62.59
highhscchoommmicroocoooooooo .icrroeconomics
70.17
69.33
high_schoo -physicss
53.64
53.64
85,50
85,50
63.89
65.74
high nigh_school__ jus history
77.94
78.92
high gighsscoollwoold _iisoryy
75.11
79.32
human .giig
69.51
64.13
human sexuality
53.44
54,96
international
78.51
74.38
jurisprudence
75.15
76,85
oggicalfflllacics
74.85
79.14
machine_ Jearning
49.11
53.57
management
80.58
82,52
marketing
86.32
85.47
medical genetics
75.00
76.00
miscellaneous
86.21
86,21
moral.dispun
63.29
59.25
moral moral_ scenarios
44,47
49.72
nutrition
72.22
72.88
philosophy
69.13
68,81
prehistory
71.30
70.06
professional accounting
47.87
53.19
professional law
47.72
48.24
professional medicine
76.47
77.94
68.
68,30
public rellatirss
69.09
63.64
sccurity_studics _studies
64.49
61.22
sociology
77.61
78.61
81.00
81.00
virology
48.80
51.20
religioss
78.36
78.36
23

---

Reflection-Windee Decoding: Text Generation with Selective Refinement
Table 13: Comparison between Top-k/Top-p and reflection-window decoding on MMLU with Mistral-Nemo
Subject
Top-k/Top-p
Sellection-Windene
abstract.alge algebra
34.00
41.00
anatomy
60.74
59.26
astronomy
67.76
66.45
besiness_ethice
61.00
56.00
clinical knowledgee
69.81
70,94
74.31
77.08
bollege_chemistry
41.00
48.00
college computer science
56.00
54,00
college mathematics
45.00
39.00
college medicine
58.96
67.63
college.physics
51.96
52.94
computer_securree
64.00
69.00
anncptuul_physiee
63.40
61.70
econometrics
51.75
$7.89
electrical eeeerical_eggineerinng
55.17
53.10
elementary mathematics
73.81
74.87
formal_logi
42.06
42.86
global.fact
45.00
41.00
sugh_school.bee bioooo.boooogy
76.77
75,16
gegh_schoo..hemiste
51.72
57.14
hog science
70,00
73.00
high_sshhol__uuuppnnhhiiie
66.67
66.06
high school geography
75.76
70.20
high school government politics
80.83
84,46
high school macroeconomics
66.92
69.49
highhscchoommaatenatiiss high_school _mathematics
$9.63
60.00
highhscchoommmicrooccooooooo .icrroeconomics
65.97
69.33
43.71
45.70
78.72 78.72
77.80
58.80
62.50
high nigh_school__ uus history
64.71
73.04
high ggg_sccoollwoold history
70.46
73.42
human .giin
61.43
58.30
human sexuality
61.83
64.89
international
68.60
66.94
jurisprudence
67.59
64.81
oggicalfalllacies
69.94
65.64
machine_leane Jearning
49.11
51.79
management
69.90
69.90
marketing
76.50
71.79
medical genetics
77.00
72.00
miscellaneous
79.69
79.57
moral.dispun
58.38
60.12 12
moral moral_ scenarios
26.15 15
27.49 49
nutrition
67.32
65,03
philosophy
61.41
67.85
prehistory
61.11
61.11
professional accounting
46.10
47.16
professional law
44.39
44,46
professional medicine
58.46
58,46
61.93
65.69
public rellatioss
65.45
59.09
sccurity_studics _studies
52.65
55.92
sociology
68.66
72.64
75.00
67.00
virology
37.95
42.17
religioss
71.35
76.02
24

---

Soliection-iidooe Decoding: Text Generation with Selective Refinement
Table 14: Comparison between Top-k/Top-p and ietecctoonwindoov decoding on MMLU with Phi3-Medium
Subject
Top-k/Top-p
Reflection- Window
abstract.alge algebra
$2.00
62.00
anatomy
70.37
72.59
astronomy
82.89
82.24
bebiness_ethice
72.00
77.00
clinical knowledgee
78.87
79.25
college_biologg
82.64
81,25
bollege_chemistry
51.00
54,00
college computer science
61.00
63.00
college mathematics
50.00
48.00
college medicine
69.
72.25
college.physics
71.57
63.73
computer_securree
74.00
74.00
amncptuul_physiee
75.74 74
74,47
cconometrics
57.89
59.65
electrical eeeerical_engineering
60.00
63.45
elementary mathematics
85.98
85.71
formal_logi
56.35
57.14
global.fact
45.00
46.00
sugh_school.bee bioooo.boooooy
84.52
86,45
gegh_schoo..hemiste
74.38
73.40
hog -ccneee science
81,00
83.00
high_sshhol__cuuppnnnhiiie _history high school geography
73.33
75.76
81.82
82.83
high school government politics
88.60
91.19
high school macroeconomics
80.00
79.74
highhscchoommaatenatiiss high_school _mathematics
60.37
63.70
highhscchoommmicrooccooooooo .icrrocconomics
84.03
87.39
58.28
70.20
85,50
85,5
70.83
72.69
high nigh_school__ jus history
82.84
78.92
high gighsscoollwoold _iitoryy
81.01
81.86
human .giin
71.30
68.16
human sexuality
73.28
77.10
international
77.69
83,47
jurisprudence
83.33
75,00
oggicalfalllacies
79. 14
82.82
machine_leane Jearning
54.46
64.29
management
76.70
80.58
marketing
83.76
81.62
medical genetics
85.00
87,00
miscellaneous
85.31
84.55
moral.dispun
71.97
67, 67.92 .2
moral moral_ scenarios
61.34
63. 24
nutrition
75.82
78,10
philosophy
74.92
71.06
prehistory
76.85
78.09
professional accounting
68.09
66.67
professional law
53.13
50.00
professional medicine
69.12
67.65
76.31
76,96
public rellatioss
64.55
69.09
sccurity_studics _studies
71.43
66.12
sociology
83.08
84.58
81.00
84,00
virology
49.40
49.40
religioss
77.78
74.85
25

---

Reflection-Windiee Decoding: Text Generation with Selective Refinement
Table 15: Comparison between Top-k/Top-p and reflection-window decoding on MMLU with Qwen2.5-14B
Subject
Top-k/Top-p
Reflection- Windoww
abstract.alge algebra
78.00
81.00
anatomy
71.85
76,30
astronomy
88.16
91.45
bebiness_ethice
76.00
75.00
clinical knowledge
81.51
82.64
90.97
86.
bollege_chemistry
64.00
60.00
college computer science
73.00
73.00
college mathematics
72.00
80.95
college medicine
73.41
90.91
college.physics
82,35
84.31
computer_securree
82,00
84.00
anncptuul_physiee
82.55
84.68
econometrics
61.40
65.79
electrical eeeerical_engineering
73.10
73.79
elementary mathematics
95.24
94,97
formal_logic
67.46
61.11
global.fact
$1.00
$2.00
sugh_school.bee bioooobooooooy
88.39
90.97
high_schoo_cchmmie science
80,30
75.37
91,00
89.00
high_sshhol__uuuppnnhhiiie _history high school geography
76.97
81.82
89.39
88.89
high school government politics
91.19
92.75
high school macroeconomics
85.64
86.41
highhscchoommaatenatiiss high_school _mathematics
84.44
84,27
highhscchoommmicrooccooooooo .icrroeconomics
89.08
88,60
82.78
79.47
90.09
91.38
83.33
81 81,48
high nigh_school__ jus history
87.25
88.73
ggg_sccoollwoold _iitoryy
88.61
86.92
human .giin
73.54
73.99
human sexuality
74.05
81.68
international
80.17
77.69
jurisprudence
78.70
84.26
oogicllffallacca
83.44
85,28
machine_leane Jearning
72.32
74.11
management
82.52
85.44
marketing
88.46
90.60
medical genetics
89.00
91.00
miscellaneous
89.40
91.42
moral.dispun
71.97
72.54
moral moral_ scenarios
69.61
68. 49
nutrition
82.35
81.70
philosophy
78.78
80.06
prehistory
86.11
83.02
professional accounting
73.76
71.99
professional law
54.04
56.00
professional medicine
81.62
86.40
77.94
79.08
public rellatioss
71.82
71.82
sccurity_studics _studies
71.43
73.06
sociology
85.07
88.06
87.00
84,00
virology
50.00
51.81
religioss
84.21
84,80
26


Reviews:
[{"review_id": "1y3a73nZvC", "reviewer": "Reviewer_KVvm", "summary": "The authors describe, theoretically and empirically, how greedy sampling is suboptimal for generating the sentence with maximum likelihood.\nThey then propose an alternative algorithm, which pauses the generation when a specific criterion is triggered, and regenerates a small portion of the text.\nThe proposed method is evaluated on several models and datasets, which shows its superiority over standard decoding strategies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer KVvm\n\nWe are very grateful for the insightful questions and constructive comments! Below please see our point-by-point response:\n\n---\n\n### **Q1:** \"the criterion requires the entropy of all past $d$ tokens to be above the threshold [..., but] would not trigger if the LLM is very uncertain about only one token. Is that a desirable behavior?\"\n\n**A1:** Thanks for carefully considering the pausing criterion. Yes, this is indeed a desirable behavior. Uncertainty can occur when the model does not know how to proceed (due to a previous error) or when there are multiple valid ways to proceed. Therefore, we aim to capture _the trend of_ increasing uncertainty, reducing the false-positive triggerings while maintaining a low computational overhead.\n\n---\n\n### **C2:** \"It may be interesting to evaluate and compare difference choices for this pausing criterion.\"\n\n**A2:** We totally agree, and this is exactly why we present Table 4 (varying entropy threshold $\\sigma$) and Table 5 (varying window size $d$) in Section 5.4, and provide additional discussions and analyses (due to space limit) in Appendix B.1 - B.3, on window size, entropy threshold, and modification rate, respectively.\n\n---\n\n### **Q3:** \"How come the proposed algorithm performs better than beam search, when it is supposed to be a cheaper approximation?\"\n\n**A3:** Thanks for the thoughtful question. If there is no limit on the computation and storage, unconstrained-beam-width beam search will yield the globally optimal output through brute force. However in practice, since maintaining a full frontier quickly becomes intractable and a fixed beam width is often introduced as a hyperparameter.\n\nOur algorithm can perform better since it tackles errors as the generation goes on, while vanilla beam search needs a larger beam width to be able to enclose all possible sequences that can be generated by our approach.\n\n---\n\n### **C4:** \"Assumption 3.3 about the LLM being an oracle (i.e. computes the exact conditional probabilities) seems farfetched\"\n\n**A4:** Thanks for carefully thinking about our theoretical results. There might be potential misunderstandings, please allow us to clarify in twofold:\n\n(1) Together with Assumption 3.3, our theoretical results indicate that, **even if** with an oracle LLM, there is still suboptimality in the purely autogressive way of decoding. In other words, even if the LLM itself perfectly decomposes the (conditional) probabilities (which, as you pointed out, is a farfetched benefit to assume in practice), there is still no guarantee in obtaining the globally optimal sequence with purely autoregressive decoding.\n\n(2) The purpose of Assumption 3.3 is to facilitate clear theoretical results, and our empirical evaluation does not rely on or employ this assumption.\n\nIn light of your comment, we have included above clarifications in our revised draft.\n\n---\n\n### **C5:** \"Overall the authors evaluate their method on a few different models and datasets. It would still be more convincing to see more diverse evaluation.\"\n\n**A5:** Thanks for the comment. In our empirical evaluations:\n\n- for **models**, we utilize models from different families, including Llama-3.1-8B-Instruct, Phi-3-Medium-128K-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-7B-Instruct, Mistral-Nemo-Instruct-2407;\n- for **benchmarks**, we consider MMLU (which include 57 diverse subjects, e.g., humanities, STEM, and social sciences, at varying difficulty levels) for evaluating reasoning performance and factual knowledge, and also MT-Bench for a fine-grained evaluation through multi-turn conversational task, including correctness, coherence, and fluency.\n\nWe provide pointers to our empirical results in List of Tables in Page 11. Please just kindly let us know if you have a specific evaluation task in mind.\n\n---\n\n### **C6:** \"Since it seems the end algorithm is a faster approximation of beam search, it would be great to measure the actual speedup.\"\n\n**A6:** We totally agree, and that's why in Section 5.4 paragraph \"Efficiency of Reflection-Window Decoding\" we provide regeneration metrics by MMLU categories (humanities, STEM, social sciences, others), and also present additional analysis in Appendix B.3.\n\n---\n\n### **Q7:** \"What model is used in the synthetic setting? The setup for these experiments needs to be more detailed.\"\n\n**A7:** Thanks for asking about the detail of our synthetic setting.\n\nWe use Llama-3.1-8B-Instruct. For each prompt, together with a certain length $\\{0, 20, 50, 200\\}$ of generation history ($0$ means only the prompt is given), we evaluate whether the joint probability of the sequence generated with greedy decoding is greater than or equal to that produced by (fixed-beam-width, set to $10$) beam search (as proxy of the global optimum). This comparison indicates the extent to which greedy decoding deviates from the globally optimal response.\n\nIn light of your comment, we have included above detail in the revised draft.", "claims_and_evidence": "The main claims of the paper are regarding the suboptimality of greedy decoding. The authors do a good job are convincing the reader that this is indeed the case.\nThe necessity to change the generation algorithm in order to generate a globally optimal sentence seems natural.", "methods_and_evaluation_criteria": "The main algorithm is interesting and flexible.\nThe choice of entropy as a pausing criterion is well motivated. However the criterion requires the entropy of all past $d$ tokens to be above the threshold, which seems less natural to me. The criterion would not trigger if the LLM is very uncertain about only one token. Is that a desirable behavior? It may be interesting to evaluate and compare difference choices for this pausing criterion.\n\nHow come the proposed algorithm performs better than beam search, when it is supposed to be a cheaper approximation? It seems there are some things going on that are not aligned with the theoretical explanation and motivations.", "theoretical_claims": "The paper starts with a theoretical analysis to characterize the behavior of greedy decoding compared. The claims seem mathematically founded and are quite intuitive.\nHowever, assumption 3.3 about the LLM being an oracle (i.e. computes the exact conditional probabilities) seems farfetched. The paper could benefit from more discussion about this, for instance by comparing results for different sizes of models (smaller models are even less likely to be oracles). An indicator that this assumption may be wrong is when we notice that beam seems to perform noticeably worse than greedy decoding.", "experimental_designs_or_analyses": "Overall the authors evaluate their method on a few different models and datasets. It would still be more convincing to see more diverse evaluation, especially when the results seem to be noisy, with sometimes marginal gains.", "relation_to_broader_scientific_literature": "The paper addresses the important problem of the suboptimality of greedy decoding. It describes well the context, the motivation, and the related literature.", "essential_references_not_discussed": "None to the best of my knowledge.", "other_strengths_and_weaknesses": "The writing is not always very clear.", "other_comments_or_suggestions": "Since it seems the end algorithm is a faster approximation of beam search, it would be great to measure the actual speedup.", "questions_for_authors": "What model is used in the synthetic setting? The setup for these experiments needs to be more detailed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Uip3pnsCF", "reviewer": "Reviewer_Lq2Q", "summary": "The paper makes the observation that, given an autoregressive language model $p_{LM}$, the token sequences $\\mathbf{\\hat{x}}$ generated via greedy decoding does not always correspond to the MAP state $\\mathbf{x}^* = argmax_{\\mathbf{x}} p\\_{LM}(\\mathbf{x})$. Theoretical analysis shows that under mild assumptions, at step $L < T$ of the autoregressive generation, if $p\\_{LM}(\\mathbf{\\hat{x}}\\_{\\leq L})$ falls below $p\\_{LM} ({\\mathbf{x}^*}\\_{\\leq L})$,  then (1) the next-token probability $p\\_{LM}(\\mathbf{\\hat{x}}\\_L | \\mathbf{\\hat{x}}\\_{< L})$ falls under some threshold and (2) there exists some $K < L$ s.t. $\\mathbf{\\hat{x}}\\_{\\leq K}$ differ from $\\mathbf{x}^*\\_{\\leq K}$ (if I interpret Theorem 3.6 correctly). To overcome this sub-optimality of greedy decoding, as an alternative to beam search, the authors propose an approach where: (1) we can pause autoregressive generation depending on certain criteria (e.g. the entropy of the next-token distribution as suggested by Theorem 3.6) and (2) then regenerate the last $d$ tokens via, e.g. beam search, to maximize the probability. Empirical evaluations on benchmarks such as MMLU and MT-bench show that the proposed approach consistently outperforms greedy decoding and beam search, as well as top-p and top-k sampling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Lq2Q\n\nThanks for the thoughtful and detailed comments, as well as the time and effort devoted! Below please see our responses to specific comments and questions:\n\n---\n\n### **Q1:** \"[Theorem 3.6] what does the ratio $\\epsilon_L$ mean or how does it correlate with the discrepancy between the sequence obtained via greedy decoding and the MAP state? why we want to regenerate the last $d$ tokens when $\\epsilon_L$ is 'small'\"\n\n**A1:** Thanks for asking about the $\\epsilon_L$ term in our theoretical result. The denominator is the ground-truth joint probability of the length-$(L-1)$ stepwise-optimal response (obtained by greedy decoding), and the numerator is that of the length-$L$ globally-optimal response (obtained by MAP). Theorem 3.6 roughly states that if the model is very uncertain when trying to generate the $L$-th token, then there is an error in the generation history at the $K$-th token, and that $K < L$. Therefore, we need to get back to the token $K$ to start the revision.\n\nIn practice, we look back $d$ tokens and regenerate them. We also provide additional discussions on the choice of $d$ in Section 5.4 and Appendix B.1.\n\n---\n\n### **Q2:** \"Is it possible to analyze when the proposed approach is guaranteed to generate sequences with probability higher than beam search?\"\n\n**A2:** Thanks for the thoughtful question and for trying to go further. If there is no limit on computation and storage, unconstrained-beam-width beam search could yield the globally optimal output (through actual MAP), i.e., guaranteed to be of the highest probability and to outperform (or at least no worse than) any other approach (including the proposed approach).\n\nWith a fixed-beam-width beam search, the theoretical characterization of the generated length-$L$ sequence is highly nontrivial, since the frontier depends on the pruning at all previous steps. Please feel free to let us know if you would like suggest a way to perform such theoretical analysis.\n\n---\n\n### **C3:** \"Table 1 [...] the sequences generated from beam search should always have higher probability than that from greedy decoding (correct [the reviewer] if [they were] wrong)\"\n\n**A3:** Thanks for carefully considering our results. The metric in Table 1 is accuracy instead of joint probability.\n\nIn benchmark evaluations, it is difficult to precisely control the output length of different decoding methods. Directly setting a hard cutoff of token numbers may yield incomplete/insensible responses. Therefore, in Table 1 we present accuracies on MMLU (consistent with previous works).\n\n---\n\n### **C4:** \"Discuss the relation with Shih et al. (2023)\"\n\n**A4:** Thanks for providing the pointer to a related work!\n\nShih et al. (2023) proposed _Long Horizon Temperature Scaling_ (LHTS), which samples from temperature-scaled joint distributions, to address the myopic temperature scaling in autoregressive models. LHTS optimizes for the long horizon likelihood of samples, and can enable a model to generate with a controllable long horizon temperature parameter through finetuning. In comparison, our work aims to address the pitfall of the purely autoregressive generation itself, and our approach is versatile and compatible with LHTS (Shih et al., 2023).\n\nIn light of your comment, we will incorporate the above discussion in the revised paper. Thanks again for providing the pointer.\n\n---\n\n### **C5:** \"Sec. 3 the notations used are overly complicated. For example, [the reviewer] don't see why $\\mathbf{v}$ and $T$ need to be carried everywhere [..., the reviewer] don't see why something like $\\widehat{\\mathbf{x}}$ and $\\mathbf{x}^*$ are not sufficient\"\n\n**A5:** Thanks for the comment on notation. When presenting the theoretical result, we explicitly keep length indices since lengths play an important role when evaluating the joint probability of a response. The longer the length, the lower the probability tends to be, and this is the case for both $\\widehat{\\mathbf{x}}$'s and $\\mathbf{x}^*$'s.\n\nFor instance (we provided this example in lines 165--177),  if we were to use 10 words to distinguish between joint and conditional densities, one might say \"joint density combines all variables; conditional adjusts for known variables.\" However, if we can use 15 words, one might say \"joint density reflects combined probabilities of all variables; conditional density adjusts probabilities given known variables.\" A fair comparison between $\\widehat{\\mathbf{x}}$ and $\\mathbf{x}^*$ should be length specific. Therefore, we think $\\widehat{\\mathbf{x}}$ and $\\mathbf{x}^*$ together with length indices help make this subtlety more transparent.\n\n---\n\n### Reference\n\nShih, Andy, Dorsa Sadigh, and Stefano Ermon. \"Long horizon temperature scaling.\" _International Conference on Machine Learning_. PMLR, 2023.", "claims_and_evidence": "Theorem 3.6 needs further clarification. In particular, the probabilistic semantics of the threshold $\\epsilon\\_{L}$ is unclear, i.e., what does the ratio mean or how does it correlate with the discrepancy between the sequence obtained via greedy decoding and the MAP state? Without such explanation, it's kind of hard to interpret why we want to regenerate the last $d$ tokens when $\\epsilon\\_{L}$ is \"small\". Further, is it possible to analyze when the proposed approach is guaranteed to generate sequences with probability higher than beam search?", "methods_and_evaluation_criteria": "Though I agree that the ultimate goal of new decoding algorithms is to improve model performance on downstream applications, to support the argument made by this paper, it would be more helpful if the authors can also directly compare the probability of the sequences generated by beam search, greedy decoding and the proposed approach. For example, Table 1 suggests that beam search performs worse than greedy decoding, which is kind of counter-intuitive as the sequences generated from beam search should always have higher probability than that from greedy decoding (correct me if I'm wrong). This is probably suggesting that higher probability does not always imply better accuracy/fluency etc. However, this is not a huge problem: as long as the proposed approach can effectively boost the probability of the generations, the main argument made by this paper should already be well-supported.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The contributions of this paper is very relevant to language generation in general, the proposed approach can be viewed as a generalization of beam search, which has been commonly used for LLM decoding.", "essential_references_not_discussed": "Perhaps discuss the relationship between this work and *Shih, Andy, Dorsa Sadigh, and Stefano Ermon. \"Long horizon temperature scaling.\" International Conference on Machine Learning. PMLR, 2023.*", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "The statement of Theorem 3.6, as well as the whole Sec. 3, should be improved. The notations used are overly complicated. For example, I don't see why $\\mathbf{v}$ and $T$ need to be carried everywhere throughout the section: $T$ can be assumed to be some constant and omitted and $v$ seem to serve as the same functionality as $w$ and $X_t$. More specifically, Defn 3.1 is really just defining the sequence obtained from greedy decoding and Defn 3.2 is just defining the sequence that maximizes the joint probability. I don't see why something like $\\mathbf{\\hat{x}}$ and $\\mathbf{x}^*$ are not sufficient.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "psqHiOr2RI", "reviewer": "Reviewer_Fghj", "summary": "This proposes Reflection-Window decoding, an attempt at addressing the limitations of autoregressive text generation in large language models (LLMs), which lack built-in mechanisms for refining or correcting generated content. The authors analyze how sequential token-by-token optimization can deviate from a globally optimal response and propose an alternative approach called Reflection-Window Decoding. This method introduces a sliding reflection window and a pausing criterion, enabling selective refinement of text as it is generated. By balancing efficiency and optimality, the proposed framework improves text generation quality, outperforming traditional autoregressive decoding while maintaining efficiency comparable to beam search. Extensive empirical evaluations validate the effectiveness of this approach in mitigating suboptimalities inherent in existing decoding strategies.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Fghj\n\nWe are very grateful for the thoughtful comments, as well as the time and effort devoted! Below please see our point-by-point responses:\n\n---\n\n### **C1:** \"There are a few more recent works/commercial practices (such as OpenAI O1/3 and DeepSeek R1) that uses a combination of reflection-aware chain-of-thought SFT and RL to achieve a similar computational goal of the proposed method.\"\n\n**A1:** Thanks for the comment. We completely agree that recent works/commercial practices have explored various methods to enhance generated content (Sections 1 and 2). However, the fundamental limitation of autoregressive decoding itself remains under-explored. This gap represents a distinct perspective, different from high-level model behaviors or inference efficiency.\n\nFurthermore, as you kindly pointed out, our method is versatile and provides practitioners additional flexibilities to incorporate different strategies (without the need to retrain or finetune).\n\n---\n\n### **C2:** \"[The paper] provides an interesting point of view for the community to rethink between the choice of 'computation through tokens' and 'computation through logits.' In this particular case, it is 'reflection as tokens' and 'reflection as logits' (change in values). This might reveal a deeper, unified story about how reflection in LLM works that could facilitate future research.\"\n\n**A2:** Thanks for sharing the insight and the perspective! In light of your comment, we would very much like to include such discussion in the paper, and we sincerely hope that our work can facilitate future research. It would be greatly appreciated if you can kindly share pointers to where these terms were discussed.\n\n---\n\n### **C3:** \"[The reviewer] wonder if the authors could combine this with contrastive methods (e.g. before computation of saliency first baselining the likelihood by a smaller proxy model) to achieve more compelling results.\"\n\n**A3:** Thanks for the thoughtful question. If we understood the comment correctly, by \"contrastive methods\" we are discussing the compatibility of our framework with contrastive decoding (CD), e.g., Li et al. (2023), O'Brien and Lewis (2023). CD utilizes a search-based decoding approach that contrasts LMs of different scales, for instance, an expert (larger LM) and an amateur (smaller LM). In principle, our approach is versatile and compatible with contrastive methods, and can be applied in various ways.\n\nFor instance, one can apply our reflection-window decoding on expert and amateur LLMs in parallel, and then continue with CD's method of factoring out undesired behaviors of smaller LMs while retaining good behaviors of larger ones. Alternatively, one can also apply CD first, and then design appropriate pausing criterion to incorporate our selective refinement framework. We leave these as interesting directions for future works.\n\nPlease kindly let us know if we accidentally misunderstood your comment.\n\n---\n\n### **Q4:** \"What's your understanding about the scalability of your proposed method, i.e. to what extent, it could enable the models to do complex tasks that it couldn't without the approach?\"\n\n**A4:** Thanks for the thoughtful question and for trying to go further. As the primary goal of this paper is to address the pitfall of the purely autoregressive way of decoding, we do not claim that our approach can enable high-level behaviors (e.g., complex new tasks) that are otherwise unattainable. However, as you kindly pointed out in **C2**, we sincerely hope our perspective can provide a distinct point of view for community to rethink about related issues, and facilitate future research.\n\n---\n\n### References\n\nLi, Xiang Lisa, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. \"Contrastive Decoding: Open-ended Text Generation as Optimization.\" _The 61st Annual Meeting Of The Association For Computational Linguistics_. 2023.\n\nO'Brien, Sean, and Mike Lewis. \"Contrastive decoding improves reasoning in large language models.\" arXiv preprint arXiv:2309.09117 (2023).", "claims_and_evidence": "- Decoding towards step-wise MAP of the autoregressive model is suboptimal - Non-trivially correct but this is well-known by the community. People still following the current practices just because this is a rather good enough approximation.\n- Saliency-based reflection - This is a rather interesting point, but I don't think its deeper insightly is sufficiently discussed and/or proven.\n- Compatibility and versatility of the proposed method - This is mostly right to me.\n- Improved efficiency with less sacrificing of quality - True but the empirical results kinda weak and the significance is also less consistent given different base models.", "methods_and_evaluation_criteria": "The evalutation is standard and correctly conducted.", "theoretical_claims": "There is but one strictly verified theorem A.1 that is non-trivially correct yet kinda well-known and thus not surprising to the community. The other major theorem 3.6, as is admitted by the authors, provides a bound that is too weak to be practically useful. While the general framework of the proposed method is of great potential and very interesting, the current progress of the algorithm is kind of disappointing and less theoretically supported than it might seem.", "experimental_designs_or_analyses": "The experiment is well formulated and correctly conducted.", "relation_to_broader_scientific_literature": "There are a few more recent works/commercial practices (such as OpenAI O1/3 and DeepSeek R1) that uses a combination of reflection-aware chain-of-thought SFT and RL to achieve a similar computational goal of the proposed method. While I agree with the authors that this proposed method has the merit of minimising the additional overhead in achieving better results, unfortunately (yes, yet another bitter lesson), it is less extensive and scalable than those rather direct methods to introduce the reflection mechanism through non-architectural/algorithmic ways.", "essential_references_not_discussed": "I am not an expert in this domain, to the best of my knowledge, apart from the aforementioned direct approaches, there is not any essential references that I know omitted by the authors.", "other_strengths_and_weaknesses": "This paper, while not very empirically strong, provides an interesting point of view for the community to rethink between the choice of \"computation through tokens\" and \"computation through logits\". In this particular case, it is \"reflection as tokens\" and \"reflection as logits' (change in values)\". This might reveal a deeper, unified story about how reflection in LLM works that could facilitate future research.", "other_comments_or_suggestions": "Using the fluctuation of likelihood to determine potential reflection is an interesting point. I wonder if the authors could combine this with contrastive methods (e.g. before computation of saliency first baselining the likelihood by a smaller proxy model) to achieve more compelling results.", "questions_for_authors": "What's your understanding about the scalability of your proposed method, i.e. to what extent, it could enable the models to do complex tasks that it couldn't without the approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jVfFbWsHRK", "reviewer": "Reviewer_ZxMG", "summary": "This paper introduces a built-in mechanism for refinement and correction during LLM generation. The authors provide theoretical analysis characterizing the sub-optimality of purely autoregressive decoding and propose a \"reflection-window\" decoding approach that allows for selective refinement during generation. The method shows improved empirical results on multiple benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ZxMG\n\nThank reviewer for the comments and questions, as well as the time devoted! Please kindly notice that there might be some **potential misunderstandings**. Below please see our point-by-point responses:\n\n---\n\n### **C1:** \"In Figure 5, the authors compare the win rate between Beam Search and Reflection-Window (Greedy). However, since Reflection-Window (Greedy) primarily relies on greedy decoding with occasional (3.5%–5.5%) use of Beam Search, it is essential to include pure greedy decoding as a baseline for comparison. [The reviewer] strongly encourage the authors to add this comparison.\"\n\n**A1:** Thanks for considering our experimental results. There might be some **potential misinterpretation**.\n\nIn Figure 5, we present the win rate of beam search _against_ greedy decoding, and that of our approach _against_ greedy decoding. In other words, the win/lose is not calculated between beam search and our approach, but compared against greedy decoding, respectively. We think the brief caption (due to space limit) might be the reason behind this misinterpretation, and we have modified it into \"Figure 5: Comparison of win rates of beam search and our reflection-window decoding (both against greedy decoding) on MT-Bench across categories.\"\n\n---\n\n### **C2:** \"In Table 2, the authors compare the performance between Top-K/Top-P and Reflection-Window. The improved performance of Reflection-Window primarily highlights the effectiveness of Beam Search over Top-K/Top-P in this task. [The reviewer] strongly encourage the authors to include a pure Beam Search baseline for comparison.\"\n\n**A2:** Thanks for the comment. There might be some **potential overlook** in our specified setting.\n\nIn Table 2, the \"regular decoding\" in our approach is Top-$k$/Top-$p$, and the difference between our approach and the vanilla Top-$k$/Top-$p$ baseline includes both the **sliding-window reflection** and the **selective refinement** mechanisms. Describing Table 2 as primarily highlighting the effectiveness of beam search might oversimplify the results, potentially overlooking the substantial improvements in computational efficiency and performance introduced by these additional mechanisms.\n\nRegarding the pure beam search baseline, we included the evaluation results in Table 11, and the pointer was provided on line 344 (Right).", "claims_and_evidence": "The potential deviation of auto-regressive generation is supported by theory and demonstrative examples.", "methods_and_evaluation_criteria": "The proposed methods exhibit some deviations from the dependence structure illustrated in Figure 1(a).", "theoretical_claims": "The proofs are sound.", "experimental_designs_or_analyses": "Some experimental designs are not convincing:\n\n1. In Figure 5, the authors compare the win rate between Beam Search and Reflection-Window (Greedy). However, since Reflection-Window (Greedy) primarily relies on greedy decoding with occasional (3.5%–5.5%) use of Beam Search, it is essential to include pure greedy decoding as a baseline for comparison. I strongly encourage the authors to add this comparison.\n\n2. In Table 2, the authors compare the performance between Top-K/Top-P and Reflection-Window. The improved performance of Reflection-Window primarily highlights the effectiveness of Beam Search over Top-K/Top-P in this task. I strongly encourage the authors to include a pure Beam Search baseline for comparison.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Stengths**  \nClear motivations supported by theoretical analysis and demonstrative examples.\n\n**Weaknesses**  \nI am mostly concerned with the experimental parts. Please refer to Experimental Designs Or Analyses. Besides, in Table 1, Reflection-Window (Greedy) shows no improvement over purely greedy decoding.", "other_comments_or_suggestions": "None", "questions_for_authors": "A more rigorous experimental evaluation and positive results would change my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the suboptimality of greedy decoding in autoregressive language models and proposes Reflection-Window Decoding, allowing for selective, uncertainty-triggered revision during generation. Both theory and experiments, spanning multiple models and benchmarks, support the effectiveness of the approach.  
   
**Review Summary:**    
- **Reviewer KVvm** values the theoretical foundation and empirical evaluation, raising some concerns about the pausing criterion and broader evaluation. These were addressed in the rebuttal, and the reviewer supports acceptance.  
- **Reviewer Lq2Q** finds the contribution relevant and mostly well-argued, suggesting clarification for some theoretical results and asking for comparison of log-likelihoods. After rebuttal, the reviewer is satisfied and leans toward acceptance.  
- **Reviewer ZxMG** initially questioned some baseline comparisons but was satisfied by the author’s clarifications and raised their score to weak accept.  
- **Reviewer Fghj** appreciates the originality and potential of the work, though notes the experiments could be stronger. They remain somewhat skeptical but acknowledge the value of the new perspective.  
   
**Author Response:**    
The authors responded thoroughly, clarifying theoretical points, providing additional results, and committing to address remaining suggestions.  
   
**Conclusion:**    
With most reviewers supporting acceptance and author response effectively addressing concerns, I recommend **accept** for this submission.  
   
---
Decision: Accept (poster)