LZ Penalty: An information-theoretic repetition penalty for autoregressive language models.
Authors: Tony A Ginart
Naveen Kodali
Jason Lee
Caiming Xiong
Silvio Savarese
John Emmons
Venue: Accepted by TMLR ()

Abstract: We introduce the Lempel-Ziv (LZ) penalty, a penalty specialized for reducing degenerate repetitions in autoregressive language models without loss of capability. The penalty is based on the codelengths in the LZ77 universal lossless compression algorithm. Through the lens of the prediction-compression duality, decoding with the LZ penalty has the interpretation of sampling from the residual distribution after removing the information that is highly compressible. We demonstrate the LZ penalty enables open-source reasoning models to operate with greedy decoding without loss of capability and without instances of degenerate repetition. In contrast, both the industry-standard frequency penalty and repetition penalty are ineffective, incurring degenerate repetition rates of up to 4% or more.

Published in Transactions on Machine Learning Research (02/2026)
Lempel-Ziv Penalty: An information-theoreti repetition penalty for autoregressive language models
Antonio A. Ginart, Naveen Kodali, Jason Lee, Caiming Xiong, Silvio Savarese, John Emmons Salesforce AI Rescarch
aginart@@alesforcee.. com, jemmona@@aleesoocce.cm
Reviewed on OpenRevicw: http:://ppanooroww.nee//rrurtt/p
Abstract
We introduce the hempel-Ziv (LZ) penalty, a penalty specialized for reducing degenerate repetitions in autoregressive language models without loss of capability. The penalty based on the codelengths in the 1.Z77 universal lossless compression algorithm. Through the lens of the prntiction-comppessitt of sampling from the residual distribution after removing the information that is highly compressible. We demonstrate that the LZ penalty enables open-source reasoning models to operate with greedy decoding without loss of capability and without instances of degenerate repetition. In contrast, the industry-standas frequency penalty and repetition penalty are ineffective, incurring degenerate repetition rates of up to 4% or more.
duality, decoding with the LZ penalty has the interpretation
1 Introduction
There has been advent in rcasoning models (Singh et al. 2025 Yang al. 2025 Guo al. 2025
Reasoning models are class of large, auloregressive foundalion models that achieve impressive capability gains in certain domains by scaling ohain-of-thoughh reasoning sequences at inference time. While reasoning models are a promising approach for scaling inference-time compute, open-source reasoning models currently suffer from some friction points that make their use problematic for downstream application developers due lack of determinism around the reasoning traces. This lack of determinism rooted the fact that reasoning models do not run well at low temperatures because the sampling distribution can mode collapse into degenerate repetitions.
Enabling deterministic algorithms for generation is useful for debugging and may be an explicit requirement for some deployments. Furthermore, even at higher temperatures, even frontier models can still fall into degenerate repetitions in real-world deployments, such as within Curso
There are two industry-standard penalties aimed reducing repetition. First, the repetition penalty Keskar et al. 2019) applies fixed logit penalty that encourages the model to use new tokens. The frequency penalty is more subtle, and applies logit penalty proportional to the token count in context. Neither penalty consistently stops degenerate repetitions without degrading the sample quality. First, the repetition penalty does not actually succeed in preventing degenerate repetitions because it applics a naive, binary modal penalty which does not take into account the number of times a loken has appeared. Furthermore, the repetition penalty is set too high in an effort to minimize this mode collapse, the sampler becomes unable to use fundamental, necessary, but frequent tokens, such as spaces or periods, resulting in poor completions. Thus, merely increasing the repetition penalty is a viable solution either.
See Appendix. www.cursor.com

---

Published in Transactions on Machine Learning Research (02/2026)
On the other hand, the frequency penalty is more adaptive. The logit penalty grows proportionally with the token count in context. However, it still fails, because it produces an (interesting) degenerate eycle effect, where a token repeats until it incurs too high a penalty, at which point the sampler picks new token to repeat. This is an excerpt from such a generation from an AIME question.
Excerpt from QwQ-32B using a frequency penalty of 0.3 and temperature of 0.
Non! Non! Non! Non! Nou! Non! Non! Non! Non! Non! Non! Non! Non! Non! Non! Non! Non! Non! Non! nono! This! This! This! This! This! This! This! Thist This! This! This! This! This! This! This! This! This! This! Thist Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Third! Tenthl Tenthl Teuth! Tenth! Teath! Teuth! Tenth! Teath! Tenth! Tenth! Teath! Tenth! Tenth! Teath! Teuth! Tenth! Tenth! Tenth! teuth! senth! tenth! tenth! tenthl tenthl tenth! tenthl tenth! tenth! tenth! tenth! tenth! tenth! tenth! tenth! tenth! Okay! We! We! We! We! We! We! We! We! We! We! We! We! We! We! We! We! We! We!
The main reason this occurs is because reasoning traces used by reasoning models such as Q=Q can become quite long, but the frequency penalty does not normalize for sequence length account. for it. Therefore, important and common tokens eventually become banned by the penalty, which degrades the complction, and eventually, results in calastrophic degeneration as seen in the excerpt.
The fundamental improvement in the LZ penalty relative to the repetition or frequency penalty is that the LZ penalty, borrowing from the sliding window matching techniques pioncered in the LZ77 (Lempel & Ziv) 1977 and LZSS (Storer & Szymanski] 1982) lossless compression algorithms, depends on the repetition of n-grams over a long but fxxed-length sliding window. By penalizing as a function of eengh-nnoraale n-gram statistics opposed to single token statistics, the penalty can be significantly more surgical in how it modulates the sampling distribution.
While there may be numerous reasonable ways to convert n-gram statistics into serviceable sampling penalties, we opt to base our penalty in the prediction-commprsiion duality principle, which has various formulations, but essentially states that for every autoregressive language model, there is a dual data compression algorithm (and vice-versa). More precisely, the duality states that logits in a language model are equivalent, in various ways that can be formalized, to codelengths in data compressor.
Following the principle, we give a quick gist of the proposed LZ penalty:
1. Simulate universal LZ sliding window compression over the causal Loken sequence to compute the code: C 10
Compute the change in codelength over the alphabet for each next-token: A|C| E RIAI
3. Apply the change in codelengths as a penalty the model's logits (denoted E): e + E+ A|C
Informally speaking, the interpretation of this penalty is that we are extracting the residual information in the language model after removing the information that is easily compressible by the Lempel-Zii Ziv universal lossless data compressor. From an information- theoretic standpoint, autoregressive generation can be viewed as a sequential compression process: the model predicts each next token so as to minimize the expected codelength of the sequence under its learned distribution. In this view, both the language model and the LZ universal compressor quantify how predictable (or equivalently, how redundant) each continuation is. The LZ penalty can thus be interpreted as biasing the model's next-token distribution away from redundancies identifiable by the LZ compressor, encouraging sampling from the residual, less-compressible information.
Limitations While the LZ penalty substantially improves resistance to degenerate repetition, several limitations remain:
Algorithmic complexity. Compared to industry-standard penalties, the LZ penalty introduces additional complexity. Three hyperparameters must be set: two intrinsic to the LZ compressor (window length and buffer length) and the standard penalty strength parameter. However, similar to how the LZ hyperparameters do not need to be tuned in gzip, we find that they do not need to be tuned here.
We refer to this cycle because, in principle, the model would eventually run out of new tokens and be forced circle back a previonsly used token.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Compute overhead. The penalty requires simulating LZ-style compression step at each decoding step to compute codelength differentials for every token in alphabet. While the overhead is minor compared to the full forward pass of a modern large language model, and can be efficiently parallelized on GPU using operations such as PyTorchhs torch. unfold, it is still worth consideration, especially as naive implementations can become burdensome during inference.
Intentional Repetitions Consider the query: Repeat the letter 'a" 100 times. While this is a contrived query, it is situation for which we may observe diminished performance.
Natural Language Assumption The 1.Z. penalty is designed and tested for natural language modeling tasks. The LZ compression algorithm itself requires the mathematical assumptions of stationarity and ergodicity for theoretical purposes, but practically, it works quite well across almost all natural language sequences of sufficient length (at least a few hundred tokens). While variations of the LZ penalty may work for other tasks, such as multimodal, it may require a domain-specific compressor for best empirical performance.
2
Background and Related Works
2.1 Language Modeling and Sampling
Language modeling traces its origins back to Shannon's testament (Shannon 1948 where he trained causal language model by computing the n-gram lrequencies over an English lext corpus. Modern language models are predominantly based on the transformer architecture (Vaswani et al. 2017). Completions from transformer language models are gencrated by autorogressive sampling of the next-token distribution. The development of transformer language models has been accompanied by significant advancements in sampling techniques that govern text generation. Early explorations of language modeling employed top-k sampling (Jozefowicz et al.) (0166) to constrain the output distribution to the k most probable tokens, a technique later refined in (Welleck et al., 2019), which paired it with unlikelihood training to mitigate repetition. Concurrently, temperature sampling (Bowman et al.] 2016) emerged as method to control randomness. Later, nucleus sampling was introduced as a proposed improvement over top-k sampling (Holtzman et al. 2020
These sampling strategies evolved alongside efforts to address text degeneration and repetition. GPT-2's implementation (Radford et al.] (219)) implicitly utilized frequency penaltics to enhance output fluency, and, concurrently, the repetition penalty (Keskar et al.) 2019)) was devised to prevent repetitions and encourage diversity in completions. Later, LaMDA (Thoppiaan et al. 2022) applied repetition penalties to improve dialog coherence, reflecting growing emphasis on balancing creativity and quality in LLM outputs. Together, these contributions and others eventually led to an saduutry-stendarl sampler which supports a temperature, a top-k; a top-p and a frequency or repetition penalty, all of which can be used in tandem to transform the raw next token logits into a final distribution for sampling. These mechanisms are related to the theory of intrinsic motivation, which defines curiosity and creativity as progress in prediction or compression (Schmidhuber 2010
Recent frontier chat models have significantly improved in addressing repetition issues through advancements in training and largely no longer require a repetition penalty even for greedy decoding. Nevertheless, specialized reasoning models contimue to exhibit challenges related to repetitive outputs, particularly during complex inference tasks or extendod reasoning chains. While visibility is limited into closed-source reasoning models, open-source reasoning models such as DeepSeek's R1 and Qwen's QwQ both require high-temperaturr sampling (generally, at least 0.5 to 0.7 is recommended) in order to prevent degenerate repetitions.
Definition 1. Data Sequence. A dala sequence of lokens will generally be denoled by x over some alphabel A.
We will write x; to refer to the i-th token in the sequence, and we will write Ige to denote the head of the sequence (x,,..,, 2z) and Zet to denote (..,)),,,,11) We write xt to denote the slice (#,,.., Dt). We will also write Ty; to denote the tail of a sequence.
Definition 2. Causal Language Model. A causal language model, LM an algorithm that maps sequences to probabilily mass function (pmf) over A.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Definition 3. Cross entropy. Let HL)(Z) denote the average cross-entropy loss of causal language model LM on data sequence x. Recall that cross-entropy is defined:
HLM(D)= = |2||
PLM(( xes)
(1)
where (LM ( is the probability assigned by the model to the i-th token given the preceding context Zei-
We will generally be working in the log-domain, we write (LM to denote the log-probabilities (or logits) and PLM to denote the corresponding pmf generated by the model given the sequence x: PLM softmax(fIM).
2.2
Data Compression
Data compression algorithms go back to the turn of the 20th century. In Shannon's lestament (Shannon he describes the first provably optimal compressor. Later, many entropy-optimal compressors achieved practical computational complexity assuming known data distributions. Later still, in 1977 and 1978, the first universal compressors were launched, LZ77 and LZ78, that could, asymptotically, achieve the entropy-rate of
1948
any stationary, ergodic data source (Lempel & Ziv 1977
1978
Wyner & Ziv 1994 Morita & Kobayashi
1993). Since then, a whole family of LZ-style compressors has emerged Fiala & Greene 1989; Miller &
Wegman 1985
Pavloy
2007 Oberhumer
1997
Yoshizaki
1988
Storer & Szymanski
1982
Welch 1984).
1,777 and 1.7.8 both operate the principle of adaptively building data structures based on previously seen tokens. Imagine scenario in which you want to train your language model from scratch while doing inference. The model updates as each new token arrives, but you also care about the model's average cross-entropy loss over the entire sequence, from start to finish, since you care about the overall compression rate. LZ algorithms are not only theoretically universal in the sense they are provably optimal for stationary ergodic data, but they are practically universal in that they generally work well on real data too, even without any prior statistical assumptions.
In this work, we will only focus on the LZ77 family, which we refer herein the LZ sliding window compression algorithm, which uses string matching from buffer over lookback window. This contrasts the L,788 family, which favors tree-style dictionaries. Sliding windows are more convenient for GPUs (for example, by using PyTorch's (Paszke et al., 2019 unfold operation) whereas tree-based dictionary methods are more inherently sequential.
Even though all T.Z sliding window algorithms work on the same basic principle of computing n-gram repetitions within sliding window, they can vary in how they encode their compressed data and how they manage lookback buffers. Concretely, LZSS (Storer & Szymanski 1982) modifies LZ77 by using 1-bit flag to indicate whether the next chunk of data is literal or length-distance pair and uses literals if a length-distance pair is below a given minimum length. Since we do not actually need to encode or decode the token sequence, the details of the encoding subroutine are not particularly important for our purposes. Instead, we should focus on how many bits are required for the encodings - the codelengths of the resulting codes. We take L7SS as our reference compressor herein, and use the L.ZSS encoding scheme the I.Z. penalty. When we refer a generic LZ sliding window algorithm, we will mean the LZSS variant.
Definition 4. Data Compression Algorithm. data compression algorithm, or data compressor, is an algorithm that injectively maps sequences over an impul alphabet set A to binary codes (0. 1) Definition 5. Single- Token Data Compressor. A single-token data compressor, C: A - {0, 1)* maps literal single-tokens to binary codes. We will assume single-toke token data compressors are complete prefix codes Cover & Thomas 2006).
Single-token data compressors can be iteratively composed to operate over full sequences. Generally speaking, they incur a small additional overhead due being unable to amortize over longer code blocks. Practical data compressors, however, do not encode on a single-token basis. They often operate over blocks of the full sequence.
4

---

Published in Transactions on Machine Learning Research (02/2026)
Definition 6. Compression Rate. The compression rate, [C], for data compressor C over sequence is given by [x) C(a)
LZ Sliding Window Compression Algorithm The state of an LZ sliding window compressor is comprised of a sliding lookback window w and a bufler b. LZ compressors work by encoding length-distance pairs for the buffer with respect to the lookback window. In asymptotic analysis these windows have max sizes which are allowed to grow sub-linearly the length of the data sequence. In real implementations, they are fixed to a constant that long enough to work practically.
Definition 7. We define findLonges tMatch as the following objective over input strings and 2. d, arg max, (maxxsiel { Ugk = 2) 2)tha)
Definition 8. Lempel-Ziv Zio (LZ) Sliding Window Compressor.
Let and be sequences with |w] |b].
Let (D, L) fundLongestMatchhi w) denote the length of the longest match to the buffer and the distance back from the end of the lookback window. Let C'C" denote string-wise concatenation of codes C' and C". Then, the LZ compressor for buffer b and window w is given by:
I(d,1) C(d, ((bbsll)) ((bb))C.2(bs/ll))
if 2 1 and [b]
Cbz(b/w)-
if 2 1 and [ << [b] ifl 0
Proposition 1. [Storer & Szymanski 1982 LZSS can encode match of length L occurring D tokens in the past using [Czz(L, D)] log + log D bits.
On the other hand, if no match is found, we require more bits to encode token literal.
Proposition 2. (Storer & Szymanski, 1982 LZSS requires [Czz(a)] = log [A| + 1 bits to encode token literols A.
Note that the encoding scheme and algorithm state alone do not fully dictate how the LZ data compression algorithm operates in practice over data stream. Def. strictly refers to the code for a buffer sequence given lookback window. In practice, there is some impplementatioonspeeffee basic control logic used to, obviously, slide the window but also flush the buffer when codeblocks are emitted and appended to the compressed sequence. However, for the sake of simplicity, given context of by setting:
always simulate a fully populated buffer and window for
b(x) = x w(x) = 2
(b|b///
(2)
By always simulating maximal buffer size, we abstract away edge effects and the details of the implementatioonsppcdife control logic while focusing on the codelengths.
Finally, it will be helpful to define the marginal compression of context sequence with respect to a next token a.
Definition 9. Marginal Compression: (x) == [C(az)] - [C(z) where ar denotes the concatemation
of a and x.
We write ([C(()) E R [Al to denote a marginal compression vector indexed over the alphabet.
2.3
The IndditionnCCopprssiooo Duality
We review the well-cstabliseed duality between language modeling and data compression. The prediction- compression duality principle has mumerous possible formalizations depending on the treatment of the subject, but for our purposes, most interested in the theme of equivalence between logits in language models
5

---

Published in Transactions on Machine Learning Research (02/2026)
and codelengths in data compressors. We refer the reader to (Delétang et al. 2024 for modern, in-depth treatment of prediction-compprssionn duality.
~ |C|
(3)
We will review such formal treatment of the duality principle. Proposition 3 (Prediction-Conies Compression Duality). Fiz alphabet A and token sequence æ I1... 21....2. Zn E A".
Compressor 3 Language-model::
Let DC a single-token compressor. Define the logits of a dual longuage model as:
(Dc(a xei) = |CDo(#; m<e))l (bits)
by the codelength it assigns to x; conditioned on the history Zex-
Define the causal probability assignment PDC = softmax(4Dc) as usual.
Then the compression rate of DC equals the per-token cross-entropy of the induced language model:
1 n Σ LDd xz xes) = Σ log logpDe(2 xxs) = HDC(z) bits bits/token.
[x) =
Language-model Compressor:
Let LM be any cousal language model that outputs PLAn( res).
Then, the Arithmetic coding construction ((Cover & Thomas 2006 Witten et al. 1987) produces a sequential prefix-free compressor CLM satisfying, for every æ E",
(LM((F) = Σ CLM(a x<e)l S loga PLM(2; xee) +
2 = HLM(2) + O(1/n) bits/token. n
Hence, up to an asymptotically negligible ( redundancy,
[TDol(x) = HDC(*),
(x) (L)
Given a language model, we also have a data compressor that compresses as well as the language model predicts, and given data compressor, we have language model that predicts well that data compressor compress. The Arithmetic code (and other codes such as the Huffman code (Huffman) 1952 Cover &
Thomas 20061), employ the prediction-cmppesss
duality to assign codelengths based on log probabilities.
The situation is more complex for constructing causal language models from online data compressors such as LZ sliding window algorithms. This is because causal language models must be able to generate a valid next-token pmf at every step whereas data compressors often buffer tokens together into a single code. Practically, this means data compressors do not necessarily produce codelength for every next-token. We address this issue by simulating full buffer and lookback window each next-token. Similar idcas have been explored in [Ryabkol 2007
3 LZ Penalty
The core essence of the T.Z. penalty is to use the prediction-comppresion duality to construct compressor's dual language model (in this case, L We can then apply the following logit update to the language model we wish to penalize, for some penalty strength 0 < a:
Refer Fig. for an architecture diagram the L.Z. penalty.

---

Published in Transactions on Machine Learning Research (02/2026)
fLM +- LLM + a& Cr.zl
(4)
where A|C is the marginal compression under a simulated LZ sliding window compressor due to each potential next-token. Note that adding a redundant loken can actually shorten the full codelength under the L. compressor, which results in negative marginal codelength to penalize overly redundant tokens.
Context
LM
LZ
Logits
Marginal Codelengths
Sampler
Penalized Logits
Figure 1: architecture diagram detailing the flow of an autoregressive sampling loop using the penalty.
Let denote the current context. We simulate the LZ sliding window w(x) and buffer b(x) in (2) We can then compule the incremental change in codelength due to each possible next-token E A relative to the simulated buffer and window. We can then compute the simulated marginal compression for LZSS for all a E A::
A[C1z((x) == AlC))( = (CLz(ab(x)) (ab((abb())))) w(x)) - (CI2(b()))w()))
(5)
Since we are operating in the log- log-domain under softmax affine invariance:
A(Czz((a) x (cz)(a
(6)
Going forward, we omit explicit dependence on æ and LZ when it is obvious. Let d, I +- findLonngesMatcch(bb))) w(x)) and +- findlonggsttatcch(ahh. w).
If 0, we know the virtual next-token a comes after literal in the encoding. This implies that:
{
L if a E w(z) and / = 0
X(a)
if & w(x) and
with 8 giving the distance of the match (if present).
If 2 1, then the virtual next token might extend a match. In the case that it does so, then = 1, because the match location shifts one spot to the right and the length increases by one. If does not extend the match, then À <
and
We proceed with case-by-case calculation of (Czzzab]vv w)l. Recall we are working in the log-domain, and that because 1, are independent of a, due to softmax affine invariance, we can ignore terms that only depend on 1, but are constant with respect to the choice a.

---

Published in Transactions on Machine Learning Research (02/2026)
Case I: 0)
((ab/w) C(alw)((blw) [C((b//w) = [C(a/w) [c(b/w)] [c(a|w)]
(7)
Furthermore, as discussed above:
(((, if C((a) x==0
C = C(alw)
(8)
Where C(1,6) 6) encodes a singleton match at distance and C(a) encodes a as literal. This gives us our first case: C1 x [C(a[w)]. Recalling Prop. 1 and and removing constants due to softmax affine invariance, obtain simple expressions:
log log [A]
if
|C1
(9)
if X = 0
Case II: (12)
Cu(ablw) C(abçi W)C((s//w) - = |Ci1 [C(ab:/ + [C(b>:/w) x (C(abg/ww)
(10)
Furthermore:
Cn = C(abei w) [
i
(11)
[((abg/l)) = C(A,))
A
Recall that, discussed above À S if and only if does not extend the match of length 1. If does extend the match, then A = I I+1 1. Reusing g we can further simplify:
c(a)c(I,d) C(1,8)C(1,)) C(2,6)
if X 0 if A 1
Cu
(12)
if A I 1+1 1
Again reusing Prop.
I and 2
[C(a)C(I,)) =(((() + |C(1, d) = log |A| + log +1 if X = 0
1nnl
(C(1,5) (C(I,d) = log(6) + log(
if
(13)
IC(A, log(Ad)
if X 1
It is expedient and permissible (due to affine invariance) to subtract the log(ld) + 1 term.
if A = 0 X 1 log(1 d-1+1 d-141)-1 - if X 1+1
log |A|
CCnl
log(ō) 6)
(14)
where log(1 d-1+1) log(Ao) log(ld) follow from X = =++ 1 and 8 d 1

---

Published in Transactions on Machine Learning Research (02/2026)
LZ Penalty Formula
Combining cases I and II above yields complete formula for the LZ penalty
adjustment:
log |A| log(o)
if if À if A = 1+11
A|C
(15)
log(1 1-dt
Assuming |A| > w/, then this provides dynamic range of [log(2/[bl) lo [Al]. For an alphabet of size 128k, a lookback window of size 512, and buffer of size 32, using binary logarithms, this yields an adjustment range from -5 to +17, with -5 adjustment going to token that would complete an immediate repetition of length 32 and a +17 going to a token that does not appear in the previous 512 tokens.
4 Results
We perform an empirical study of how the LZ penalty affects repetition and capability in reasoning benchmarks and a performance study of the SGLang reference implementation.
4.1 Repetition and Capability Benchmarks
We apply the T.Z penalty to QwQ-32F and R1-Distill-14f We GPQA and AIME benchmarks (averaging scores and computing std. dev. over 5 runs). We set max token limit of 24k. We fixed the top-p to 0.95 and the top-k to 40 for all runs For all runs also using the LZ penalty, we fix the penalty strength a to 0.15, the window size to 512 and the buffer size to 32. We found that this configuration of hyperparameters seemed to work well across both models and both datasets with minimal tuning required We detect degenerate repetitions via dual verification of a GPT-4o based judge and a naive search for exact repetitions
Baselines We compare the LZ penalty against two industry-standard penalties: the repetition penalty and the frequency penalty. In both cascs, we fincly sweep small values up until getting to large values. For the results the full sweep of penalty values, refer to the Appendix.
Discussion Based on Fig. 2, we observe that neither penalty is reliable solution. The frequency penalty fails dramatically even for low values. We suspect that this is because of the length of the generations. Reasoning models produce reasoning traces that can be several thousand tokens long, which simply overwhelms the frequency penalty on common but essential tokens. The repetition penalty works significantly better than the frequency penalty and does seem to provide some modest relief. However, it is far from complete solution, with low temperature degenerate repetition rates up to about ~ 4% depending domain. This would be disqualifying for any kind of serious application. On the other hand, the LZ penalty achieves effectively zero degenerate repetitions without affecting top-line benchmark scores. The LZ penalty works because it adaptively penalizes based on both the length of the match as well as how far back the match occurs. LZ penalty's strength increases quickly in match length and attenuates gradually with distance. Neither the repetition penalty nor the frequency penalty can forget tokens, whercas the LZ penalty quickly and then gradually weakens as the token becomes less recent, until it moves beyond the lookback window hetogether.
model and task

---

Published in Transactions on Machine Learning Research (02/2026)
Model Performance Temperature Different Penalties
Accuracy
Degenerate Regetition %
16.0%
0.6
QNQ-328 GPQA
4.0%
0.-
2.0%
Repetttonn
1.0%
0.5%
0.2%
<0.1%
0.0
0.0
0.4 Temperature
02
Temperature
Accuracy
Degenerele ebe Repetition %
64.0%
0.0
37.0%
0.7
QwQ AIME
0.5
Repetiton
2.0%
1.0%
82
Temperature
Degenerate Repetition
0.6
16.0%
R1 Distilll144 GPOA
0.5
8.0%
0.4
Repecition (Loq
0.3
0.0
0.6
0.2
10
Tempersture
Temperature
Accuracy
Degenerate Regetition
32.0%
0.7
16.0%
Ri-Distill-1a 40 AIME
0.6
8.0%
0.5
0.6
2.0%
0.3
02
0.5%
011
-0.11%
00
0.0
02
0-6
8.6
0.0
1.0
0.0
02
0.8
0.6
0.0
100
Temperature
Temperoture
Ponalty (Baseino)
Repatition
Frequency
Figure 2: Line charts showing the accuracy and repetition percentage for baseline (repetition penalty of 1, frequency penalty of 0), the LZ penally, the repetition penalty, and the frequency penally. Accuracy error bars indicate the empirical std. dev. 5 runs. We feature the best performing choice of repetition penalty and frequency penalty strengths.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Model Sixe Med. Latency (ms)
Med. Throughput (tok/4)
Slowdown (%)
1.5B
4.43
14349.88
1,5B LZ
14370.95
0.55
7B
7.96
4020.00 06
7B + LZ
7.97
4014.29
0.14
32B
26.71
299,35
32B + LZ
26.71
299.47
0.03
Table Median latency, throughput, and LZ penalty's throughput slowdown for Qwen-2.5 architecture using SGLang's default benchmarking script. Context length: 1024, generation length: 64. Batch sizes: 64 (1.5B), 32 (7B), 8 (32B).
4.2 Latency and Throughput Benchmark
Although our SGLang reference implementation is not fully optimized, it is vectorized and batched. We run SGLang's built-in benchmark script on an 8xH100 node and compare the effect of adding non-zero L.Z penalty. While the LZ penalty adds an ultimately negligible amount of computation, it still is significantly more than, say, the repetition penalty, so is worthwhile to confirm that we can maintain inference performance.
We see that for larger models, the LZ penalty's overhead is increasingly negligible. Even for models as small 1.5B, the penalty overhead is tolerable 0.55% throughput slowdown. For latency, the overhead is more trivial and is not even measurable at the 32B size.
5 Conclusion
We presented the Lempel-Ziv (LZ) penalty, an informatiinn-teoreliice decoding strategy that suppresses degenerate repetitions in autoregressive language models by leveraging the prediction compression duality, Unlike frequency and repetition penaltics, the L. penalty adaptively accounts for both match length and recency, enabling reasoning models to decode determánistically without loss capability. Empirical studies show that the 1.7. penalty eliminates degenerate loops while preserving reasoning benchmark accuracy, with negligible computational overhead. These findings suggest that cumpression-inforeee penalties offer a principled and practical path toward more reliable language model decoding.
References
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. In Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning, pp. 10-21, 2016.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, Hoboken, NJ, edition, 2006.
Grégoire Delétang, Anian Ruoss, Pabl-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Mareus Hutter, and Joel Veness. Language modeling compression, 2024. URL hhttss//aaxiv. org/ abs/2399. 10668
Edward R. Fiala and Daniel H. Greene. Data compression with finite windows. Communications of the ACM, (244::90-505: 1989.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong
4Quea/0uQ-32B
Sheepseek-ai/eeepeeee -ateeeee-------isii----eeeeeeeee
6These the recommended sampling parameters by the Qwen team
"We only tested other penalty strengths in original sweep of 0.1, 0.2, and 0.3. We found that 0.2 sometimes too high and 0.1 was sometimes too low. Thus 0.15 seemed be sweet spot. The window and buffer sizes were selected intuition and did not seem require changing.
8 Any substring repeated least. 20 times.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhnang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xi Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl incentivizes reasoning in iims through reinforcement learning. Nature, (80588888):333 638, September 2025. ISSN 1476-4687. doi: 10. 10.1008/ //////41556000 02590222- URL http://dt:::: doi org. 10.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
David A. Huffman. A method for the construction of minimun-redundancy codes. Proceedings of the IRE, 40(9):1098 1101, 1952.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling, arXiv preprint 160:01160..02400, 2016.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprinl arxiv: N5::1909.0588 009.05558, 2019.
Abraham Lempel and Jacob Ziv. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3):337-334 1977.
Abraham Lempel and Jacob Ziv. Compression of individual sequences via variable-rate coding. IEFE Transactions on Information Theory, 24(5):530-553 536, 1978.
Victor S. Miller and Mark N. Wegman. Variations on a theme by ziv and lempel. Combinatorial Algorithms on Words, pp. 131-140, 1985. NATO ASI Scries, Volume F12.
H. Morita and K. Kobayashi. On asymptotic optimality of a sliding window variation of lempel-ziv codes. IEEE Transactions on Information Theory, 3046)18840-1846, 1993. doi: 10.1109/18.265494.
Markus F. X. J. Oberhumer. Lzo: A real-time data compression library. Available at bettps://www.oberhhe com/ conn/ppensource//zo// 1997. First released in 1997, Accessed: March 26, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library, 2019. URL https://pytorch.orgg org/
Igor Pavlov. Lzma sdk (software development kit). Available at httpt::// 7 7-2ip. wtt.iiiig///aa...ee sdk.mmkll 2007. Accessed: March 26, 2025.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Nla Sutskever. Language models are unsupervised multitask learners. OpenAT I Blog, 2019. Technical report.
Boris Ryabko. Compression-based methods for nonparametric density estimation, on-line prediction, regression and classification for time series, 2007. URL
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010)- IEEE Transac- lions on Aulonomous Mental Development, 2(3):230-244) 2010.
Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27(3):379 423, July 1948. doi: 10. 153000//1118--7000000 7305. tb01338..
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLanghlin, Aiden Low, AJ Ostrow, Akhila A Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb. Alex Beutel, Alex Karpenko, Alex Makelov, Alex Neitz, Alex Wei, Alexandra Barr, Alexandre Kirchmeyer, Alexey Ivanov, Alexi Christakis, Alistair Gillespie, Allison Tam, Ally Bennett, Alvin Wan, Alyssa Huang, Amy McDonald Sandjideh, Amy Yang, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrei Gheorghe, Andres Garcia Garcia, Andrew Braunstein, Andrew Liu, Andrew Schmidt, Andrey Mereskin, Andrey Mishchenko, Andy Applebaum, Andy Rogerson, Ann Rajan, Annie Wei, Anoop Kotha, Anubha Srivastava, Anushree Agrawal, Arun Vijayvergiva, Ashley Tyra, Ashvin Nair, Avi Nayak, Ben Eggers, Bessie Ji, Beth Hoover, Bill Chen, Blair Chen, Boaz Barak, Borys Minaiev, Botao Hao, Bowen Baker, Brad Lightcap, Brandon McKinzie, Brandon Wang, Brendan Quinn, Brian Fioca, Brian Hsu, Brian Yang, Brian Yu, Brian Zhang, Brittany Brenner, Callie Riggins Zetino, Cameron Raymond, Camillo Lugaresi, Carolina Paz, Cary Hudson, Cedric Whitney, Chak Li, Charles Chen, Charlotte Cole, Chelsca Voss, Chen Ding, Chen Shen, Chengdu Huang, Chris Colby, Chris Hallacy, Chris Koch, Chris Lu, Christina Kaplan, Christina Kim, CJ Minott-Henrighe Frey, Cody Yu, Coley Czarnecki, Colin Reid, Colin Wei, Cory Decareaux, Cristina Scheau, Cyril Zhang, Cyrus Forbes, Da Tang, Dakota Goldberg, Dan Roberts, Dana Palmie, Daniel Kappler, Daniel Levine, Daniel Wright, Dave Leo, David Lin, David Robinson, Declan Grabb, Derek Chen, Derek Lim, Derek Salama, Dibya Bhattacharjee, Dimitris Tsipras, Dinghua Li, Dingli Yu, DJ Strouse, Drew Williams, Dylan Hunn, Ed Bayes, Edwin Arbus, Ekin Akyurek, Elaine Ya Le, Elana Widmann, Eli Yani, Elizabeth Prochl, Enis Sert, Enoch Cheung, Eri Schwartz, Eric Han, Eric Jiang, Eric Mitchell, Eric Sigler, Eric Wallace, Erik Ritter, Erin Kavanaugh, Evan Mays, Evgenii Nikishin, Fangyuan Li, Felipe Petroski Such, Filipe de Avila Belbute Peres, Filippo Raso, Florent Bekerman, Foivos Tsimpourass, Fotis Chantzis, Francis Song, Francis Zhang, Gaby Raila, Garrett McGrath, Gary Briggs, Gary Yang, Giambattista Parascandolo, Gildas Chabot, Grace Kim, Grace Zhao, Gregory Valiant, Guillaume Leclere, Hadi Salman, Hanson Wang, Hao Sheng, Haoming Jiang, Haoyu Wang, Haozhun Jin, Harshit Sikchi, Heather Schmidt, Henry Aspegren, Honglin Chen, Huida Qiu, Hunter Lightman, Ian Covert, Ian Kivlichan, Ian Silber, Ian Sohl, Ibrahim Hammoud, Ignasi Clavera, Ikai Lan, Ilge Akkaya, Ilya Kostrikov, Irina Kofman, Isak Etinger, Ishaan Singal, Jackie Hehir, Jacob Huh, Jacqueline Pan, Jake Wilezynski, Jakub Pachocki, James Lee, James Quinn, Jamie Kiros, Janvi Kalra, Jasmyn Samaroo, Jason Wang, Jason Wolfe, Jay Chen, Jay Wang, Jean Harb, Jeffrey Han, Jeffrey Wang, Jennifer Zhao, Jeremy Chen, Jerene Yang, Jerry Tworek, Jesse Chand, Jessica Landon, Jessica Liang, Ji Lin, Jiancheng Liu, Jianfeng Wang, Jie Tang, Jihan Yin, Joanne Jang, Joel Morris, Joey Flynn, Johannes Ferstad, Johannes Heidecke, John Fishbein, John Hallman, Jonah Grant, Jonathan Chien, Jonathan Gordon, Jongsoo Park, Jordan Liss, Jos Kraaijeveld, Joseph Guay, Joseph Mo, Josh Lawson, Josh McGrath, Joshua Vendrow, Joy Jiao, Julian Lee, Julie Steele, Julie Wang, Junhua Mao, Kai Chen, Kai Hayashi, Kai Xiao, Kamyar Salahi, Kan Wu, Karan Sekhri, Karan Sharma, Karan Singhal, Karen Li, Kenny Nguyen, Keren Gu-Lemberg, Kevin King, Kevin Liu, Kevin Stone, Kevin Yu, Kristen Ying, Kristian Georgiev, Kristie Lim, Kushal Tirumala, Kyle Miller, Lama Ahmad, Larry Lv, Laura Clare, Laurance Fauconnet, Lauren Itow, Lauren Yang, Laurentia Romaniuk, Leah Anise, Lee Byron, Leher Pathak, Leon Maksin, Leyan Lo, Leyton Ho, Li Jing, Liang Wu, Liang Xiong, Lien Mamitsuka, Lin Yang, Lindsay McCallum, Lindsey Held, Liz Bourgeois, Logan Engstrom, Lorenz Kuhn, Louis Feuvrier, Lu Zhang, Lucas Switzer, Lukas Kondraciuk, Lukasz Kaiser, Manas Joglekar, Mandeep Singh, Mandip Shah, Manuka Stratta, Marcus Williams, Mark Chen, Mark Sun, Marsclus Cayton, Martin Li, Marvin Zhang, Marwan Aljubeh, Matt Nichols, Matthew Haines, Max Schwarzer, Mayank Gupta, Meghan Shah, Melody Huang, Meng Dong, Mengqing Wang, Mia Glaese, Micah Carroll, Michael Lampe,
Cliff
13

---

Published in Transactions on Machine Learning Research (02/2026)
Michael Malck, Michael Sharman, Michael Zhang, Michele Wang, Michelle Pokrass, Mihai Florian, Mikhail Pavlov, Miles Wang, Ming Chen, Mingxuan Wang, Minnia Feng, Mo Bavarian, Molly Lin, Moose Abdool, Mostafa Rohaninejad, Nacho Soto, Natalie Staudacher, Natan LaFontaine, Nathan Marwell, Nelson Liu, Nick Preston, Nick Turley, Nicklas Ansman, Nicole Blades, Nikil Pancha, Nikita Mikhaylin, Niko Felix, Nikunj Handa, Nishant Rai, Nitish Keskar, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk Olivia Watkins, Oona Gleeson, Pamela Mishkin, Patryk Lesiewicz, Paul Baltescu, Pavel Belov, Peter Zhokhov, Philip Pronin, Phillip Guo, Phoebe Thacker, Qi Liu, Qiming Yuan, Qinghua Liu, Rachel Dias, Rachel Puckett, Rahul Arora, Ravi Teja Mullapudi, Raz Gaon, Reah Miyara, Rennie Song, Rishabh Aggarwal, RJ Marsan, Robel Yemiru, Robert Xiong, Rohan Kshirsagar, Rohan Nuttall, Roman Tsiupa, Ronen Eldan, Rose Wang, Roshan James, Roy Ziv, Rui Shu, Ruslan Nigmatullin, Saachi Jain, Saam Talaie, Sam Altman, Sam Arnesen, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Sarah Yoo, Savannah Heon, Scott Ethersmith, Sean Grove, Sean Taylor, Sebastien Bubeck, Sever Banesiu, Shaokyi Amdo, Shengjia Zhao, Sherwin Wu, Shibani Santurkar, Shiyu Zhao, Shraman Ray Chaudhuri, Shreyas Krishnaswamy, Shuaiqi, Xia, Shuyang Cheng, Shyamal Anadkat, Simón Posada Fishman,, Simon Tobin, Siyuan Fu, Somay Jain, Song Mei, Sonya Egoian, Spencer Kim, Spug Golden, SQ Mah, Steph Lin, Stephen Imm, Steve Sharpe, Steve Yadlowsky, Sulman Choudhry, Sungwon Eum, Suvansh Sanjeev, Tabarak Khan, Tal Stramer, Tao Wang, Tao Xin, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Degry, Thomas Shadwell, Tianfu Fu, Tianshi Gao, Timur Garipov, Tina Sriskandarajah, Toki Sherbakov, Tomer Kaftan, Tomo Hiratsuka, Tongzhou Wang, Tony Song, Tony Zhao, Troy Peterson, Val Kharitonov, Victoria Chernova, Vineet Kosaraju, Vishal Kuo, Vitchyr Pong, Vivek Verma, Vlad Petrov, Wanning Jiang, Weixing Zhang, Wenda Zhou, Wenlei Xie, Wenting Zhan, Wes McCabe, Will DePue, Will Ellsworth, Wulfie Bain, Wyatt Thompson, Xiangning Chen, Xiangyu Qi, Xin Xiang, Xinwei Shi, Yann Dubois, Yaodong Yu, Yara Khakbaz, Yifan Wu, Yilei Qian, Yin Tat Lee, Yinbo Chen, Yizhen Zhang, Yizhong Xiong, Yonglong Tian, Young Cha, Yu Bai, Yu Yang, Yuan Yuan, Yuanzhi Li, Yufeng Zhang, Yuguang Yang, Yujia Jin, Yun Jiang, Yunyun Wang, Yushi Wang, Yutian Liu, Zach Stubenvoll, Zehao Dou, Zheng Wu, and Zhigang Wang. Openai gpt-5 system card, 2025. URL https://arxii: http://axxvv...aaaaa/// /wtgg/bb//601.....
James A. Storer and Thomas G. Szymanski. Data compression via textual substitution. Journal of the ACM, (4 951, 1982.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulikov, Ameet Prasad, Sharan Narang, al. Lamda: Language models for dialog applications. arXiv preprint 0rv::220010822 2022.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Terry Welch. A technique for high-performance data compression. Computer, 17(6):8 19, 1984.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, 2019. URL
lan H. Witten, Radford M. Neal, and John G. Cleary. Arithmetic coding for data compression. Communications of the ACM, 30(6):520-540, 1987.
A.D. Wyner and J. Ziv. The sliding-window window lempel-ziv algorithm is asymptotically optimal. Proceedings of the IEEE, 82(6):872 877, 1994. doi: 00.1109/5.286190
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu,, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingleng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxxi.. httpp::///aaaiv.orrr/b///////// wwtppaa////......0 09388
14

---

Published in Transactions on Machine Learning Research (02/2026)
Haruyasu Yoshizaki. Lha: A data compression and archiving tool. Software implementation, no formal publication, 1988. First released in 1988, combines LZSS with Huffman coding.
15


Reviews:
[{"review_id": "VLmsoce2Q6", "reviewer": "Reviewer_pfie", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a technique to mitigate repetition issues in large language model (LLM) outputs through the lens of code compression. Experiments on various mid-sized LLMs demonstrate that the proposed method significantly reduces repetition compared to the baseline while maintaining comparable accuracy. The main weakness is that the contribution seems a little marginal compared to repetition penality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments, particularly those shown in Figure 2, demonstrate that the proposed algorithm serves as a strong alternative to existing baselines such as repetition penalty and frequency penalty.", "audience_yn": "Yes", "audience_explanation": "The repetition issue in large language models (LLMs) is a universally important problem, particularly for smaller models. This paper investigates this important topic and provides valuable new algorithm into mitigating repetition.", "requested_changes": "The compression perspective is not clearly presented in the introduction, which defers the reader’s understanding to much later sections. I suggest improving the intuitive explanation — for instance, it would be helpful to clarify what is meant by “compression of the generation.”\n\nSecondly, while the reduction in repetition shown in Figure 2 is noticeable, the improvement over the repetition penalty baseline appears relatively marginal. Moreover, the accuracy on R1-DISTILL-14B-AIME is even lower than that of the repetition penalty method. I recommend enhancing the proposed algorithm to better demonstrate its practical effectiveness and ability to outperform existing repetition-control baselines.\n\nFurthermore, how does your method perform in smaller models where the repitition errors are more phenomenal?", "broader_impact_concerns": "N/A", "additional_comments": "Typos and cross-references. There are several typographical errors (e.g., “ir still fails” → “it still fails”) and incorrect figure references. In particular, the sentence in Section 4.1 should cite Figure 2, not Figure 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ilvfF0jGxk", "reviewer": "Reviewer_cxDk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper demonstrate the LZ penalty enables open-source reasoning models to operate with greedy decoding without loss of capability and without instances of degenerate repetition.", "claims_and_evidence_yn": "No", "claims_explanation": "It is not clear what's the assumption behind this paper. LZ77 and corresponding compression algorithms does not seem to be a natural correspondnece to real world data. (As real world data might have a lot of discrepancy across data distributions). It is necessary to discuss when and how this LZ77 will help the performance of the model .", "audience_yn": "Yes", "audience_explanation": "This is a very typical regularization approach so if we can have a better understanding of this method, we might help the model learn the target distribution more efficiently.", "requested_changes": "Try to provide examples of when and how LZ77 type regularization method do not help. Analyze when to use this method how to make the performance maximized. If a clear theoretical justification is hard to achieve, feel free to provide folklore statement.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rufIQjQMQV", "reviewer": "Reviewer_aPYm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a decoding \"penalty\" for LLMs that is based on the Lempel-Ziv77 compression algorithm. Intuitively, the method increases logits with a large code length more and increases less (or decreseases) logits with a small code length. In contrast to previous methods like frequency or repitition penalty, the proposed LZ penalty considers length-normalized n-gram statistics as opposed to single token statistics. This way, the penalty is more \"surgical\" in how it modulates the sampling distribution and thereby prevents degenerate cycling effects common with frequency penalty, while being more nuanced than the binary repitition penalty.\n\n**Strengths**\n\nOverall, I think it is a great paper. I enjoyed reading it (apart from the rendering issues).\n\n1. The proposed LZ penalty is plausible and works well, as clearly shown in the experiments.\n   \n2. The proposed LZ penalty seems easy to integrate.\n\n3. Overall, the paper is easy to read (though some things could be written more clearly, see *requested changes*).\n\n**Weaknesses**\n\n1. A proper related work section is missing (there is only a background section) and generally, there are few references. I am not an expert at LLM decoding algorithms, but I would be suprised if not more similar ideas (besides [1], which the authors cite) exist. If indeed there is little, one could broaden the scope. For instance, I feel loosely reminded of [2] who incentivize an \"artist/scientist\" agent to generate data that is not easily compressable by some compression method. I am not saying that [2] is the paper that must be discussed: It is likely that more closely related works than [2] exist by now. Anyway, a related work section of some kind should be present so the reader can better categorize the contribution within the existing literature.\n\n2. A limitations section is also missing. As a reader, I need to guess what the shortcomings of the proposed method are and I believe that the authors need to state and discuss this more clearly in such a dedicated section (TMLR is not like the toxic ML conferences, so there should be no incentive to be secretive about limitations). I believe the main shortcomings are (please correct me if I am wrong): (1) Three tuning parameters instead of just a single one for the baselines (repition and frequency penalty) (2) Additional runtime due to the $\\texttt{findLongestMatch}$ function, which seems quadratic in complexity (experiments for this point exist, which is good, but no critical discussion). Please add such a section to discuss these and potentially further limitations that I did not catch.\n\n3. A conclusion is also missing. I believe this is important.\n\n\n[1] Boris Ryabko. \"Compression-based methods for nonparametric density estimation, on-line prediction, regression and classification for time series\" (2007).\n\n[2] Schmidhuber, Jürgen. \"Formal theory of creativity, fun, and intrinsic motivation (1990–2010).\" IEEE transactions on autonomous mental development 2.3 (2010): 230-247.", "claims_and_evidence_yn": "Yes", "claims_explanation": "A descent experimental evaluation is present.", "audience_yn": "No", "audience_explanation": "The paper is not rendered correctly. I would normally not read a paper with such rendering issues. Therefore, I deem it not interesting.", "requested_changes": "I will proceed to list requested changes one-by-one. Points that are critical for meeting the acceptance criteria are highlighted by *(critical)*.\n\n**Overall**\n\n- *(critical)* The rendering must be fixed. In the current state, the paper must not be accepted.\n\n- *(critical)* A discussion/limitations section is missing.\n\n- *(critical)* A conclusion section is missing.\n\n**Title**\n\nThis is highly subjective, but I do not like acronyms in the title. Would it be possible to just write out Lempel-Ziv penalty in the title? I would do that as long as there is not good reason against doing so.\n\n**Abstract**\n\n- *\"Both the industry-standard frequency penalty and repetition penalty are\nineffective, incurring degenerate repetition rates of up to 4% or more.\"* If you state how bad existing methods are, I would expect that you write how much better your method is in comparison. Otherwise, the statement is meaningless. Please highly consider adding that info in the abstract.\n\n**1 Introduction**\n\n- *\"In recent months, there has been an advent in reasoning models.\"* Please consider adding some references to support this claim.\n\n- *\"Reasoning models do not run well at low temperatures because the sampling distribution\nmode collapses into degenerate repetitions.\"* This sentence is out of context to me. The previous sentence was about lack of determinism around the reasoning trace and now we talk about problems at low temperatures.\n\n- *\"Furthermore, if the repetition penalty is set too high\"* So why not just tune the penalty then? Please consider adding an explanation for why that is difficult or cannot be done.\n\n- *\"On the other hand, the frequency penalty is more adaptive\"* Why *\"on the other hand\"*? Where is the *\"on the one hand\"*? Consider instead writing *\"In constrat to the repition penalty, the frequency penalty is more adaptive\"*\n\n- *\"Ultimately, this still fails. Instead, it produces an interesting degenerate\"* This is an odd formulation. Why not just write *\"However, it still fails, because it produces an (interesting) degenerate ...\"*?\n\n- *\"This is a real excerpt\"* Please consider dropping *\"real\"*... I hope that all of the results presented in this paper are real.\n\n- *\"Excerpt from QwQ-32B using a frequency penalty of 0.3 and temperature of 0\"* From my understanding, this cannot be. If the temperature is truly $0$, then you divide by $0$, no? Please consider clarifying.\n\n- *\"Following the principle, we give a quick gist of the proposed LZ penalty:\"* I like the simplified three-step presentation. However, the notation is not clear. For instance, what is $l$? A logit? I would suggest either explaining the symbols or omitting them (the latter is ok, given that this is the introduction).\n\n**2.1 Language Modeling and Sampling**\n\n- *where he trained a casual language mode*. Typo: \"casual\" -> \"causal\".\n\n- *Definition 1. Data Sequence.* This is a subjective comment, but I find all of these definitions a bit of an overkill. This paper is not very mathematical, so I do not see why the notation needs to be introduced like this.\n\n**2.3 The Prediction-Compression Duality**\n\n- *Proposition 3* I believe that an additive constant is missing in the final equation, because of the normalization. I.e., I think it should be $ \\frac{1}{n} \\sum_{i=1}^n \\ell_{DC} (x_i | x_{<i})  =  -\\frac{1}{n} \\sum_{i=1}^n \\text{log}  p_{DC} (x_i | x_{<i}) + const.$.\n\n- *\"Constructing casual language models\"* Typo: \"casual\" -> \"causal\".\n\n**3 LZ Penalty**\n\n- *\"duality to construct a an LZ\"* Typo: Remove \"an\".\n\n- *equation (3)* The equation makes sense, but I believe the terminology is incorrect. Let me clarify: If a token has not been sampled often yet, then it has a large code length and its probability of being sampled increases. Likewise, if a token has already been used many times, its codelength is smaller and its probability of being sampled will be smaller compared to \"rarer\" tokens. Is that correct? If yes, then *penalty* does not seem like the right word, because penalization would imply that a large penalty should make the token less likely, no? Anyway, using such an example like I just did would greatly help understanding what is going on.\n\n- *this yields on adjustment range from −5 to +17, with a −5 adjustment going to a token that would complete an immediate repetition of length 32 and a +17 going to a token that does not appear in the previous 512 tokens.* This seems to confirm my belief of how this works. However, as mentioned in the comment above, I believe the term *penalty* could be missleading. I would suggest replacing this term by *adjustment* or *offset* or something like that.\n\n**Results**\n\n- *(critical)* *\"For all runs also using the LZ penalty, we fix the penalty\nstrength $\\alpha$ to 0.15, the window size to 512 and the buffer size to 32\"* In the introduction, the authors were mentioning that it is difficult to tune the repition penalty for the baseline. However, it seems that the proposed method has three parameters that all need to be tuned (not just a single one). Something like this must be critically discussed.", "broader_impact_concerns": "None", "additional_comments": "I decided to write a review for the paper in spite of the rendering issues, which was quite annoying. To show respect towards the reviewers, I would kindly ask the authors to check for such issues more carefully before uploading a paper next time.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)