<figure><img src="image_1.png" /></figure>

HAL
open science

# Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation

Ivanhoé Botcazou, Tassadit Amghar, Sylvain Lamprier, Frédéric Saubion

<figure><img src="image_2.png" /></figure>

## To cite this version:

Ivanhoé Botcazou, Tassadit Amghar, Sylvain Lamprier, Frédéric Saubion. Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation. 2025. hal-05462602

HAL Id: hal-05462602

https://hal.science/hal-05462602v1

Preprint submitted on 19 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation

Ivanhoé Botcazou

Tassadit Amghar

Sylvain Lamprier

Frédéric Saubion

LERIA, University of Angers

{ivanhoe botcazou, tassadit amghar, sylvain lamprier, frederic saubion}@univ-angers.fr

## Abstract

Modern neural language models achieve high accuracy in text generation, yet precise control over generation length remains underdeveloped. In this paper, we first investigate a recent length control method based on *Reverse Positional Embeddings (RPE)* and show its limits when control is requested beyond the training distribution. In particular, using a discrete countdown signal tied to the absolute remaining token count leads to instability. To provide robust length control, we introduce *Progress Ratio Embeddings (PRE)*, as continuous embeddings tied to a trigonometric *impatience* signal. PRE integrates seamlessly into standard Transformer architectures, providing stable length fidelity without degrading text accuracy under standard evaluation metrics. We further show that PRE generalizes well to unseen target lengths. Experiments on two widely used news-summarization benchmarks validate these findings.

## 1 Introduction

Over the past few years, Language Models (LMs) have achieved remarkable progress in natural language processing, reaching state-of-the-art performance across a wide range of text generation tasks. Despite these advances, generating outputs that meet specific structural or stylistic constraints remains challenging. Among such constraints, length control is particularly fundamental, as the appropriate degree of compression or elaboration often depends on the task and the desired granularity of information. In tasks such as summarization, for instance, the compression ratio between the source and the output naturally correlates with the number of words or tokens produced. Yet, controlling this aspect precisely remains both a technical and a cognitive challenge for neural language models (Kikuchi et al., 2016).

During inference, LMs generate tokens sequentially until the process terminates by the sampling

of the <EOS> token. However, this stopping criterion is inherently stochastic, and the decision mechanism guiding token selection at each step depends primarily on the previously generated context. As a result, the overall planning of the generation process lacks transparency, offering only limited opportunities for explicit control. In practice, users can request outputs of varying lengths, ranging from concise phrases to long, elaborated texts. For example, in abstractive summarization, one may seek either very short TL;DR-style summaries or lightly compressed paragraphs that preserve much of the original content. While instruction-following Large Language Models (LLMs) can express length preferences through prompting (Lou et al., 2024; Dong et al., 2024), this form of in-context control remains restricted. It is effective only for very large models trained on extensive amounts of general-purpose data. In this paper, we focus on the setting of Small Language Models (SMLs) trained on restricted, domain-specific datasets, where length control must instead be achieved through architectural design.

Recent approaches have addressed the problem of target length control in text generation, through the use of length-informed positional embeddings integrated into the inputs fed to the decoder of the LM. For example, (Miculicich et al., 2023; Butcher et al., 2025) propose to feed the Transformer decoder with reverse positional encodings that depend on the remaining number of tokens to be decoded given a desired target length. Although this enables effective length control for summaries whose lengths fall within a restricted window centered on the mode of the training distribution, in this paper, we show that it performs poorly for out-of-distribution target lengths.

To avoid this limitation, we introduce continuous *Progress Ratio Embeddings* (PRE), which reduce the dependence on the discrete target length provided to the model. In contrast to discrete Re-

arXiv:2512.06938v1 [cs.CL] 7 Dec 2025
verse Positional Embeddings (RPE), PRE offers a smooth and continuous representation of progress that generalizes beyond the training distribution of reference lengths. PRE embeddings are introduced through trigonometric impatience signals whose frequency gradually increases until the end of generation. This design aligns the decoding process more closely with the user-specified target length and, by construction, makes it less dependent on the distribution of training reference lengths. We further integrate PRE into two notable pre-trained Transformer architectures (Vaswani et al., 2017), achieving robust length invariance without compromising relevance or informativeness in abstractive summarization tasks.

Our main contributions are threefold:

(1) We reproduce and adapt the RPE method on a classical encoder-decoder architecture, and provide an analysis of its strengths and weaknesses;

(2) We propose a novel length control strategy, called Progress Ratio Embeddings (PRE), based on continuous “impatience” signals that modulate the decoding process according to the completion progress;

(3) We demonstrate, on standard text generation tasks, that PRE models can generalize length control beyond the training distribution, producing outputs of previously unseen lengths while preserving fidelity and relevance. augmentation through varying length compression generated with our finetuned model.

The paper is organized as follows. Section 2 presents related works. In Section 3 we introduce our method, called PRE. Section 4 describes the setup that is used for the experiments presented in Section 5 and Section 6.

## 2 Related Work

The large number of parameters and the probabilistic nature of LMs associate them with Black Box algorithms, which suffer from well-known weaknesses such as a lack of explicability and limited controllability in planning (Zhao et al., 2023). Given an input context, the vast space of possible continuations forces an autoregressive model to select a trajectory via sampling to maximize a context-conditioned probability.

To improve controllability, recent research has shown that LM outputs can be empirically enhanced through *chain-of-thought prompting* (CTP). Within this perspective, existing studies (Jie et al., 2024) propose prompt-based methods combined with reinforcement learning to train prompt extractors that enhance both performance and controllability. Moreover, by integrating prompting with iterative trajectory sampling under a Metropolis-Hastings framework, Gu et al. (2024) demonstrate how LLMs can be regulated to enforce constraints such as output length, without requiring parameter modifications. In this paper, we instead focus on smaller language models trained on domain-specific or resource-constrained datasets, where reference summaries are distributed within a narrow length range. In such constrained settings, where contextual information about length is limited and prompting is not feasible, explicit architectural mechanisms become necessary to guarantee effective and reliable length control.

An alternative line of research focuses on modifying the decoding process itself. Decoding-based approaches, such as *LenAtten* (Yu et al., 2021), extend the autoregressive generation mechanism by conditioning the prediction head not only on the decoder's hidden state but also on an additional length-context vector, thereby enabling finer-grained control over the generated output length. Beyond prompting and decoding, other studies introduce structural modifications to the model architecture. For example, the Length-Aware Attention Mechanism (LAAM), (Liu et al., 2022) operates at the attention level by gradually adjusting the cross-attention weights in a length-dependent manner, particularly between the <EOS> token in the input text and each decoder token (Appendix E).

In parallel, RPE, initially introduced by (Takase and Okazaki, 2019), encodes the remaining length directly into positional representations. Based on this concept, Miculicich et al. (2023) and Butcher et al. (2025) fine-tune pre-trained Transformers to incorporate such embeddings. At each decoding step, a discrete countdown index representing the remaining length is converted into a vector of the same dimension as the token and positional embeddings, and injected into the decoder (Appendix B). Through teacher-forcing training, this additional signal enables the model to predict more accurately the expected stopping point.

However, the RPE method is explicitly dependent on the target length observed during training.
As shown in Section 5, this dependency constitutes a critical limitation: RPE achieves accurate control only within the training length distribution, but globally fails to generalize for out-of-distribution lengths. This weakness highlights the need for novel architectural mechanisms that enable deterministic and more invariant length control in text generation. To address this challenge, we introduce the PRE method.

# 3 Continuous Length Control

In this section, we provide a brief overview of the role of positional embeddings in Transformer models. We then introduce our PRE method, which injects a lightweight *impatience signal* into the decoder inputs of a vanilla Transformer. We justify the theoretical foundation from the digital signal processing and expose the global fine-tuning aspects.

## 3.1 Transformers needs Positional Embeddings

Transformer-based LMs process sequences of ordered tokens, where each token $t$ is mapped to a token embedding<fnref n="1" /> $E_t \in \mathbb{R}^{d_{model}}$. Because of the self-attention setup by itself, the input tokens are permutation invariant, the model must be given positional information. Each token index position is mapped to a positional embedding $P_t \in \mathbb{R}^{d_{model}}$, which is added to the token embedding $E_t$ before the first Transformer layer<fnref n="2" />:

$$X_t = E_t + P_t. \qquad (1)$$

This addition places content and position information in the same vector $X_t$ and allows attention to condition on where a token occurs, not only on what it is. Depending on the model architecture, positional information can be encoded with fixed sinusoidal embeddings as initially introduced in (Vaswani et al., 2017). Alternatively, some models use learned embedding vectors optimized during pre-training as presented in (Gehring et al., 2017; Radford et al., 2018). A third family incorporates relative positional embeddings that capture relative distance between sequence elements (Shaw et al., 2018). In all cases, without positional information, any permutation of the input tokens yields the same multiset of attention scores.

<figure><img src="image_4.png" /><figcaption>Figure 1: Illustration of Progress Ratio Embeddings added to token and positional embeddings to form the input of the decoder.</figcaption></figure>

Positional embeddings make attention sensitive to token order, either for context encoding or for the sequence of already decoded tokens while inference. During decoding, RPE methods provide position embeddings in reverse order to indicate how many tokens remain to be generated, rather than only giving the absolute position from the start of generation. When trained with such reversed embeddings counting down from the initial target length to zero over the training reference the decoder can modulate generation length at inference time according to a user-specified target length. However, we argue that this approach, while effective for lengths seen during training, does not generalize well to unseen target lengths, which are often required in practical applications. In the following, we propose a new method to overcome this limitation.

## 3.2 Progress Ratio Embeddings as an Impatience Signal

We introduce the PRE method based on progress ratios $r \in [0, 1]$ that quantify the advancement of the decoding process. At decoding step $t$ for a requested target length $l$, we can define a progress ratio: $r_t = \frac{t}{l}$. As illustrated in Figure 1, a ratio-dependent embedding is added to the initial decoder input embeddings :

$$X_t = E_t + P_t + \xi(r_t) \qquad (2)$$

where $\xi: [0, 1] \to \mathbb{R}^{d_{model}}$ denotes what we call the *Progress Ratio Embedding function*. A crucial advantage over the finite set of reversed positional em-

<footnote n="1">¹An embedding is a dense vector of dimension $d_{model}$</footnote>

<footnote n="2">²Modern LLMs typically use RoPE (Rotary Positional Embeddings), which are relative positional encodings applied inside each attention block.</footnote>
beddings used in RPE prosess is that PRE defines embeddings continuously for any ratio within the interval [0, 1]. This allows the model to compute length-control information on the fly, covering the entire progress range rather than being restricted to discrete positions seen during training.

Having identified progress-ratio information as valuable for controlling output length, we now examine how to encode this information. Our objective is to design a smooth, continuous embedding that avoids overfitting to the discrete ratios observed during training, and that generalizes to unseen target lengths at inference time. Specifically, we assume that for any possible ratio $r \in [0, 1]$, there exists a regular periodic function that represents the *impatience signal* of the process. By analogy, this signal can be seen as the impatience of a listener awaiting the end of a message. Building on signal sampling theory, we design a stress signal with pulsation $\omega_r$ (i.e., corresponding to a frequency $F = \omega_r/(2\pi)$) that increases with the progress ratio $r$, which enables the decoder to receive embeddings that encode the generation progress across [0, 1]. We consider in the following $\omega_r = r \times M$, with $M$ an hyper-parameter that scales the frequency increase of the signal (Appendix A).

The Nyquist–Shannon theorem (Shannon, 1949) guarantees that any periodic, band-limited function can be perfectly reconstructed from samples taken at a rate $F_s$ exceeding twice its highest frequency component $F_{max}$. Additionally, sinusoids provide a natural basis for such signals, and using both sine and cosine components ensures an unambiguous, continuous encoding of phase. Following this principle, we design our embedding so that each pair of dimensions encodes a single frequency component via cosine and sine in alternating dimensions. Formally, for $1 \le j \le d_{model}$,

$$ \xi(r)_j = \begin{cases} \cos\left(\frac{2\omega_r \lfloor j/2 \rfloor}{d_{\text{model}}}\right), & \text{if } j \text{ is even,} \\ \sin\left(\frac{2\omega_r \lfloor j/2 \rfloor}{d_{\text{model}}}\right), & \text{if } j \text{ is odd,} \end{cases} \quad (3) $$

where $\omega_r$ controls the frequency of oscillations along the progress ratio. This formulation uses $x_j = \lfloor 2j/2 \rfloor$ as the independent variable for each dimension $j$, which corresponds to a sampling rate of $F_s = d_{model}/2$ evenly spaced points over [0, 1]. Taking $M = d_{model} \cdot \pi/2$ allows to benefit from the full capacity of the considered representation, while ensuring that the Nyquist–Shannon condition is satisfied, since we have in that case:

$F = \frac{\omega_r}{2\pi} \le 1 \cdot d_{model}/4 = \frac{d_{model}}{4}$, and thus: $F_s = \frac{d_{model}}{2} = 2F_{max}$. In practice, we use $M = \frac{d_{model}}{2}$, leaving a small safety margin below the Nyquist bound to accommodate rounding and numerical precision limits.

### 3.3 Finetuning with PRE

Let $A := (A_0, \dots, A_n)$ denote the sequence of tokens corresponding to the context of the task (e.g., the source article for summarization), and $S := (S_0, \dots, S_l)$ the sequence of tokens in a "gold" reference (e.g., a target summary), where $l$ is the expected number of tokens in the produced text. Following Section 3.2, we consider an associated family of Progress Ratio Embeddings $\Xi = (\xi(r_0), \dots, \xi(r_l))$ related to the expected target length $l$. Given a pretrained model<fnref n="3" />, our objective is to adapt its parameters so that it predicts the conditional probability $P(S | A, l)$.

Following the maximum likelihood estimation (MLE) principle, we optimize the model parameters $\theta$ to maximize the product of conditional probabilities. This product can be factorized using the chain rule (\*\*) and the causal property (\*\*\*) of the decoder:

$$ \begin{aligned} P_{\theta}(S | A, \Xi) &\stackrel{*}{=} \prod_{t=0}^{l} P_{\theta}(S_t | S_{<t}, A, \Xi) \\ &\stackrel{\text{\*\*}}{=} \prod_{t=0}^{l} P_{\theta}(S_t | S_{<t}, A, \Xi_{<t}). \end{aligned} $$

The training objective is defined by the standard cross-entropy loss, applied over mini-batches of size $m$. We minimize the empirical loss over a batch $B = \{(A^i, S^i, \Xi^i)\}_{1\le i\le m}$, defined as the average negative log-likelihood across all tokens.

$$ \mathcal{L}_B(\theta) = -\frac{1}{m} \sum_{i=1}^{m} \sum_{t=1}^{l_i} \log P_{\theta}(S_t^i | S_{<t}^i, A^i, \Xi_{\le t}^i) $$

Minimizing the sequence loss while conditioning the decoder on a normalized progress ratio provides a consistent source of temporal side information. To enhance generalization, Gaussian noise is injected into each ratio (Eq. 4), exposing the model to a continuous spectrum of values over [0,1] and

<footnote n="3">³In our experiments, we focus on encoder-decoder models, although our technique can be straightforwardly applied to decoder-only architectures by initializing the progress ratios at the start-of-sequence token.</footnote>
promoting smooth interpolation rather than dependence on a few discrete targets.

$$r = \text{Clip} \left( r + \frac{2\delta}{d_{\text{model}}} ; 0 ; 1 \right) \quad (4)$$

where $\delta \sim \mathcal{N}(0, 1)$

Once the generated sequence reaches or exceeds the target length, the ratio is clipped at 1 causing the impatience signal to saturate and thereby discouraging further generation.

# 4 Experimental Setup

In this section, we present and justify our choices of model, datasets, hyperparameters, and evaluation metrics as integral components of the PRE method's implementation and evaluation.

## 4.1 Model

For our experimentations, we used the proven BART encoder-decoder model as backbone (Lewis et al., 2019), it achieves strong performances on specific tasks and remains highly suitable for fine-tuning. As a sequence-to-sequence model pre-trained with a denoising objective inspired by BERT (Devlin et al., 2019), it has been shown to be effective in a range of NLP applications, including abstractive summarization. Released by Meta AI with about 400M parameters, it can be trained on standard computing resources and supports inputs of up to 1,024 tokens. For this model $d_{model} = 1,024$.

## 4.2 Datasets

We conduct experiments on two widely used news-summarization benchmarks and on an extractive question-answering dataset.

**CNN/DM** (Nallapati et al., 2016) contains articles paired with summaries that are typically closely aligned with source sentences.

**XSum** (Narayan et al., 2018) provides highly abstractive one-sentence summaries that require substantial rephrasing beyond the source text.

**SQuAD** (Rajpurkar et al., 2016) is the Stanford Question Answering Dataset, consisting of questions posed on Wikipedia articles.

## 4.3 Metrics

For each input article, there may exist multiple human-judged acceptable candidates that preserve the same level of information. Due to the existence of these valid alternatives, identifying metrics that

accurately capture the relevance and informative-ness of a generated summary remains highly challenging (Koh et al., 2022).

Given an input text and a gold-standard output summary, the most common metrics used to evaluate the relevance of abstractive candidates are ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004), which provide an F1-score based on n-gram overlap between the generated output and the reference summary. However, these metrics are often questioned, as they do not correlate strongly with human judgments (Deutsch et al., 2022). Despite this limitation, ROUGE remains the primary score reported in a large number of studies, including ours. Acknowledging the weaknesses of ROUGE, the BERTScore (B.S.) (Zhang et al., 2020) has been proposed to avoid this lack of relevance. We evaluate length controllability using the mean absolute error (MAE) between the generated and target lengths. Unless otherwise stated, lengths are measured in tokens as defined by each model's tokenizer.

## 4.4 Hyperparameters and devices

PRE-BART-L model was trained on the full training set with a batch size of 5, using the AdamW optimizer (Loshchilov and Hutter, 2019) with $\beta_1 = 0.9$, $\beta_2 = 0.99$, $\epsilon = 1 \times 10^{-8}$, and a learning rate of $1 \times 10^{-5}$. Experiments were conducted on NVIDIA V100 GPUs 4.

# 5 RPE results and limitations

We begin by reproducing the RPE method with BART-L and fine-tuning it on benchmark datasets<fnref n="5" />. This reproduction confirms the reported length-control behavior, as shown in Table 1, consistently regarding results from the original paper (Miculicich et al., 2023). It also indicates improved summarization quality relative to baselines without length control, which we reported in Table 2. RPE-BART-L achieves accurate length control within each 10-token target-length bucket.

However, in approximately 2% of cases, the MAE between the generated and expected lengths exceeds 10 tokens (see outliers in Figure 2). Moreover, as shown in Figure 3, when the requested

<footnote n="4">⁴grant 2025-AD011016042</footnote>

<footnote n="5">⁵To the best of our knowledge, the original method was not publicly available. We propose an implementation based on an open-source encoder-decoder BART model.</footnote>
target length exceeds 350 tokens<fnref n="6" />, the effectiveness of length control degrades markedly. This lack of generalization is precisely what motivated the introduction of our - length invariant - PRE approach, that we experiment in the next section.

<figure><img src="image_5.png" /><figcaption>Figure 2: MAE by target-length bucket (10 tokens) for RPE-BART-L on CNN/DailyMail.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 3: MAE by target-length bucket (25 tokens) for RPE-BART-L on CNN/DailyMail when target lengths above 300 tokens are requested.</figcaption></figure>

## 6 PRE results and analysis

Reported scores in Table 2 show that our PRE method consistently yields high-quality generated summaries while providing precise length control. The PRE method maintains high ROUGE/BERTScore and achieves a low MAE between target and generated lengths across the full

range of requested target lengths (Figures 4 and 14). Table 1 further contrasts PRE with both no-control baselines and our RPE reimplementation, confirming that PRE delivers superior controllability without sacrificing quality.

<figure><img src="image_7.png" /><figcaption>Figure 4: MAE by target-length bucket (10 tokens) for PRE-BART-L on CNN/DailyMail.</figcaption></figure>

<table><caption>Table 1: Length control evaluation on CNN/DailyMail and XSum using MAE and standard deviation (SD). **Legend:** G = reference length; N = no explicit length control. Lower values indicate better control.</caption><thead><tr><th>Dataset</th><th>Model</th><th>Type</th><th>MAE<sup>↓</sup> ± SD</th></tr></thead><tbody><tr><td rowspan="3">CNN/DM</td><td>BART-L</td><td>N</td><td>19.2 ± 17</td></tr><tr><td>RPE-BART-L</td><td>G</td><td>1.6 ± 3.6</td></tr><tr><td>PRE-BART-L</td><td>G</td><td>0.5 ± 0.3</td></tr><tr><td rowspan="3">XSum</td><td>BART-L</td><td>N</td><td>5.8 ± 5</td></tr><tr><td>RPE-BART-L</td><td>G</td><td>0.7 ± 1.1</td></tr><tr><td>PRE-BART-L</td><td>G</td><td>0.1 ± 0.2</td></tr></tbody></table>

<table><thead><tr><th colspan="5">CNN/DailyMail</th></tr><tr><th>Model</th><th>Type</th><th>R-1<sup>↑</sup></th><th>R-2<sup>↑</sup></th><th>R-L<sup>↑</sup></th><th>B.S.<sup>↑</sup></th></tr></thead><tbody><tr><td>BART-L (Lewis et al., 2019)</td><td>N</td><td>44.2</td><td>21.1</td><td>40.9</td><td>69.7</td></tr><tr><td>RPE-BART-L</td><td>G</td><td>44.5</td><td>21.2</td><td>41.3</td><td>69.4</td></tr><tr><td>PRE-BART-L</td><td>G</td><td>45.3</td><td>21.9</td><td>42.2</td><td>69.8</td></tr></tbody></table>

<table><caption>Table 2: ROUGE and BERTScore results on CNN/DailyMail and XSum. **Legend:** G = length from the reference summary is given; N = no explicit length control.</caption><thead><tr><th colspan="5">XSum</th></tr><tr><th>Model</th><th>Type</th><th>R-1<sup>↑</sup></th><th>R-2<sup>↑</sup></th><th>R-L<sup>↑</sup></th><th>B.S.<sup>↑</sup></th></tr></thead><tbody><tr><td>BART-L (Lewis et al., 2019)</td><td>N</td><td>45.14</td><td>22.27</td><td>37.25</td><td>73.3</td></tr><tr><td>RPE-BART-L</td><td>G</td><td>44.5</td><td>20.8</td><td>35.6</td><td>72.3</td></tr><tr><td>PRE-BART-L</td><td>G</td><td>45.2</td><td>21.3</td><td>36.4</td><td>72.7</td></tr></tbody></table>

<footnote n="6">⁶Target lengths were chosen according to the input article length and sampled at random from out-of-distribution lengths.</footnote>
## Generalization to out-of-distribution Data

We have introduced the PRE method to achieve more robust generalization of length control across the full range of lengths that the model can handle. Figure 5 illustrates that PRE-BART-L is actually able to generate summaries with accurate length control, even when the requested output length exceeds 300 tokens.

<figure><img src="image_8.png" /><figcaption>Figure 5: MAE by target-length bucket (25 tokens) for PRE-BART-L on CNN/DailyMail when target lengths above 300 tokens are requested.</figcaption></figure>

To evaluate behavior along all manageable context windows, we also investigate a comparative analysis between PRE and RPE methods on 6,000 randomly selected examples from the CNN/DailyMail test set. For each selected article, we randomly assigned a target summary length smaller than the article length and asked both models to generate a summary under this constraint. Table 3 reports corresponding results, in terms of the percentage of produced outliers for which MAE greater than 20 tokens (% Out). We clearly observe that the proportion of outliers is considerably smaller with PRE, highlighting its robustness in handling long and variable-length generation tasks. PRE encodes a continuous ratio signal normalized between 0 and 1. This distinction is crucial for handling arbitrary user-specified output lengths. RPE is more sensitive to the distribution of target lengths in the training data. As shown in Appendix C, long target lengths are underrepresented in the original dataset. Consequently, models trained with RPE rarely encounter such cases during fine-tuning, thus their parameters are not adapted to accommodate very long summaries.

<table><caption>Table 3: Comparison of outlier rate between RPE-BART-L and PRE-BART-L across length buckets. The Count column reports, respectively, the number of non-outlier samples for each model.</caption><thead><tr><th>Bucket</th><th>% Out RPE ↓</th><th>% Out PRE ↓</th><th>Count</th></tr></thead><tbody><tr><td>[300, 350)</td><td>10.8%</td><td>0.4%</td><td>839 / 922</td></tr><tr><td>[350, 400)</td><td>15.4%</td><td>1.0%</td><td>705 / 809</td></tr><tr><td>[400, 450)</td><td>22.3%</td><td>1.5%</td><td>608 / 745</td></tr><tr><td>[450, 500)</td><td>28.0%</td><td>3.4%</td><td>453 / 629</td></tr><tr><td>[500, 550)</td><td>37.0%</td><td>4.3%</td><td>357 / 577</td></tr><tr><td>[550, 600)</td><td>50.1%</td><td>5.4%</td><td>245 / 490</td></tr><tr><td>[600, 650)</td><td>51.4%</td><td>4.0%</td><td>216 / 429</td></tr><tr><td>[650, 700)</td><td>53.9%</td><td>7.4%</td><td>179 / 374</td></tr><tr><td>[700, 750)</td><td>64.8%</td><td>10.9%</td><td>124 / 351</td></tr><tr><td>[750, 800)</td><td>70.3%</td><td>9.9%</td><td>93 / 273</td></tr><tr><td>[800, 850)</td><td>75.1%</td><td>9.0%</td><td>76 / 242</td></tr><tr><td>[850, 900)</td><td>81.3%</td><td>8.4%</td><td>43 / 230</td></tr><tr><td>[900, 950)</td><td>84.9%</td><td>8.9%</td><td>35 / 214</td></tr><tr><td>[950, 1000)</td><td>95.8%</td><td>19.8%</td><td>4 / 65</td></tr></tbody></table>

## Question generation accommodation

Based on the SQuAD dataset training part, we fine-tune the BART model with both RPE and PRE for the question generation task, where the objective is to generate a relevant question given a context passage and its associated answer. As shown in Tables 4 and 5, the PRE variant exhibits clear advantages: it enables highly accurate length-controlled question generation while preserving the semantic quality of the outputs. Moreover, our method achieves strong results across multiple evaluation metrics, including BLEU, ROUGE, and BERTScore.

These experiments demonstrate the scalability of the PRE method to a broad range of text generation tasks. In particular, PRE provides more precise control over the target length than the RPE method (Figures 6 and 7), especially near the extremities of the desired length distribution observed during training. This highlights the robustness and reliability of PRE for fine-grained length-controlled generation, even under demanding representation constraints.

<table><caption>Table 4: Length control evaluation on SQuAD using MAE and standard deviation (SD). Legend: G = reference length; N = no explicit length control. Lower values indicate better control.</caption><thead><tr><th>Model</th><th>Type</th><th>MAE↓ ± SD</th></tr></thead><tbody><tr><td>BART-L</td><td>N</td><td>3.12 ± 3.3</td></tr><tr><td>RPE-BART-L</td><td>G</td><td>0.8 ± 3.6</td></tr><tr><td>PRE-BART-L</td><td>G</td><td>0.0 ± 0.1</td></tr></tbody></table>
<table><caption>Table 5: BLEU, ROUGE and BERTScore results on the SQuAD dataset.</caption><thead><tr><th colspan="4">SQuAD</th></tr><tr><th>Score</th><th>BART-L</th><th>RPE-BART-L</th><th>PRE-BART-L</th></tr></thead><tbody><tr><td>BLEU↑</td><td>16.7</td><td>16.4</td><td>18.6</td></tr><tr><td>R-1↑</td><td>54.4</td><td>52.1</td><td>55.3</td></tr><tr><td>R-2↑</td><td>32</td><td>30.1</td><td>32.9</td></tr><tr><td>R-L↑</td><td>50.1</td><td>47.7</td><td>50.8</td></tr><tr><td>B.S.↑</td><td>75.2</td><td>72.6</td><td>74.8</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 6: MAE by target-length bucket (1 tokens) for RPE-BART-L on SQuAD dataset.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 7: MAE by target-length bucket (2 tokens) for PRE-BART-L on SQuAD dataset.</figcaption></figure>

# 7 Conclusion

In this paper, we have introduced Progress Ratio Embeddings (PRE) as a novel and robust mechanism for length control in neural text generation. Unlike traditional Reverse Positional Embeddings (RPE), which rely on discrete countdown signals

and struggle with generalization beyond the training distribution, PRE uses continuous impatience signals to encode progress in a more scalable way. Our experiments across different tasks generation highlight that PRE significantly improves length controllability while maintaining or enhancing text generation quality, according to classic metrics. Furthermore, PRE enables generalization to out-of-distribution target lengths, reducing the proportion of outliers and mean absolute error compared to RPE. Beyond providing length control in abstractive summarization (Almohaimeed and Azmi, 2025), PRE opens up new directions for fine-grained control over the generation process, enabling more progressive summarization and helping to mitigate hallucinations while improving explainability.

# 8 Limitations

Several limitations can be addressed in our work. Let us briefly discuss them. First, we have only implemented our PRE method within an encoder-decoder architecture, which is not the prevailing trend of current decoder-only language models. A natural next step would be to evaluate the robustness of length control with PRE method on decoder-only models, as has been done for the RPE approach in previous work (Butcher et al., 2025). Moreover, length control is not only relevant for summarization and question generation tasks but many other test-to-text tasks could benefit from this capability to regulate the generated length. Finally, an interesting direction for future research would be to explore length control within *chain-of-thought* models. Being able to regulate the reasoning length could make such models more efficient at inference time, by preventing unnecessary over-generation and by allowing a deterministic adjustment of the reasoning depth.

# 9 Ethical Considerations

We have designed our methods to be simple to reproduce as they rely on publicly available pre-trained language models. We acknowledge that any text generation system carries potential risks of producing biased or misleading content. To mitigate these risks, we restricted our experiments to widely used, publicly datasets whose content is considered as relatively safe for research purposes. Our work complies with the EACL ETHICS POLICY a.

<footnote>a. https://www.aclweb.org/portal/content/acl-code-ethics</footnote>
# 10 LLM Usage Declaration

During the preparation of this manuscript, we used Large Language Models to assist with text clarity, grammar, and formulation, particularly in polishing the abstract and certain explanatory sentences. The scientific content, experimental design, results, and interpretations were entirely conceived and written by the authors. LLMs were not used to generate original ideas, proofs, or analyses; its contribution was limited to language refinement.

## References

Norah Almohaimeed and Aqil M. Azmi. 2025. *Abstractive text summarization: A comprehensive survey of techniques, systems, and challenges*. Computer Science Review, 57:100762.

Bradley Butcher, Michael O'Keefe, and James Titchener. 2025. *Precise length control for large language models*. Natural Language Processing Journal, 11:100143.

Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. *ReExamining System-Level Correlations of Automatic Summarization Evaluation Metrics*. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 6038–6052. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. *Bert: Pre-training of deep bidirectional transformers for language understanding*. Preprint, arXiv:1810.04805.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. *A survey on in-context learning*. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 1107–1128, Miami, Florida, USA. Association for Computational Linguistics.

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. *Convolutional sequence to sequence learning*. Preprint, arXiv:1705.03122.

Google. Google. https://research.google/. Accessed: 2025-09-22.

Yuxuan Gu, Wenjie Wang, Xiaocheng Feng, Weihong Zhong, Kun Zhu, Lei Huang, Tat-Seng Chua, and Bing Qin. 2024. *Length controlled generation for black-box llms*. Preprint, arXiv:2412.14656.

Renlong Jie, Xiaojun Meng, Lifeng Shang, Xin Jiang, and Qun Liu. 2024. *Prompt-Based Length Controlled Generation with Multiple Control Types*. In *Findings of the Association for Computational Linguistics* ACL 2024, pages 1067–1085. Association for Computational Linguistics.

Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. *Controlling Output Length in Neural Encoder-Decoders*. In *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pages 1328–1338. Association for Computational Linguistics.

Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. *An empirical survey on long document summarization: Datasets, models, and metrics*. ACM Computing Surveys, 55(8):1–35.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. *Bert: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension*. Preprint, arXiv:1910.13461.

Chin-Yew Lin. 2004. *ROUGE: A package for automatic evaluation of summaries*. In *Text Summarization Branches Out*, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Yizhu Liu, Qi Jia, and Kenny Zhu. 2022. *LAAM: Length Control in Abstractive Summarization by Pretraining Information Selection*. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 6885–6895. Association for Computational Linguistics.

Ilya Loshchilov and Frank Hutter. 2019. *Decoupled weight decay regularization*. Preprint, arXiv:1711.05101.

Renze Lou, Kai Zhang, and Wenpeng Yin. 2024. *Large language model instruction following: A survey of progresses and challenges*. Computational Linguistics, 50(3):1053–1095.

Meta AI. Meta ai research. https://ai.meta.com/. Accessed: 2025-09-22.

Lesly Miculicich, Yujia Xie, Song Wang, and Pengcheng He. 2023. *REPILOT: Summarization with Precise Length Control*. Preprint, arXiv:2305.05171.

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gülçehre, and Bing Xiang. 2016. *Abstractive text summarization using sequence-to-sequence RNNs and beyond*. In *Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning*, pages 280–290, Berlin, Germany. Association for Computational Linguistics.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. *Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization*. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. *Preprint*, arXiv:1910.10683.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. *Preprint*, arXiv:1606.05250.

C.E. Shannon. 1949. Communication in the presence of noise. *Proceedings of the IRE*, 37(1):10–21.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)*, pages 464–468, New Orleans, Louisiana. Association for Computational Linguistics.

Shlomo Stept. 2023. T5-large fine-tuned on xsum + cnn/dailymail for abstractive summarization.

Sho Takase and Naoaki Okazaki. 2019. Positional Encoding to Control Output Sequence Length. *Preprint*, arXiv:1904.07418.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc.

Zhongyi Yu, Zhenghao Wu, Hao Zheng, Zhe Xuan Yuan, Jefferson Fong, and Weifeng Su. 2021. LenAtten: An Effective Length Controlling Unit For Text Summarization. *Preprint*, arXiv:2106.00316.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. *Preprint*, arXiv:1904.09675.

Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2023. Explainability for large language models: A survey. *Preprint*, arXiv:2309.01029.

## A Appendix: Stress signal plot for varying pulsations

We use trigonometric functions to represent the concept of a periodic impatience signal as:

$$s: \mathbb{R}^{+} \times [0, 1] \rightarrow [-1, 1]^2 \\ (\omega, x) \mapsto (\cos(\omega x), \sin(\omega x)),$$

where $\omega$ denotes the pulsation of the periodic signal with a corresponding frequency $F = \frac{\omega}{2\pi}$.

<figure><img src="image_11.png" /><figcaption>Figure 8: Impatience signal curves plot for different pulsation $\omega$ based on $M = \frac{BART_{d_{model}}}{2} = 512$.</figcaption></figure>

## B Appendix: Illustration of the RPE method in relation to the decoder block input

Reverse Positional Embeddings (RPE) extend the standard sinusoidal formulation (Vaswani et al., 2017) by explicitly encoding the number of tokens remaining until the desired target length. Instead of encoding the absolute position $i$, RPE use the countdown index $(l-i)$, where $l$ is the target length.

$$RPE(i, 2k) = \sin\left(\frac{l-i}{10000^{2k/d_{model}}}\right), \quad (5)$$

$$RPE(i, 2k + 1) = \cos\left(\frac{l - i}{10000^{2k/d_{model}}}\right), \quad (6)$$

where $d_{model}$ is the embedding dimension and $k$ indexes the dimensions. This design injects a discrete countdown signal into the decoder, enabling the model to condition generation explicitly on the remaining length.
<figure><img src="image_12.png" /><figcaption>Figure 9: Illustration of Reverse Positional Embeddings (RPE) added to token and positional embeddings before being passed to the decoder block.</figcaption></figure>

## C Appendix: Summary lengths distribution through datasets

<figure><img src="image_13.png" /><figcaption>Figure 10: Data visualisation over the CNN/DailyMail summary length distribution.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 11: Data visualisation over the XSum summary length distribution.</figcaption></figure>

## D Appendix: Results of the PRE methode adaptation on the T5 encoder-decoder architecture

The T5-Large model (Raffel et al., 2023) treats every NLP task as text-to-text. Pre-trained on the C4 corpus with a span-corruption objective, it contains roughly 770M parameters and accepts inputs of up to 512 tokens. Developed by Google, this open-source model offers strong transferability across tasks and remains competitive for abstractive summarization. For this model $d_{model} = 512$.

<table><caption>Table 6: Length control evaluation on CNN/DailyMail and XSum using MAE and standard deviation (SD). **Legend:** G = guided by reference length; N = no explicit length control. Lower values indicate better control.</caption><thead><tr><th>Dataset</th><th>Model</th><th>Type</th><th>MAE<sup>↓</sup> ± SD</th></tr></thead><tbody><tr><td rowspan="2">CNN/DM</td><td>T5-L</td><td>N</td><td>20.1 ± 17.9</td></tr><tr><td>PRE-T5-L</td><td>G</td><td><b>7.9</b> ± 7.8</td></tr><tr><td rowspan="2">XSum</td><td>T5-L</td><td>N</td><td>6.5 ± 5.4</td></tr><tr><td>PRE-T5-L</td><td>G</td><td><b>1.9</b> ± 3.3</td></tr></tbody></table>

<table><caption>Table 7: ROUGE and BERTScore results on CNN/DailyMail and XSum for T5 models. **Legend:** G = length from the reference summary is given; N = no explicit length control.</caption><thead><tr><th colspan="6">CNN/DailyMail</th></tr><tr><th>Model</th><th>Type</th><th>R-1<sup>↑</sup></th><th>R-2<sup>↑</sup></th><th>R-L<sup>↑</sup></th><th>B.S.<sup>↑</sup></th></tr></thead><tbody><tr><td>T5-L (Raffel et al., 2023)</td><td>N</td><td>42.50</td><td>20.68</td><td>39.75</td><td>68.2</td></tr><tr><td>PRE-T5-L</td><td>G</td><td>44.6</td><td>21.0</td><td>36.0</td><td>69.4</td></tr></tbody><tbody><tr><th colspan="6">XSum</th></tr><tr><th>Model</th><th>Type</th><th>R-1<sup>↑</sup></th><th>R-2<sup>↑</sup></th><th>R-L<sup>↑</sup></th><th>B.S.<sup>↑</sup></th></tr><tr><td>T5-L (Stept, 2023)</td><td>N</td><td>36.77</td><td>14.69</td><td>30.06</td><td>68.9</td></tr><tr><td>PRE-T5-L</td><td>G</td><td>42.8</td><td>19.1</td><td>34.3</td><td>71.7</td></tr></tbody></table>

## E Appendix: Reproduction of the LAAM Method

The Length-Aware Attention Mechanism (LAAM) (Liu et al., 2022) modifies the cross-attention of a seq2seq model by boosting the top-$\ell_t$ attended encoder tokens according to the remaining target length. This encourages the encoder-decoder attention to select information compatible with the desired output size. In our reproduction, LAAM is applied only to the last decoder layer of BART and fine-tuned on CNN/DailyMail and on SQuAD. Below we report the full set of scores obtained from our implementation.
<table><caption>Table 8: LAAM performance on CNN/DailyMail.</caption><thead><tr><th>Stat</th><th>R-1</th><th>R-2</th><th>R-L</th><th>B.S.</th><th>MAE</th></tr></thead><tbody><tr><td>Mean</td><td>44.2</td><td>21.1</td><td>41.1</td><td>69.6</td><td>17.5</td></tr><tr><td>Std</td><td>12.5</td><td>13.6</td><td>12.6</td><td>6.9</td><td>16.2</td></tr><tr><td>Min</td><td>0.0</td><td>0.0</td><td>0.0</td><td>42.4</td><td>0.0</td></tr><tr><td>25%</td><td>36.0</td><td>11.6</td><td>32.7</td><td>65.4</td><td>5.0</td></tr><tr><td>50%</td><td>44.0</td><td>18.8</td><td>40.5</td><td>69.6</td><td>10.0</td></tr><tr><td>75%</td><td>51.9</td><td>27.9</td><td>48.5</td><td>74.0</td><td>21.0</td></tr></tbody></table>

<figure><img src="image_15.png" /><figcaption>Figure 12: MAE by target-length bucket (25 tokens) for LAAM-BART-L on CNN/DailyMail dataset.</figcaption></figure>

<table><caption>Table 9: LAAM performance on SQuAD.</caption><thead><tr><th>Stat</th><th>R-1</th><th>R-2</th><th>R-L</th><th>BLEU</th><th>B.S.</th><th>MAE</th></tr></thead><tbody><tr><td>Mean</td><td>54.7</td><td>32.1</td><td>50.2</td><td>16.8</td><td>75.4</td><td>3.12</td></tr><tr><td>Std</td><td>23.4</td><td>27.6</td><td>24.3</td><td>27.1</td><td>12.4</td><td>3.18</td></tr><tr><td>Min</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>38.9</td><td>0.0</td></tr><tr><td>25%</td><td>37.5</td><td>10.0</td><td>31.3</td><td>0.0</td><td>66.5</td><td>1.0</td></tr><tr><td>50%</td><td>54.5</td><td>26.7</td><td>47.7</td><td>0.0</td><td>75.1</td><td>2.0</td></tr><tr><td>75%</td><td>71.4</td><td>50.0</td><td>66.7</td><td>30.2</td><td>84.2</td><td>4.0</td></tr></tbody></table>

<figure><img src="image_16.png" /><figcaption>Figure 13: MAE by target-length bucket (2 tokens) for LAAM-BART-L on SQuAD dataset.</figcaption></figure>

# F Appendix: Statistical Significance of Length Fidelity

**Statistical Significance (Paired t-tests).** We performed paired Student's t-tests on example-level MAE values from the CNN/DailyMail test set to assess whether the improvements of PRE over existing baselines are statistically reliable.

* PRE-BART vs. BART baseline:
  $t = -98.95, p < 10^{-300}$

* PRE-BART vs. RPE-BART:
  $t = -19.83, p = 4.8 \times 10^{-86}$

These results demonstrate that PRE yields highly statistically significant improvements in length fidelity.

# G Appendix: Density comparison of summary length distributions

The PRE method produces summaries whose length distribution closely matches the true distribution observed in the reference data.
<figure><img src="image_17.png" /><figcaption>(a) Reference length distribution.</figcaption></figure>

BART-L-CNN generated token length distribution

<figure><img src="image_18.png" /><figcaption>(b) BART-L length distribution.</figcaption></figure>

PRE-BART-L-CNN generated token length distribution

<figure><img src="image_19.png" /><figcaption>(c) PRE-BART-L length distribution.</figcaption></figure>

<caption>Figure 14: Density comparison of summary length dis- tributions on CNN/DailyMail.</caption>