Decoupling The "What" and "Where" With Polar Coordinate Positional Embedding

## Abstract

Abstract The attention mechanism in a Transformer architecture matches key to query
based on both content—the what —and position
in a sequence—the where . We present an analysis indicating
that what and where are entangled in the popular
RoPE rotary position embedding. This entanglement can impair performance
particularly when decisions require independent matches
on these two factors. We propose an improvement to RoPE, which
we call Polar Coordinate Position Embeddings or PoPE , that
eliminates the what-where confound. PoPE is far superior on
a diagnostic task requiring indexing solely by position or
by content. On autoregressive sequence modeling in music,
genomic, and natural language domains, Transformers using PoPE
as the positional encoding scheme outperform baselines using
RoPE with respect to evaluation loss (perplexity) and downstream
task performance. On language modeling, these gains persist
across model scale, from 124M to 774M parameters.
Crucially, PoPE shows strong zero-shot length extrapolation capabilities, whereas RoPE’s performance degrades significantly on longer sequences at test time without fine tuning or the use
of position-interpolation methods.

## 1 Introduction

In the prehistory of deep learning, a key challenge was representing
sequential or position-coded data.
Tasks of interest included recognizing words from letter orderings (McClelland & Rumelhart, 1981), recognizing speech from power spectra (Waibel et al., 1989), compressing long sequences with time-lags between predictable events (Schmidhuber et al., 1993), classifying hand-drawn symbols from strokes (Yaeger, 1996), and forecasting time series from samples (Mozer, 1993).

Two approaches were common: recurrent neural nets (RNNs) and slot-based encodings.
RNNs are a means of encoding sequence position implicitly in the ordering of inputs.
By contrast, slot-based encodings partition an input vector into separate components for each sequence step.
A significant advantage of RNNs is their approximate
*translation equivariance*, meaning that shifting the absolute position of a sub-sequence in the input yields analogous shifts in the model state.
Slot-based representations do not share this property: learning to respond to content in one slot does not generalize to the same content in other slots.

The Transformer architecture (Vaswani et al., 2017) changed the game for slot-based representations.
Through its self-attention mechanism, the basic Transformer is not only translation equivariant but also translation and permutation *invariant* (subject to causal attention boundaries).
The challenge with Transformers thus became how to obtain translation
equivariance without losing sensitivity to the relative positions of input elements.
The solutions that emerged involve enriching latent representations to encode not only their content—the *what*—but also the relationship between their sequence positions—the *where* (Vaswani et al., 2017; Dai et al., 2019; Su et al., 2024).
These solutions rightly assume that both content and position
are essential to modeling complex sequences like language.

In this article, we argue that the Rotary Position Embedding (RoPE) (Su et al., 2024),
a widely adopted solution, entangles the what and where in a way that can impair model
performance, particularly when
decision making requires independent matches on these two factors.
We propose an alternative technique, which we call PoPE, that shares the advantages
of RoPE over other positional encodings while allowing the Transformer
to implement rules in which the key-query match can be characterized as a
conjunction of a what match and a where match. Although PoPE is
only a minor modification of RoPE, it introduces a powerful inductive bias that
improves data efficiency, asymptotic accuracy, and yields superior
context-length generalization.

## 2 Background

RoPE (Su et al., 2024) is the dominant approach to incorporate positional information in many frontier language models, including Llama 3 (Grattafiori et al., 2024), DeepSeekv3 (Liu et al., 2024), Gemma 3 (Team et al., 2025), and Qwen3 (Yang et al., 2025).
It produces an attention score for each query-key pair that is based on both how well they match and their relative positions in the input sequence.

To explain RoPE, consider a specific attention head in a specific layer which performs a match between a query in position $t$, denoted $\bm{q}_{t}$, and a key in position $s$, denoted $\bm{k}_{s}$. (If it helps to remember notation, $t$ and $s$ could denote target and source of attention, respectively.)
The key and query are
$d$-dimensional vectors that are partitioned into $d/2$ two-dimensional components.
We denote component $c\in\{1,\ldots,d/2\}$ of query and key by $\bm{q}_{tc}$ and $\bm{k}_{sc}$, respectively.
RoPE first rotates each component $c$ in the 2D plane by an angle proportional to the key and query positions.
If $\bm{R}(\phi)$ is a $2\times 2$ matrix that performs a rotation by angle $\phi$, the rotated query and key are $\bm{R}(t\theta_{c})\,\bm{q}_{tc}$ and $\bm{R}(s\theta_{c})\,\bm{k}_{sc}$, where $\theta_{c}$ is a component-specific , i.e. $\theta_{c}=\theta^{-2(c-1)/d}:c=1,\ldots,d/2$ and $\theta$ is called the base wavelength.
The formation of query (or key) components and their rotation in 2D are depicted on the left side of [Figure 1](https://arxiv.org/html/2509.10534v1#S3.F1).

The corresponding components of key and query are matched via a dot product and summed to obtain an attention score:

$$ $a_{ts}^{\text{RoPE}}=\sum_{c=1}^{d/2}\left[\bm{R}(t\theta_{c})\,\bm{q}_{tc}\right]^{\textsc{t}}\left[\bm{R}(s\theta_{c})\,\bm{k}_{sc}\right]=\sum_{c=1}^{d/2}\bm{q}_{tc}^{\textsc{t}}~\bm{R}((s-t)\theta_{c})~\bm{k}_{sc}~.$ (1) $$

The rotation to bring components into alignment depends only on the relative positions of the key and query, not their absolute positions.

Our perspective on RoPE begins by re-expressing the key and query components from Cartesian to
polar coordinates,
$\bm{k}_{sc}\Leftrightarrow(\mu_{k_{sc}},\phi_{k_{sc}})$ and
$\bm{q}_{tc}\Leftrightarrow(\mu_{q_{tc}},\phi_{q_{tc}})$, where

$$ $\bm{k}_{sc}=\bm{R}(\phi_{k_{sc}})\begin{bmatrix}\mu_{k_{sc}}\\ 0\end{bmatrix}~~\text{and}~~\bm{q}_{tc}=\bm{R}(\phi_{q_{tc}})\begin{bmatrix}\mu_{q_{tc}}\\ 0\end{bmatrix}.$ $$

With this notation, we can compose the rotations and write the
attention score as

$$ $\displaystyle a_{ts}^{\text{RoPE}}$ $\displaystyle=\sum_{c=1}^{d/2}\begin{bmatrix}\mu_{q_{tc}}&0\end{bmatrix}\bm{R}((s-t)\theta_{c}-\phi_{q_{tc}}+\phi_{k_{sc}})\begin{bmatrix}\mu_{k_{sc}}\\ 0\end{bmatrix}$ $\displaystyle=\sum_{c=1}^{d/2}\mu_{q_{tc}}\mu_{k_{sc}}\cos((s-t)\theta_{c}+\phi_{k_{sc}}-\phi_{q_{tc}}).$ (2) $$

This algebra makes clear that each two-element component of the embedding
is transformed into a single magnitude and also introduces, via $\phi_{q_{tc}}$ and $\phi_{k_{sc}}$, an
adjustment to the relative position (phase) that yields the maximal response.
Thus, both the key and the query confound information
about the presence or absence of features (the ‘what’)
and relative positions (the ‘where’). Our hypothesis is that we
can improve model performance by disentangling these two distinct sorts
of information, specifically by removing the interaction term $\phi_{k_{sc}}-\phi_{q_{tc}}$.

## 3 Method

Figure: Figure 1: Illustration compares how RoPE and PoPE encode relative positions via rotations of queries. Left: Three complex-valued RoPE components having magnitudes $\mu_{q_{tc}}$ (black arrows) and initial phases $\phi_{{q}_{tc}}$ (green arcs) are constructed from three pairs of embedding features (orange arrows) of the query vector $\bm{q}_{t}$ (gray box) at sequence position $t$. These RoPE components are then rotated by angles $t\theta_{c}$ (blue arcs). Right: Three magnitude components $\mu_{q_{tc}}$ (black arrows) of complex-valued PoPE components are constructed from three embedding features of the query vector $\bm{q}_{t}$ (gray box) by applying softplus activation. These magnitudes (complex numbers with zero phases) are then rotated by angles $t\theta_{c}$ (blue arcs). PoPE uses twice the number of components than RoPE as it applies rotations to each component of the query vector $\bm{q}_{t}$.
Refer to caption: x1.png

In RoPE, we interpreted the $d/2$ components of the key and query as complex numbers. In the method we propose, we utilize an alternative form of polar-coordinate representation of key and query.
We refer to our method as *PoPE*, for *Polar Coordinate Positional Embedding*.
In PoPE, we transform the key and query into corresponding $d$-element complex vectors, which we refer to as
$\tilde{\bm{k}}_{s}$ and $\tilde{\bm{q}}_{t}$.
In each, the magnitude of element $c\in\{1,\ldots,d\}$ is a rescaling of the corresponding element of the original real-valued key or query:

$$ $\mu_{\tilde{k}_{sc}}=\sigma(k_{sc})\quad\text{and}\quad\mu_{\tilde{q}_{tc}}=\sigma(q_{tc}),$ (3) $$

where $\sigma(x)=\text{ln}(1+e^{x})$ denotes the softplus
activation function.
The softplus ensures the $\mu$ are non-negative, permitting them to be interpreted as magnitudes.
The phases are position dependent:

$$ $\phi_{\tilde{k}_{sc}}=s\theta_{c}\quad\text{and}\quad\phi_{\tilde{q}_{tc}}=t\theta_{c},$ (4) $$

where $\theta_{c}$ is a component-specific frequency, i.e. $\theta_{c}=\theta^{(c-1)/d}:c=1,\ldots,d$.
With this definition of the key $\tilde{\bm{k}}_{s}\in\mathbb{C}^{d}$ and query $\tilde{\bm{q}}_{t}\in\mathbb{C}^{d}$, the attention score can elegantly be defined as:

$$ $\bm{a}_{ts}^{\text{PoPE}}=\Re\left[{\tilde{\bm{q}}}_{t}^{H}~{\tilde{\bm{k}}}_{s}\right]=\Re\left[\sum_{c=1}^{d}\tilde{\bm{q}}^{H}_{tc}~\tilde{\bm{k}}_{sc}\right]=\sum_{c=1}^{d}\mu_{\tilde{q}_{tc}}\mu_{\tilde{k}_{sc}}\cos((s-t)\theta_{c}),$ (5) $$

where $H$ denotes the conjugate transpose which involves applying complex conjugation to each component.
The PoPE attention score (Equation [5](https://arxiv.org/html/2509.10534v1#S3.E5)) is quite similar in form to the RoPE attention score (Equation [2](https://arxiv.org/html/2509.10534v1#S2.E2)), except that:
i) $c$ is an index over individual elements of the key and query and not over pairs of elements, thereby doubling the number of frequencies from $d/2$ to $d$, and ii) the interaction term causing key
and query to influence phase has been eliminated.

Extending this formulation, each attention head might benefit by introducing a learnable but fixed bias term to replace the RoPE interaction term:

$$ $\bm{a}_{ts}^{\text{PoPE}}=\sum_{c=1}^{d}\mu_{\tilde{q}_{tc}}\mu_{\tilde{k}_{sc}}\cos((s-t)\theta_{c}+\delta_{c}),$ (6) $$

where $\delta_{c}\in\mathbb{R}$ is a learnable bias that tunes the optimal relative offset for each frequency $c$.
[Figure 1](https://arxiv.org/html/2509.10534v1#S3.F1) visualizes the different ways by which RoPE and PoPE encode relative positions as rotations applied to queries and keys.
There are several ways to initialize the bias terms $\delta_{c}$.
We find that two good options are to initialize it either with $\delta_{c}=0$ or $\delta_{c}\sim\text{Uniform}(-2\pi,0)$.
Further, we bound $\delta_{c}$ so that it to always lies in the interval $[-2\pi,0]$, i.e. $\theta_{c}=\text{min}(\text{max}(\theta_{c},-2\pi),0)$ and found this improves stability.
We find that the zero initialization is important for length generalization while the uniform one gives slightly better in-distribution performance.

#### Efficient Implementation.

We implemented PoPE using Triton, starting from the example code for Flash Attention 2 (Dao, 2024).(^1^11[https://triton-lang.org/main/getting-started/tutorials/06-fused-attention.html](https://triton-lang.org/main/getting-started/tutorials/06-fused-attention.html))
We modify the kernel to take complex-valued keys and queries in Cartesian form and compute the real part of their product inside the kernel, without ever materializing the resulting complex matrix of the query-key dot product.
We can compute the real and imaginary components of the complex-valued $\tilde{q}_{tc}$ from its polar form as:

$$ $x_{\tilde{q}_{tc}}=\mu_{\tilde{q}_{tc}}\cos(\phi_{\tilde{q}_{tc}})\quad\text{and}\quad y_{\tilde{q}_{tc}}=\mu_{\tilde{q}_{tc}}\sin(\phi_{\tilde{q}_{tc}}),$ (7) $$

where $x_{\tilde{q}_{tc}}$ and $y_{\tilde{q}_{tc}}$ denote the real and imaginary components of a complex-valued query feature $\tilde{q}_{tc}\in\mathbb{C}$.
Similarly, we can obtain the real and imaginary components of $\tilde{k}_{sc}$ also additionally accounting for the learnable phase-shifts $\delta_{c}$ as:

$$ $x_{\tilde{k}_{sc}}=\mu_{\tilde{k}_{sc}}\cos(\phi_{\tilde{k}_{sc}}+\delta_{c})\quad\text{and}\quad y_{\tilde{k}_{sc}}=\mu_{\tilde{k}_{sc}}\sin(\phi_{\tilde{k}_{sc}}+\delta_{c}),$ (8) $$

Then, note that the multiplication of complex numbers in Cartesian form is:

$$ $\tilde{q}_{tc}^{H}\tilde{k}_{sc}=(x_{\tilde{q}_{tc}}-iy_{\tilde{q}_{tc}})(x_{\tilde{k}_{sc}}+iy_{\tilde{k}_{sc}})=x_{\tilde{q}_{tc}}x_{\tilde{k}_{sc}}-i^{2}y_{\tilde{q}_{tc}}y_{\tilde{k}_{sc}}+i(x_{\tilde{q}_{tc}}y_{\tilde{k}_{sc}}-y_{\tilde{q}_{tc}}x_{\tilde{k}_{sc}})$ (9) $$

The efficient computation of the attention score for complex-valued query $\bm{\tilde{q}}_{t}$ and key $\bm{\tilde{k}}_{s}$ is:

$$ $\bm{a}_{ts}^{\text{PoPE}}=\Re\left[\bm{\tilde{q}}_{t}^{H}\bm{\tilde{k}}_{s}\right]=\sum_{c=1}^{d}x_{\tilde{q}_{tc}}x_{\tilde{k}_{sc}}+y_{\tilde{q}_{tc}}y_{\tilde{k}_{sc}}$ (10) $$

Thus, our custom Flash Attention for PoPE requires only a single additional multiplication compared to the standard one.
However, it requires twice the memory usage and bandwidth to store and load the complex-valued keys and values from the global memory.
It is possible to avoid this memory overhead by loading the magnitudes of the keys and queries directly and performing the rotation inside the kernel.
Now, the only overhead is the additional multiplication.
We chose to go with the slower, but general variant, which takes as arguments complex-valued keys and queries in Cartesian form and not perform the memory optimization, since this would prevent us from easy prototyping of different PoPE variants.

## 4 Results

In all experiments, we compare our method, PoPE, to the popular RoPE (Su et al., 2024) scheme using two Transformers with identical model and training hyperparameters, the only difference being their positional encoding schemes.
For more details on how the datasets are generated, preprocessed, and tokenized, refer to [Section A.1](https://arxiv.org/html/2509.10534v1#A1.SS1).
In all experiments, we use a decoder-only Transformer architecture (Vaswani et al., 2017; Radford et al., 2018) with causal masking for autoregressive sequence modeling.
The only change applied is the use of RMSNorm (Zhang & Sennrich, 2019) instead of LayerNorm (Ba et al., 2016) for normalization, as is common in the latest frontier models.
For more details on the model configuration and training hyperparameters used for each experiment refer to [Section A.2](https://arxiv.org/html/2509.10534v1#A1.SS2) and [Section A.3](https://arxiv.org/html/2509.10534v1#A1.SS3) respectively.

#### Indirect Indexing.

We introduce a task that requires identifying a target character within a variable-length source string. The target is defined to be at a specified relative offset (left or right) from a specified source character.
For example, in the input QEOHoUbKfeSrMVNlCzXu, z, -3, N; the target N is three places to the left of source z in the string. Predicting the final (target) character of this sequence
requires models to learn to independently manipulate the content and positional information of tokens and to apply pointer arithmetic operations.

**Table 1: Accuracy (with standard deviation) on the test split for the Indirect Indexing task.**
| Positional Enc. | Indirect Idx. |
| --- | --- |
| RoPE | 11.16 $\pm$ 2.45 |
| PoPE | 94.82 $\pm$ 2.91 |

The dataset is constructed by procedurally generating examples of source strings, source symbols, and relative shifts.
We compare RoPE (Su et al., 2024) against PoPE by training two Transformer models with cross-entropy loss applied only on the final (target) token and evaluated on the accuracy of final token.
[Table 1](https://arxiv.org/html/2509.10534v1#S4.T1) shows the mean and standard deviation (3 seeds) in final token accuracy on the test set.
While RoPE struggles to learn this task and reaches an average accuracy of just 11%, our method PoPE solves the task almost perfectly, reaching an average accuracy of nearly 95%.
This result highlights the difficulty that RoPE has in teasing apart ‘what’ and ‘where’ information, i.e., identifying the position of certain content and determining the content at a certain relative position.
In contrast, PoPE efficiently learns a generalizable solution for the task, presumably because ’what’ and ’where’
information can be disentangled in key-query matching.

Next, we test our method on sequence modeling in the domains of music and genomic data.
Like human language, both domains have a hierarchical organization and are rule governed systems.
In contrast to human language, structural rules can be quite strict and precise positional information is critical.
Musical pieces contain hierarchical repeating structures (chords, phrases, motifs), where relative pitch and timing changes are more predictive than their absolute values (Huang et al., 2019).
Huang et al. (2019) also note that the characteristic grammar in piano gestures relies more on relative intervals.
Likewise, genomic sequences contain local patterns that rely on relative position and ordering of elements.

**Table 2: Best NLL on the test split for Transformer models with RoPE or PoPE positional encodings on music datasets (JSB and MAESTRO).**
| Positional Enc. | JSB | MAESTRO |
| --- | --- | --- |
| RoPE | 0.5081 | 1.501 |
| PoPE | 0.4889 | 1.486 |

#### Sequence modeling of symbolic music.

We train Transformer models using cross-entropy loss on MIDI-based inputs with a maximum length of 2048 from two popular music datasets, Bach-Chorales (JSB) (Boulanger-Lewandowski et al., 2012) and MAESTRO (Hawthorne et al., 2019).
We closely follow the preprocessing steps from Huang et al. (2019).
[Table 2](https://arxiv.org/html/2509.10534v1#S4.T2) reveals that PoPE achieves a decrease in negative log likelihood (NLL) compared to RoPE on both datasets.

**Table 3: Best NLL on the test split for the Human Reference Genome dataset.**
| Positional Enc. | Human Ref. Genome |
| --- | --- |
| RoPE | 4.217 |
| PoPE | 4.152 |

#### Sequence modeling of human genome.

We train a Transformer on sequences from the Human Reference Genome dataset (Dalla-Torre et al., 2025) using the standard next-token prediction loss.
We follow the preprocessing and tokenization procedure from the recent state-of-the-art Nucleotide Transformer (Dalla-Torre et al., 2025) to obtain sequences with a maximum length of 1000 tokens and vocabulary size of 4107.
Our PoPE-based model achieves a significant drop in NLL compared to the baseline RoPE model ([Table 3](https://arxiv.org/html/2509.10534v1#S4.T3)).

Although we find a benefit of incorporating PoPE into Transformers on diverse domains like music
and genomic sequences, we chose these domains specifically because they appear to require the separation of position and
content as well as precise positional information.
It is much less clear that these properties hold true for human language.
In our next set of experiments, we pretrain Transformers of varying size on OpenWebText.

**Table 4: Perplexity on the validation split of OpenWebText for Transformer models with RoPE or PoPE positional encodings.**
| Positional Enc. | 124M | 253M | 774M |
| --- | --- | --- | --- |
| RoPE | 21.55 | 18.88 | 15.85 |
| PoPE | 21.33 | 18.55 | 15.45 |

#### Language modeling on OpenWebText.

We test PoPE’s efficacy on language modeling by training Transformers of three sizes on the OpenWebText dataset (Gokaslan & Cohen, 2019).
At respective model sizes, both models use identical architecture and training parameters but differ only in the positional encoding.
Across model sizes, PoPE consistently achieves lower perplexities than RoPE ([Table 4](https://arxiv.org/html/2509.10534v1#S4.T4)).
It is also interesting to note that the performance gap between RoPE and PoPE holds steady or possibly increases with model size.
We run ablation experiments by training the 124M model with PoPE variants that do not use either the softplus activation, $\sigma()$, or the learnable bias vector $\bm{\delta}$ and find these components each contribute to the performance gains (refer to [Appendix B](https://arxiv.org/html/2509.10534v1#A2) for more details).
Next, we move beyond perplexity to evaluate model efficacy on a standard suite of downstream tasks.

Table: Table 5: Zero-shot performance on downstream tasks using Transformer models pretrained on OpenWebText with RoPE or PoPE positional encoding.

#### Zero-shot performance on downstream tasks.

We evaluate the zero-shot performance of the Transformers pretrained on OpenWebText on six downstream tasks, namely: LAMBADA Paperno et al. (2016), BLiMP (Warstadt et al., 2020), Children’s Book Test (CBT) (Hill et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), and ARC-E (Clark et al., 2018).
Following Gao et al. (2024), we use the detokenized version from OpenAI for LAMBADA and evaluate the top-one accuracy on the last word (which can be multiple tokens; we use greedy decoding).
For CBT and BLiMP, we measure the accuracy for each task and report the average accuracy over all tasks.
[Table 5](https://arxiv.org/html/2509.10534v1#S4.T5) reports accuracy for three models sizes and for each of the six downstream tasks. The last column of the Table presents the mean accuracy across the tasks.
For all three model sizes, PoPE-based Transformers have higher mean accuracy than RoPE-based Transformers.

Figure: Figure 2: Length extrapolation at test-time on PG-19 dataset for different model sizes. We evaluate the models using RoPE (red) and PoPE (green) without any fine-tuning or position interpolation techniques at test-time on longer sequences (multiples of 1024 up to 10240).
Refer to caption: x2.png

#### Test-time length extrapolation.

We measure the ability of PoPE to generalize to test-time sequences that are longer than those presented during training.
We examine models pretrained on OpenWebText using a sequence length (context window) of 1024 tokens and assess zero-shot perplexity on much longer sequences (up to 10240 tokens) from the test split of the PG-19 dataset (Rae et al., 2020).
PoPE shows strong out-of-the-box length extrapolation capabilities without any fine-tuning or position interpolation techniques at test time on 10x longer sequences ([Figure 2](https://arxiv.org/html/2509.10534v1#S4.F2)).
In stark contrast, RoPE’s performance significantly degrades on longer sequences at test time.
It is also interesting to note that RoPE’s extrapolation performance degrades with model size,
PoPE’s extrapolation performance remains largely stable.
RoPE’s failure is due to its allowance of a what-where interaction: aspects of the key
and query representation can dynamically shift the position tuning of a component.
These shifts become problematic, especially for the lowest frequency components,
as they exert their influence only when the context window is expanded.

Figure: Figure 3: 2-norm plotted over 2D RoPE ‘chunks’ of queries (left) and keys (right) in each layer of the 124M Transformer over different RoPE frequencies. Mean over 10 different Shakespeare sonnets and 12 attention heads at each layer.
Refer to caption: x3.png

Figure: Figure 4: Magnitude of each complex-valued features of queries (left) and keys (right) in each layer of the 124M Transformer over different PoPE frequencies. Mean over 10 different Shakespeare sonnets and 12 attention heads at each layer.
Refer to caption: x4.png

#### Frequency usage analysis.

Barbero et al. (2025) analyze RoPE’s use of features at different period lengths by plotting the mean norm over the 2D RoPE components for all layers.
They observe that the Gemma 7B model prefers to maintain low norms on the
high-frequency channels to minimize interference, because the contribution of these
channels to the attention dot product behaves similar to random noise.
Gemma has high norms only for features in a sparse set of low frequencies.
We apply their analysis on our 124M Transformer models pretrained on OpenWebText (from [Table 4](https://arxiv.org/html/2509.10534v1#S4.T4)).
In our 124M Transformer, we similarly find that RoPE produces high norms for only a sparse
subset of frequency channels across all layers
([Figure 3](https://arxiv.org/html/2509.10534v1#S4.F3)).
While there are qualitative differences in Barbero et al. (2025) visualizations of Gemma 7B
and our 124M RoPE baseline model, we note that the pretraining and model scales are
vastly different, which might explain the different behaviors they’ve extracted from
training data.
In contrast, the PoPE Transformer assigns high norms on the high frequency features across all layers except the first ([Figure 4](https://arxiv.org/html/2509.10534v1#S4.F4)).
PoPE also shows a more distributed usage of features across the full frequency range compared to RoPE.
Notice the doubling in the number of frequencies (rows of the heatmap) with
PoPE ([Figure 4](https://arxiv.org/html/2509.10534v1#S4.F4)) versus RoPE ([Figure 3](https://arxiv.org/html/2509.10534v1#S4.F3)).

## 5 Related Work

#### RoPE and its extensions.

The most popular positional encoding used by current LLMs (Touvron et al., 2023; Grattafiori et al., 2024; Liu et al., 2024; Team et al., 2025; Yang et al., 2025) is RoPE (Su et al., 2024).
However, RoPE is known to be unable to generalize to longer sequences than it was trained on.
Pretraining on long sequences is expensive; thus, the models are typically trained on relatively short context lengths, and then extended to longer context lengths during post-training.
This can be done, for example, by changing the rotation frequency for each position so that the total amount of rotations for each component stays the same at the maximum context length as during pretraining (Chen et al., 2023).
A more performant alternative is YaRN (Peng et al., 2024) that scales higher frequencies less than the lower ones, retaining the ability of the model to recall small relative distances precisely.
Ding et al. (2024) shows that this strategy can be further improved by skipping the scaling for early token positions and searching for more optimal scaling actors using a genetic algorithm.
Furthermore, they perform iterative scaling followed by tuning phases, which allows them to expand the context window to 2M tokens. Sun et al. (2023) takes a different approach: the authors add a decay factor similar to ALiBi (Press et al., 2022) and block-wise masking to limit the span of the attention during inference.
Although this prevents a sudden performance drop in pure RoPE, it does not allow us to recall information from long distances.
Wang et al. (2024) proposes rounding all RoPE wavelengths to integers, such that there is no increasing shift in the rotation angle after each time the given channel wraps around.
This further improves the performance of YaRN, but is also effective without length extension techniques.

#### Alternative positional embeddings.

The original Transformer (Vaswani et al., 2017) uses sinusoidal embeddings to encode absolute positions; which are added to the token embeddings only at the input layer.
The sinusoidal positional embeddings also generally underperform w.r.t more modern methods for relative positional embeddings (Su et al., 2024) which inject this information at every layer.
It has been shown that autoregressive Transformers do not require explicit encoding of positional information to operate (Irie et al., 2019; Irie, 2025).
This is also somewhat reminiscent of fast-weight programmers (FWPs) (Schmidhuber, 1992), which are now known as unnormalized linear Transformers that also did not use explicit positional encoding.
Such Transformers tend to have better length extrapolation properties than both absolute and relative positional embeddings (Kazemnejad et al., 2023), although this comes at the expense of their in-distribution performance.
Shaw et al. (2018) introduced a relative positional embedding which uses a separate set of keys that are selected based on the distance of the key and query.
Music Transformer (Huang et al., 2019) proposed a more efficient implementation of this relative positional embedding. Dai et al. (2019) propose a different variant, where instead of learning a separate key for each offset, they use a learned projection of sinusoidal “offset encodings”, which are inspired by the absolute positional encodings.
This approach generalizes better to longer lengths in practice.
Additionally, the authors train sequentially within documents and allow attention to the previous batch, enhancing the long-range capabilities of the models further.
Wang et al. (2020) generalized this idea of “offset encodings” by using complex-valued embeddings of inputs to encode information about the content, global position and their order relationships within the sequence.
T5 (Raffel et al., 2020) takes a different approach: their method groups token pairs in log-sized buckets based on their distance, and it adds a learned per-bucket bias to the attention logit, allowing it to decrease the importance of far-away context.
ALiBi Press et al. (2022) takes inspiration from this, adding learned scores to the attention matrix that decay with relative distance.
Geometric (Csordás et al., 2022) or stick-breaking attention (Tan et al., 2025) take a radically different approach: it replaces the softmax activation function with a stick-breaking process that gives priority to good matches that are nearby without explicitly encoding positions or offsets. The authors claim superior length generalization.

## 6 Conclusion

We proposed a new relative positional encoding technique called PoPE whose
query-key attention scores are based on a computation that decouples
the match based on content and the match based on position.
In contrast, RoPE confounds the ‘what’ and ‘where’ information between keys and queries which leads to difficulties in
learning to match based solely on ’what’ or ’where’, as we highlight via a diagnostic task we introduce called Indirect Indexing.
A RoPE-based Transformer struggles to solve this task whereas
a PoPE-based Transformer learns this task easily.
On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to training loss (perplexity) and downstream task performance.
On language modeling, these gains persist across model scale,
from 124M to 774M parameters.
Crucially, PoPE shows strong zero-shot length extrapolation capabilities, whereas RoPE’s performance degrades significantly on longer sequences at test time and requires fine tuning or position interpolation methods.

## References

- Ba et al. (2016)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization.
*arXiv preprint arXiv:1607.06450*, 2016.
- Barbero et al. (2025)
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković.
Round and round we go! What makes rotary positional encodings useful?
In *Int. Conf. on Learning Representations (ICLR)*, 2025.
- Bisk et al. (2020)
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.
Piqa: Reasoning about physical commonsense in natural language.
In *Proc. AAAI Conf. on Artificial Intelligence*, 2020.
- Boulanger-Lewandowski et al. (2012)
Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent.
Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription.
In *Proc. Int. Conf. on Machine Learning (ICML)*, 2012.
- Chen et al. (2023)
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.
Extending context window of large language models via positional interpolation.
*arXiv preprint arXiv:2306.15595*, 2023.
- Clark et al. (2018)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
*ArXiv*, abs/1803.05457, 2018.
- Csordás et al. (2022)
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber.
The neural data router: Adaptive control flow in transformers improves systematic generalization.
In *Int. Conf. on Learning Representations (ICLR)*, 2022.
- Dai et al. (2019)
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov.
Transformer-xl: Attentive language models beyond a fixed-length context.
In *Proc. Association for Computational Linguistics (ACL)*, 2019.
- Dalla-Torre et al. (2025)
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P. de Almeida, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot.
Nucleotide transformer: building and evaluating robust foundation models for human genomics.
*Nature Methods*, 22(2):287–297, 2025.
- Dao (2024)
Tri Dao.
Flashattention-2: Faster attention with better parallelism and work partitioning.
In *Int. Conf. on Learning Representations (ICLR)*, 2024.
- Ding et al. (2024)
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang.
Longrope: Extending LLM context window beyond 2 million tokens.
In *Proc. Int. Conf. on Machine Learning (ICML)*, 2024.
- Gao et al. (2024)
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.
The language model evaluation harness, 07 2024.
URL [https://zenodo.org/records/12608602](https://zenodo.org/records/12608602).
- Gokaslan & Cohen (2019)
Aaron Gokaslan and Vanya Cohen.
Openwebtext corpus.
[http://Skylion007.github.io/OpenWebTextCorpus](http://Skylion007.github.io/OpenWebTextCorpus), 2019.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Hawthorne et al. (2019)
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck.
Enabling factorized piano music modeling and generation with the MAESTRO dataset.
In *Int. Conf. on Learning Representations (ICLR)*, 2019.
- Hill et al. (2016)
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston.
The goldilocks principle: Reading children’s books with explicit memory representations.
In *Int. Conf. on Learning Representations (ICLR)*, 2016.
- Huang et al. (2019)
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, and Douglas Eck.
Music transformer: Generating music with long-term structure.
In *Int. Conf. on Learning Representations (ICLR)*, 2019.
- Huang & Yang (2020)
Yu-Siang Huang and Yi-Hsuan Yang.
Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions.
In *Proc. ACM Int. Conf. on Multimedia (MM)*, 2020.
- Irie (2025)
Kazuki Irie.
Why are positional encodings nonessential for deep autoregressive transformers? revisiting a petroglyph.
*Preprint arXiv:2501.00659*, 2025.
- Irie et al. (2019)
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney.
Language modeling with deep Transformers.
In *Proc. Interspeech*, 2019.
- Kazemnejad et al. (2023)
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy.
The impact of positional encoding on length generalization in transformers.
In *Proc. Advances in Neural Information Processing Systems (NeurIPS)*, 2023.
- Liu et al. (2024)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- McClelland & Rumelhart (1981)
James L McClelland and David E Rumelhart.
An interactive activation model of context effects in letter perception: I. an account of basic findings.
*Psychological Review*, 88(5):375–407, 1981.
- Mozer (1993)
Michael C. Mozer.
Neural network architectures for temporal pattern processing.
In Andreas S. Weigand and Neil A. Gershenfeld (eds.), *Time Series Prediction: Forecasting the Future and Understanding the Past*, volume XVII of *Sante Fe Institute Studies in the Sciences of Complexity*, pp.  243–264, Redwood City, CA, 1993. Addison-Wesley Publishing.
- Paperno et al. (2016)
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.
The LAMBADA dataset: Word prediction requiring a broad discourse context.
In *Proc. Association for Computational Linguistics (ACL)*, 2016.
- Peng et al. (2024)
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.
YaRN: Efficient context window extension of large language models.
In *Int. Conf. on Learning Representations (ICLR)*, 2024.
- Press et al. (2022)
Ofir Press, Noah A. Smith, and Mike Lewis.
Train short, test long: Attention with linear biases enables input length extrapolation.
In *Int. Conf. on Learning Representations (ICLR)*, 2022.
- Radford et al. (2018)
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.
Improving language understanding by generative pre-training.
Technical report, OpenAI, 2018.
- Radford et al. (2019)
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners.
In *Proc. Advances in Neural Information Processing Systems (NeurIPS)*, 2019.
- Rae et al. (2020)
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap.
Compressive transformers for long-range sequence modelling.
In *Int. Conf. on Learning Representations (ICLR)*, 2020.
- Raffel et al. (2020)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
Exploring the limits of transfer learning with a unified text-to-text transformer.
*Journal of Machine Learning Research (JMLR)*, 21:140:1–140:67, 2020.
- Schmidhuber et al. (1993)
J Schmidhuber, MC Mozer, and D Prelinger.
Continuous history compression.
In *Proc. of Intl. Workshop on Neural Networks*, pp.  87–95, 1993.
- Schmidhuber (1992)
Jürgen Schmidhuber.
Learning to control fast-weight memories: An alternative to recurrent nets.
*Neural Computation*, 4(1):131–139, 1992.
- Shaw et al. (2018)
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.
Self-attention with relative position representations.
In *Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL-HLT)*, 2018.
- Su et al. (2024)
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
*Neurocomputing*, 568, 2024.
- Sun et al. (2023)
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei.
A length-extrapolatable transformer.
In *Proc. Association for Computational Linguistics (ACL)*, 2023.
- Tan et al. (2025)
Shawn Tan, Songlin Yang, Aaron C. Courville, Rameswar Panda, and Yikang Shen.
Scaling stick-breaking attention: An efficient implementation and in-depth study.
In *Int. Conf. on Learning Representations (ICLR)*, 2025.
- Team et al. (2025)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
*arXiv preprint arXiv:2503.19786*, 2025.
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
LLaMA: Open and efficient foundation language models.
*Preprint arXiv:2302.13971*, 2023.
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In *Proc. Advances in Neural Information Processing Systems (NIPS)*, 2017.
- Waibel et al. (1989)
A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K.J. Lang.
Phoneme recognition using time-delay neural networks.
*IEEE Transactions on Acoustics, Speech, and Signal Processing*, 37(3):328–339, 1989.
- Wang et al. (2020)
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen.
Encoding word order in complex embeddings.
In *Int. Conf. on Learning Representations (ICLR)*, 2020.
- Wang et al. (2024)
Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu.
Resonance rope: Improving context length generalization of large language models.
In *Proc. Association for Computational Linguistics (ACL)*, 2024.
- Warstadt et al. (2020)
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman.
BLiMP: The benchmark of linguistic minimal pairs for English.
*Transactions of the Association for Computational Linguistics (TACL)*, 8:377–392, 2020.
- Yaeger (1996)
L. Yaeger.
Neural networks provide robust character recognition for newton pdas.
*IEEE Expert*, 11(4):10–11, 1996.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Zellers et al. (2019)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
HellaSwag: Can a machine really finish your sentence?
In *Proc. Association for Computational Linguistics (ACL)*, 2019.
- Zhang & Sennrich (2019)
Biao Zhang and Rico Sennrich.
Root mean square layer normalization.
In *Proc. Advances in Neural Information Processing Systems (NeurIPS)*, 2019.

## Appendix A Experimental Details

Following sections provide details on datasets, preprocessing, models and training configurations.

### A.1 Datasets

#### Indirect Indexing.

This diagnostic task (Indirect Idx.) requires the model to locate a target character in a variable-length source string that is at a certain relative distance (left or right) from a source character.
It tests for the structured manipulation (pointer arithmetic operations) of the content and positional information of tokens by a sequence processing architecture.
The dataset for this task is constructed by procedurally generating examples of source strings, source character and relative shifts.
We generate source strings of length between 20 and 40 characters from the set of uppercase [A-Z] and lowercase [a-z] letters by uniform sampling without replacement.
Then, we randomly pick a source character given a randomly sampled source string.
Next, we uniformly sample shifts in the range [-15, +15] (where - and + indicate left and right shifts respectively) and consequently obtain our target character.
We generate a train/validation/test splits of size 1M/10k/10k respectively.
We use character-level tokenization, i.e. all uppercase and lowercase letters, individual digits, the delimiter symbol, plus and minus signs are separate tokens.
The format of each examples is: $<$source string$>$, $<$source character$>$, $<$shift$>$, $<$target character$>$ and ’,’ as a delimiter and the model is given the entire sequence except the target character.
Below are a few samples from the dataset for reference.
TzbkWoKDyscBepYvfwxEVQtgPa, c, -8, b
NZTUIGWkXFrhCJDzscat, N, +4, I
RBEvOPgtaGDnjhbJCLScruZpMNsyWfQxXFAzUT, x, +2, F

#### OpenWebText.

This dataset is an open source effort to reproduce OpenAI’s WebText dataset (Gokaslan & Cohen, 2019) which was used to train GPT-2 (Radford et al., 2019).
The training and validation splits roughly contain 9B and 4M tokens respectively and maximum sequence length of 1024 for pretraining.
We use the GPT-2 tokenizer with a vocabulary size of 50257.

#### Bach-Chorales.

This dataset (JSB) consists of 4-part scored choral music, which are represented as a matrix with rows corresponding to voices and columns to time discretized to 16th notes.
The entries in this 2D matrix are integers that denote the pitch being played.
We serialize this matrix in raster-scan fashion by first going down the rows and then moving right through the
columns as in prior work (Huang et al., 2019).
We use the variant of the dataset with 16th note temporal “quantizations” where silence is represented by a pitch of -1 rather than NaN, available in JSON file format (^2^22[https://github.com/czhuang/JSB-Chorales-dataset](https://github.com/czhuang/JSB-Chorales-dataset)).
We use a maximum sequence length of 2048 for training with 229/76/77 sequences present in the train/validation/test sets.
We use a vocabulary size of 90 which includes the MIDI notes, silence and padding tokens.

#### MAESTRO.

The dataset contains about 200 hours of paired audio and MIDI recordings from ten years of International Piano-e-Competition.
The MIDI data includes key strike velocities and sustain/sostenuto/una corda pedal positions.
We use version v3.0.0 of this dataset in MIDI format (^3^33[https://magenta.withgoogle.com/datasets/maestro](https://magenta.withgoogle.com/datasets/maestro)) for our experiments.
We apply data augmentation by using pitch transpositions sampled uniformly from $\{-3,-2,-1,0,+1,+2,+3\}$ similar to prior work (Huang et al., 2019), divide it into sequences with a maximum length of 2048 and use a 90/5/5 percent split for train/validation/test sets.
We use the REMI tokenizer (Huang & Yang, 2020) with EOS, BOS, MASK and PAD tokens leading to a total vocabulary size of 328.

#### Human Reference Genome.

The human reference genome (HRG) dataset was constructed by considering all autosomal and sex chromosomes sequences from reference assembly GRCh38/hg38 (^4^44[https://huggingface.co/datasets/InstaDeepAI/human_reference_genome](https://huggingface.co/datasets/InstaDeepAI/human_reference_genome)) and reached a total of 3.2 billion nucleotides.
We follow the preprocessing and tokenization procedures from the recent state-of-the-art model for genomic sequence modeling, the Nucleotide Transformer (Dalla-Torre et al., 2025).

#### PG-19.

This dataset includes a set of books extracted from the Project Gutenberg books library, that were published before 1919.
It is a popular dataset first introduced by (Rae et al., 2020) to benchmark long-range language models.
In this work, we use it to evaluate the test-time length extrapolation capabilities of Transformer models that use different relative positional encoding schemes.
We use the test split of the dataset containing 100 books or roughly 7M tokens. (^5^55[https://huggingface.co/datasets/emozilla/pg19-test](https://huggingface.co/datasets/emozilla/pg19-test))

### A.2 Models

[Table 6](https://arxiv.org/html/2509.10534v1#A1.T6) contains the Transformer model configuration used on each dataset.

Table: Table 6: Transformer model configurations for the different datasets. For the OpenWebText language modeling dataset we train 3 model sizes 124M/253M/774M and the hyperparams for each of these model sizes as a triple in column 2.

### A.3 Training Details

[Table 7](https://arxiv.org/html/2509.10534v1#A1.T7) contains the Transformer model hyperparameters for all the datasets.

Table: Table 7: Hyperparameter configurations for training on different datasets. For OpenWebText dataset we train Transformer models at sizes 124M/253M/774M using identical hyperparameters.

## Appendix B Additional Results

**Table 8: Validation set perplexity scores for the 124M Transformer model with ablated versions of PoPE pretrained on OpenWebText.**
| Positional Encoding | 124M |
| --- | --- |
| PoPE without $\sigma()$ | 21.57 |
| PoPE without $\bm{\delta}$ | 21.42 |
| Full PoPE | 21.33 |

To test the efficacy of key design choices, we run ablations by training 124M Transformer models on OpenWebText with identical configurations, comparing our full PoPE method against variants that remove either the softplus activation $\sigma()$ or the bias vector $\bm{\delta}$ that tunes the optimal relative offset for each channel $c$.
From [Table 8](https://arxiv.org/html/2509.10534v1#A2.T8) we see that by removing each of these components the performance of our PoPE variants worsens with increases in perplexity scores.
The full PoPE model outperforms these ablated variants and indicates the efficacy of these design choices.

## Appendix C Additional Visualizations

We apply the frequency usage analysis for the 253M-sized Transformer model pretrained on OpenWebText (from [Table 4](https://arxiv.org/html/2509.10534v1#S4.T4)).
From [Figure 5](https://arxiv.org/html/2509.10534v1#A3.F5) and [Figure 6](https://arxiv.org/html/2509.10534v1#A3.F6), we see a similar pattern emerge as before, i.e. RoPE tends to use a sparse set of frequencies indicated by the high norms and does not use the highest frequencies.
Whereas PoPE uses a broader set of frequencies most notably the highest ones.

Figure: Figure 5: 2-norm plotted over 2D RoPE components of queries (left) and keys (right) in each layer of the 253M Transformer over different RoPE frequencies. Mean over 10 different Shakespeare sonnets and 16 attention heads at each layer.
Refer to caption: x5.png

Figure: Figure 6: Magnitude of each complex-valued features of queries (left) and keys (right) in each layer of the 253M Transformer over different PoPE frequencies. Mean over 10 different Shakespeare sonnets and 16 attention heads at each layer.
Refer to caption: x6.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Could you estimate the potential throughput (TFLOP/s) overhead of a memory-optimized PoPE kernel compared to a standard Flash Attention implementation?
2. How does PoPE's performance on the PG-19 length extrapolation task compare against a RoPE baseline that has been enhanced with YaRN?
3. Could you provide a visualization of the learned `δc` values to reveal if any structural patterns emerge across layers or frequencies?
4. Is RoPE's failure on the "Indirect Indexing" task highly sensitive to the choice of the `base` frequency `θ`? Would changing `θ` mitigate this failure?
5. The softplus function is crucial for decoupling. Did you experiment with other non-negativity constraints (e.g., ReLU), and how did they perform? This would help clarify its exact role.
6. The evaluation of long-context capabilities is narrow, resting on perplexity on a single benchmark (PG-19). This assesses modeling fluency but fails to test functional abilities like long-range information retrieval or QA, which are crucial for substantiating the paper's primary contribution.
soundness: 2
strengths: The paper offers a clear and insightful diagnosis of a fundamental limitation in RoPE—the entanglement of content and position. The proposed PoPE method is theoretically clean, directly addressing the identified issue with scientific taste. The central claims are backed by strong experimental evidence across diverse domains (synthetic, language, music, genomics) and model scales. In particular, the method's ability to achieve robust zero-shot length extrapolation without special tricks is highly compelling and demonstrates significant value.
summary: This paper presents an analysis of the popular Rotary Position Embedding (RoPE), arguing that it entangles content ("what") and position ("where") information within the attention mechanism. The authors hypothesize that this entanglement impairs model performance, especially on tasks requiring independent matching of these two factors. To address this, they propose Polar Coordinate Positional Embedding (PoPE), a novel method that decouples content and position by re-framing queries and keys in polar coordinates, where content is mapped to magnitude and position to phase.
weaknesses: 1. In its current implementation, PoPE doubles the K/V cache memory footprint, which is a serious practical barrier for training and deploying large-scale models.
2. The length extrapolation evaluation lacks a direct comparison to RoPE augmented with established extension techniques like YaRN or PI, making it difficult to assess its relative advantage over existing SOTA solutions.
3. The paper does not deeply investigate the role or the learned patterns of the `δc` bias term, despite showing it is critical for performance.
4. On standard language modeling benchmarks like OpenWebText, the performance improvements from PoPE, while consistent, are relatively modest in absolute terms.
5. The finding that different `δc` initialization strategies benefit in-distribution vs. extrapolation performance suggests the method's optimal configuration may be task-dependent, potentially complicating its use.

---

confidence: 5
contribution: 1
initial_rating: 2
presentation: 2
questions: See the weaknesses part.
soundness: 2
strengths: (1) PoPE shows an improvement over RoPE on the Indirect Indexing synthetic task. This task involves string sequences where, given a source character and an offset relative to that character, the model needs to retrieve the target character obtained by applying the offset to the source. Experimental results demonstrate that, compared to RoPE, PoPE—whose cosine function phase depends only on relative positional relationships—enables more accurate localization for this type of task.  

(2) PoPE demonstrates better length extrapolation ability than RoPE during testing. According to Figure 2 in the paper, when models are trained on sequences of length 1024 and then evaluated on sequences up to 10240 in length, PoPE’s language modeling perplexity loss remains nearly unchanged. In contrast, RoPE shows a noticeable increase in perplexity, highlighting PoPE’s advantage in handling longer sequences against RoPE.
summary: In this paper, the authors rewrite RoPE positional encoding in polar coordinates and discover that the phase of the cosine function in RoPE is related not only to relative position ("where," i.e., $(s - t)\theta_c$ in Equation (2)), but also to the query/key themselves ("what," i.e., $\phi_{k_{sc}} - \phi_{q_{tc}}$ in Equation (2)). The authors argue that the coupling effect of 'what' and 'where' on the phase of the cosine function may be detrimental to model performance. Therefore, they propose PoPE positional encoding, which removes the 'what' term from the cosine function's phase, so that the phase depends solely on relative positional relationships. In their experimental setup, the authors compare RoPE and PoPE on tasks such as Indirect Indexing, synthetic music sequence modeling, human genome sequence modeling, language modeling and downstream evaluation, as well as length extrapolation during testing.
weaknesses: The paper lacks citations and discussion of works with very similar formulations. This work shares an almost identical formulation with [1] and complex version of [2], yet it neither cites nor discusses the relationship between them. Additionally, the improvements appear marginal, and there is no efficiency/speed comparison provided.

Citations:
1. https://arxiv.org/abs/2202.08791
2. https://arxiv.org/abs/2307.09270

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Can you formalize why removing the $\phi_k-\phi_q$ interaction improves expressivity or inductive bias, beyond the empirical results?
2. Did you test other non-negative mappings (e.g., ReLU, square, exponential)? How sensitive are results to the choice of activation?
3. Could this reintroduce a form of coupling or shift-dependent bias that undermines the core disentanglement principle? 
When $s \approx t$, the query and key correspond to nearby positions in the sequence, and one would expect 
$\cos((t - s)\theta_c) \approx 1$ in Eq.(5). 
However, introducing a learnable bias term $\delta_c$ could force 
$\cos((t - s)\theta_c + \delta_c) \neq 1$, 
which appears counterintuitive and may distort locality or positional consistency.
soundness: 2
strengths: 1. The paper provides a geometric explanation of RoPE’s limitation, mixing content and position, and introduces PoPE as a principled disentanglement.
2. The polar-coordinate derivation clearly isolates the “interaction term” $\phi_k-\phi_q$ and justifies its removal to achieve independence between what/where components.
3. Experiments span diagnostic synthetic tasks, structured symbolic domains (music, genomics), and large-scale language modeling, showing consistent, domain-general gains.
4. The length extrapolation results (up to 10× longer sequences) are compelling and address a long-standing weakness of RoPE-based models.
5. PoPE can be dropped into any Transformer with almost no code change and negligible computational overhead (only one extra multiplication).
summary: This paper proposes Polar Coordinate Positional Embedding (PoPE), a modification of the widely used Rotary Position Embedding (RoPE) for Transformer architectures. The authors argue that RoPE entangles content (“what”) and position (“where”) information in query–key attention, which may hinder performance on tasks that require independent reasoning about these two aspects.

PoPE reformulates RoPE in polar coordinates, where the magnitudes (content) and phases (position) are explicitly separated. The magnitudes of queries and keys are made positive via a softplus transformation, and the position is represented purely by the phase terms. This eliminates the cross-term between the query/key phases that causes what–where coupling in RoPE.

Empirically, PoPE shows strong improvements across multiple domains: 1. Indirect Indexing diagnostic task: 95% vs 11% accuracy (RoPE) 2. Sequence modeling: lower NLL on music (JSB, MAESTRO) and genomic (HRG) datasets; 3. Language modeling: reduced perplexity on OpenWebText across 124M–774M parameter scales 4. Downstream and extrapolation tasks: better zero-shot accuracy and robust long-context generalization without interpolation.
weaknesses: 1. The “disentanglement” argument is mostly qualitative. There is no formal analysis or proof that the removed phase interaction necessarily leads to improved generalization.

2. Softplus is heuristic. While used for ensuring positive magnitudes, its necessity and effect are not theoretically grounded beyond empirical ablation.

3. It would be useful to see comparisons to YaRN, ALiBi, or other strong length-extrapolatable positional encodings, not just RoPE.

4. While conceptually appealing, it remains unclear how the model behaviorally separates content vs. position in downstream reasoning.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: See above.
soundness: 3
strengths: - The method is motivated mathematically and intuitive to follow.
- The method is simple and not architecture dependent (aside from being used in Transformers), thus I am under the impression it can be integrated with any existing method on this front.
- The experiments are comprehensive and demonstrate the benefits of PoPE in relation to RoPE.
summary: The authors introduce Polar Coordinate Position Embeddings (PoPE), an improvement to RoPE which uses polar coordinates to encode positional information. The primary benefit is that by disentangling content and position in RoPE through the use of angle and magnitude information, there is better mixing of information that can be performed by the model in a context-specific manner. This is demonstrated through superior performance in various sequence modeling tasks and zero-shot length extrapolation.
weaknesses: - While the method does show improvements relative to RoPE in the context of the models being studied, some standard benchmarks could be included (ex. LongBench, RULER, etc.) to better show some of the long-context scaling behaviour beyond just perplexity. Additionally, results after instruction-tuning the model would be appreciated to better highlight the benefits of the method.
- One small consideration is that given the additional parameters that are stored for RoPE, this could affect cache optimization at inference time. I would expect this to potentially lead to come limitations on this front, and this could also lead to some problems with current cache optimization techniques or inference-time scaling techniques (ex. attention sinks or meta-tokens).


Decision: Reject