SeqPE: Transformer with Sequential Position Encoding
Authors: name: Huyang Li; name: Yahui Liu; name: Hongyu Sun; name: Deng Cai; name: Leyang Cui; name: Wei Bi; name: Peilin Zhao; name: Taro Watanabe
Published: 2025-06-16 09:16:40

Abstract: Since self-attention layers in Transformers are permutation invariant by
design, positional encodings must be explicitly incorporated to enable spatial
understanding. However, fixed-size lookup tables used in traditional learnable
position embeddings (PEs) limit extrapolation capabilities beyond pre-trained
sequence lengths. Expert-designed methods such as ALiBi and RoPE, mitigate this
limitation but demand extensive modifications for adapting to new modalities,
underscoring fundamental challenges in adaptability and scalability. In this
work, we present SeqPE, a unified and fully learnable position encoding
framework that represents each n-dimensional position index as a symbolic
sequence and employs a lightweight sequential position encoder to learn their
embeddings in an end-to-end manner. To regularize SeqPE's embedding space, we
introduce two complementary objectives: a contrastive objective that aligns
embedding distances with a predefined position-distance function, and a
knowledge distillation loss that anchors out-of-distribution position
embeddings to in-distribution teacher representations, further enhancing
extrapolation performance. Experiments across language modeling, long-context
question answering, and 2D image classification demonstrate that SeqPE not only
surpasses strong baselines in perplexity, exact match (EM), and
accuracy--particularly under context length extrapolation--but also enables
seamless generalization to multi-dimensional inputs without requiring manual
architectural redesign. We release our code, data, and checkpoints at
https://github.com/ghrua/seqpe.