Rethinking Addressing in Language Models via Contexualized Equivariant
  Positional Encoding
Authors: name: Jiajun Zhu; name: Peihao Wang; name: Ruisi Cai; name: Jason D. Lee; name: Pan Li; name: Zhangyang Wang
Published: 2025-01-01 03:23:00

Abstract: Transformers rely on both content-based and position-based addressing
mechanisms to make predictions, but existing positional encoding techniques
often diminish the effectiveness of position-based addressing. Many current
methods enforce rigid patterns in attention maps, limiting the ability to model
long-range dependencies and adapt to diverse tasks. Additionally, most
positional encodings are learned as general biases, lacking the specialization
required for different instances within a dataset. To address this, we propose
conTextualized equivariAnt Position
Embedding (TAPE), a novel framework that enhances
positional embeddings by incorporating sequence content across layers. TAPE
introduces dynamic, context-aware positional encodings, overcoming the
constraints of traditional fixed patterns. By enforcing permutation and
orthogonal equivariance, TAPE ensures the stability of positional encodings
during updates, improving robustness and adaptability. Our method can be easily
integrated into pre-trained transformers, offering parameter-efficient
fine-tuning with minimal overhead. Extensive experiments shows that TAPE
achieves superior performance in language modeling, arithmetic reasoning, and
long-context retrieval tasks compared to existing positional embedding
techniques.