Where to Attend: A Principled Vision-Centric Position Encoding with Parabolas
Authors: name: Christoffer Koo Øhrstrøm, name: Rafael I. Cabral Muchacho, name: Yifei Dong, name: Filippos Moumtzidellis, name: Ronja Güldenring, name: Florian T. Pokorny, name: Lazaros Nalpantidis
Organization: DTU - Perception and Cognition for Autonomous Systems

We propose Parabolic Position Encoding (PaPE), a parabola-based position encoding for vision modalities in attention-based architectures. Given a set of vision tokens-such as images, point clouds, videos, or event camera streams-our objective is to encode their positions while accounting for the characteristics of vision modalities. Prior works have largely extended position encodings from 1D-sequences in language to nD-structures in vision, but only with partial account of vision characteristics. We address this gap by designing PaPE from principles distilled from prior work: translation invariance, rotation invariance (PaPE-RI), distance decay, directionality, and context awareness. We evaluate PaPE on 8 datasets that span 4 modalities. We find that either PaPE or PaPE-RI achieves the top performance on 7 out of 8 datasets. Extrapolation experiments on ImageNet-1K show that PaPE extrapolates remarkably well, improving in absolute terms by up to 10.5% over the next-best position encoding. Code is available at https://github.com/DTU-PAS/parabolic-position-encoding.