Title: Position-Aware Attention Mechanism: A Mathematical Framework for Enhanced Spatial Information Processing in Transformer Architectures
Primary area: foundation or frontier models, including LLMs
Keywords: Attention Mechanism, Position-Aware Attention, Triple-Attention, Parameter Sensitivity Analysis

Abstract: We propose a position-aware attention mechanism that addresses the limitations of traditional attention mechanisms in capturing positional relationships through a positional effect function. Building upon the foundation of relative position representations and rotary position embeddings, we prove its mathematical properties (continuity, differentiability, monotonicity) and demonstrate that parameters $\alpha$ and $\beta$ control position influence intensity and spatial decay rate, achieving fine-grained attention allocation. To address over-attenuation at long distances, we introduce enhancement coefficient $\gamma$, extending beyond the standard exponential decay patterns. We develop an adaptive triple-attention architecture with task-aware and content-aware modules for dynamic weight adjustment, inspired by recent advances in multi-scale attention mechanisms. Our method includes a maximum benefit position formula and consistency metric for evaluation across different information distribution patterns. Experimental results show excellent performance in structured and clustered scenarios, particularly for information retrieval and document understanding tasks, demonstrating advantages over existing efficient attention approaches.

TL;DR: We propose a triple-attention architecture that combines position-aware, task-aware, and content-aware attention mechanisms with mathematical frameworks for optimal information distribution and adaptive attention allocation