Title: Gaussian Process Attention with Kernel Modeling
Primary area: interpretability and explainable AI
Keywords: Transformers, Kernel Methods, Gaussian Processes, Interpretability, Language Modeling

Abstract: Transformers dominate NLP, yet their core component, self-attention, remains a heuristic, lacking a robust theoretical foundation. This paper reinterprets self-attention combined with rotary positional embeddings (RoPE) as an instance of Nadaraya-Watson (NW) kernel regression, unlocking a novel framework for enhancing attention through kernel modeling. 
We introduce Gaussian Process Attention (GPA), a novel technique, which leverages kernel theory to augment RoPE with a bank of decaying periodic functions.  We demonstrate the result better captures the fine-grain characteristics of mutual information between tokens as a function of the distance between them. 
Tested on a GPT model using character-based tokenization and trained on a corpus of 14-million-characters, GPA outperforms baseline RoPE, reducing mean cross-entropy loss.  More importantly, with only a nominal increase in the number of model parameters, GPA reduces computation time by 75\% for an equivalent level of mean cross entropy loss. Moreover, we show that our GPA model opens a new avenue for studying mechanistic interpretability, revealing  structures, such as paragraph lengths, and identifying redundant attention heads for model pruning. Our work bridges the application of kernel methods and the study of Transformers, providing a theoretical lens for analyzing self-attention while also delivering practical, scalable gains in performance and interpretability.

TL;DR: Gaussian Process Attention reframes self‑attention as kernel regression, augmenting RoPE with learned decaying periodic kernels to improve efficiency, accuracy, and interpretability in Transformers.