Title: Dimension-Free Minimax Rates for Learning Pairwise Interactions in Attention-Style Models
Primary area: learning theory
Keywords: Attention mechanism, Interacting particle systems, Minimax rates, Nonparametric estimation

Abstract: We study the convergence rate of learning pairwise interactions in single-layer attention-style models, where tokens interact through a weight matrix and a non-linear activation function. We prove that the minimax rate is $M^{-\frac{2\beta}{2\beta+1}}$ with $M$ being the sample size, depending only on the smoothness $\beta$ of the activation, and crucially independent of token count, ambient dimension, or rank of the weight matrix. These results highlight a fundamental dimension-free statistical efficiency of attention-style nonlocal models, even when the weight matrix and activation are not separately identifiable and provide a theoretical understanding of the attention mechanism and its training.

TL;DR: We prove that attention style models learn pairwise token interactions at an optimal minimax rate that depends only on the smoothness of the activation function, and it is independent of the embedding dimension.