NeurIPS 2025

Benign Overfitting in Single-Head Attention
Roey Magen∗
Weizmann Institute of Science
roey.magen@weizmann.ac.il
Shuning Shang∗†
Princeton University
shuning@princeton.edu
Zhiwei Xu
University of Michigan
zhiweixu@umich.edu
Spencer Frei‡
UC Davis
sfrei@ucdavis.edu
Wei Hu§
University of Michigan
vvh@umich.edu
Gal Vardi§
Weizmann Institute of Science
gal.vardi@weizmann.ac.il
Abstract
The phenomenon of benign overfitting, where a trained neural network perfectly fits
noisy training data but still achieves near-optimal test performance, has been exten-
sively studied in recent years for linear models and fully-connected/convolutional
networks. In this work, we study benign overfitting in a single-head softmax
attention model, which is the fundamental building block of Transformers. We
prove that under appropriate conditions, the model exhibits benign overfitting in
a classification setting already after two steps of gradient descent. Moreover, we
show conditions where a minimum-norm/maximum-margin interpolator exhibits
benign overfitting. We study how the overfitting behavior depends on the signal-
to-noise ratio (SNR) of the data distribution, namely, the ratio between norms of
signal and noise tokens, and prove that a sufficiently large SNR is both necessary
and sufficient for benign overfitting.
1
Introduction
Neural networks often exhibit a remarkable phenomenon, known as benign overfitting, where they
achieve a perfect fit to noisy training examples and still generalize well to unseen data [1, 2]. This
phenomenon contradicts classical wisdom in machine learning, and has become a central research
question in the theory of deep learning. Existing works on benign overfitting study under what
conditions the phenomenon occurs in different architectures. These works focus on linear models,
and on shallow fully-connected and convolutional neural networks.
In recent years, Transformers [3] have emerged as a leading neural network architecture, with
impactful applications ac...