Title: On Rademacher Complexity-based Generalization Bounds for the Transformer Architecture
Primary area: learning theory
Keywords: Generalization Bounds, Rademacher Complexity, Transformers, Self-Attention, Deep Learning Theory

Abstract: We derive the first end-to-end, data-dependent generalization bound for the Transformer architecture to explain its strong empirical performance. Using Rademacher complexity and a novel Lipschitz analysis of self-attention, we construct a bound for deep, L-layer models. The bound demonstrates that generalization capacity is governed by depth, sequence length, and a polynomial of the model's weight norms. A numerical sanity check validates our theoretical scaling with model depth, providing a new, formal lens to understand and improve Transformers.

TL;DR: We derive the first formal generalization bound for Transformers using Rademacher complexity, showing that their performance on unseen data is controlled by weight norms, depth, and sequence length