Title: Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective
Primary area: learning theory
Keywords: manifold, equivariance, generalization bounds

Abstract: Understanding generalization is a central issue in machine learning. Recent work has identified two key mechanisms to explain it: the strong memorization capabilities of neural networks, and the task-aligned invariants imposed by their architecture as well as training procedure. Remarkably, it is possible to characterize the neural network behavior for some classes of invariants widely used in practice. Leveraging this characterization, we introduce the representation gap, a metric that generalizes empirical risk and enables asymptotic analysis across three common settings: (i) unconditional generative modeling, where we obtain a precise asymptotic equivalent; (ii) supervised prediction; and (iii) ambiguous prediction tasks. A central outcome is that generalization is governed by a single parameter -- the intrinsic dimension of the task -- which captures task difficulty. As a corollary, we prove that popular strategies such as equivariant architectures improve performance by explicitly reducing this intrinsic dimension.

TL;DR: We introduce the representation gap to study neural networks generalization through a geometric perspective, deriving asymptotic bounds in the context of generative modeling and supervised prediction.