Title: Why High-rank Neural Networks Generalize?: An Algebraic Framework with RKHSs
Primary area: learning theory
Keywords: Generalization bound, Deep neural network, Koopman operator, Reproducing kernel Hilbert space

Abstract: We derive a new Rademacher complexity bound for deep neural networks using Koopman operators, group representations, and reproducing kernel Hilbert spaces (RKHSs).
The proposed bound describes why the models with high-rank weight matrices generalize well.
Although there are existing bounds that attempt to describe this phenomenon, these existing bounds can be applied to limited types of models.
We introduce an algebraic representation of neural networks and a kernel function to construct an RKHS to derive a bound
for a wider range of realistic models.
This work paves the way for the Koopman-based theory for Rademacher complexity bounds to be valid for more practical situations.

TL;DR: We derive a new Rademacher complexity bound that describes why the models with high-rank weight matrices generalize well, which is valid for a wide range of models.