Title: Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds
Primary area: learning theory
Keywords: Fisher information, informaton geometry, Hutchinson's trick, deep learning theory

Abstract: The high dimensional parameter space of modern deep neural networks — the neuromanifold — is endowed with a unique metric tensor defined by the Fisher information, estimating which is crucial for both theory and practical methods in deep learning. To analyze this tensor for classification networks, we return to a low dimensional space of probability distributions — the core space — and carefully analyze the spectrum of its Riemannian metric. We extend our discoveries there into deterministic bounds of the metric tensor on the neuromanifold. We introduce an unbiased random estimate of the metric tensor and its bounds based on Hutchinson’s trace estimator. It can be evaluated efficiently through a single backward pass, with a standard deviation bounded by the true value up to scaling.

TL;DR: We discover an efficient way to estimate the Fisher information in deep classifiers with bounded variance.