Title: When Bias Helps Learning: Bridging Initial Prejudice and Trainability
Primary area: learning theory
Keywords: Trainability, initial guessing bias, mean field regime, phase diagrams

Abstract: Understanding the statistical properties of deep neural networks (DNNs) at initialization is crucial for elucidating both their trainability and the intrinsic architectural biases they encode prior to data exposure. Mean-field (MF) analyses have demonstrated that the parameter distribution in randomly initialized networks dictates whether gradients vanish or explode. Recent work has shown that untrained DNNs exhibit an initial-guessing bias (IGB), in which large regions of the input space are assigned to a single class. 
In this work, we provide a theoretical proof linking IGB to MF analyses, establishing that a network’s predisposition toward specific classes is intrinsically tied to the conditions for efficient learning. This connection leads to a counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral. We validate our theory through experiments across multiple architectures and datasets.

TL;DR: We prove theoretically that the optimal trainable state is necessarily biased in a wide range of models.