Title: Spectral Norm Variance Regularization for Balanced Layer-Wise Dynamics in Training Deep Neural Networks
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Layer-wise Balancing, Spectral Norm Variance Regularization, Deep Neural Network

Abstract: While deep neural networks (DNNs) are widely adopted in machine learning applications, addressing the systematic imbalance in their layer-wise training dynamics, manifested as discrepancies in learning behavior between earlier and deeper layers, remains a continuous research challenge for improving generalization ability. In this work, we propose spectral norm variance regularization (SNVR) to promote balanced layer-wise dynamics in training DNNs. The spectral norm of a hidden layer, corresponding to the largest singular value of its weight matrix, serves as a proxy for how well the layer has been trained. We introduce a regularization term that penalizes the variance of spectral norms across layers of the same type and incorporate it directly into the learning objective. SNVR requires no strong assumptions about network architecture or learning objective, and its use involves hyperparameter tuning only for controlling the regularization strength. Moreover, it does not significantly increase computational cost, since spectral norms can be efficiently approximated via the power iteration method. Experimental results on multiple benchmark datasets with various network architectures demonstrate that SNVR improves generalization performance compared with baseline methods. Further analyses of layer-wise spectral densities, gradient norms, and information plane provide additional evidence of its balancing effect.

TL;DR: Achieving balanced layer-wise dynamics through direct regularization on the variance of spectral norms across layers.