Universality Patterns in the Training of Neural Networks
This paper proposes and demonstrates a surprising pattern in the training of neural networks: there is a one to one relation between the values of any pair of losses evaluated for a model arising at a training run.This pattern is universal in the sense that this one to one relationship is identical across architectures, algorithms and training loss functions.
We identify some universal patterns (i.e., holding across architectures) in the behavior of different surrogate losses (CE, MSE, 0-1 loss) while training neural networks and present supporting empirical evidence.