text: 322,276,1405179533335072768,636167919,Andreas Loukas,"Are well-generalizing neural nets (NNs) easier to train?🤔Aiming to shed light on this hypothesis, we studied the relation between the complexity of the learned NN and the training behavior. <LINK>📰with @MarinosPoiitis & @StefanieJegelka 👇 <LINK> Some evidence for our hypothesis already exists: e.g., it is known that training (shallow) NNs is more tedious for noisy data and easier for more separable classes. Moreover, the beautiful theory of stability says that NNs trained for few epochs have bounded sample complexity. Differently, we connect aspects of the optim. trajectory with the NN Lipschitz constant (wrt input) close and far from the training data: we find that the trajectory of high complexity NNs is longer, veers further from initialization, and exhibits higher variance near convergence Intriguingly, we also find that steady training with Dropout implies a training- and data-dependent generalization bound that grows poly-logarithmically with the number of parameters (typical Lipschitz constant-based generalization bounds grow exponentially 🚀with NN depth). Overall, our results support the hypothesis that good training behavior can be a useful bias towards good generalization",https://arxiv.org/abs/2106.04186,"This work explores the Benevolent Training Hypothesis (BTH) which argues that the complexity of the function a deep neural network (NN) is learning can be deduced by its training dynamics. Our analysis provides evidence for BTH by relating the NN's Lipschitz constant at different regions of the input space with the behavior of the stochastic training procedure. We first observe that the Lipschitz constant close to the training data affects various aspects of the parameter trajectory, with more complex networks having a longer trajectory, bigger variance, and often veering further from their initialization. We then show that NNs whose 1st layer bias is trained more steadily (i.e., slowly and with little variation) have bounded complexity even in regions of the input space that are far from any training point. Finally, we find that steady training with Dropout implies a training- and data-dependent generalization bound that grows poly-logarithmically with the number of parameters. Overall, our results support the intuition that good training behavior can be a useful bias towards good generalization. ",What training reveals about neural network complexity,5,"['Are well-generalizing neural nets (NNs) easier to train?🤔Aiming to shed light on this hypothesis, we studied the relation between the complexity of the learned NN and the training behavior.\n\n<LINK>📰with @MarinosPoiitis &amp; @StefanieJegelka\n👇 <LINK>', 'Some evidence for our hypothesis already exists: e.g., it is known that training (shallow) NNs is more tedious for noisy data and easier for more separable classes. Moreover, the beautiful theory of stability says that NNs trained for few epochs have bounded sample complexity.', 'Differently, we connect aspects of the optim. trajectory with the NN Lipschitz constant (wrt input) close and far from the training data: we find that the trajectory of high complexity NNs is longer, veers further from initialization, and exhibits higher variance near convergence', 'Intriguingly, we also find that steady training with Dropout implies a training- and data-dependent generalization bound that grows poly-logarithmically with the number of parameters (typical Lipschitz constant-based generalization bounds grow exponentially 🚀with NN depth).', 'Overall, our results support the hypothesis that good training behavior can be a useful bias towards good generalization']",21,06,1200