[8] L. Chizat, E. Oyallon, and F. Bach. On Lazy Training in Differentiable Programming. In NeurIPS, 2019.
[9] J. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient Descent on Neural Networks. Typically Occurs at the Edge of Stability. In ICLR, 2021.
[10] A. Damian, E. Nichani, and J. Lee. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. In NeurIPS, 2022.
[11] X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun. Repvgg: Making vgg-style convnets great again. In CVPR, 2021.
[12] X. Ding, H. Chen, X. Zhang, K. Huang, J. Han, and G. Ding. Re-parameterizing your optimizers rather than architectures. In ICLR, 2023.
[13] S. S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient Descent Finds Global Minima of Deep Neural Networks. In ICML, pages 1675-1685, 2019.
[14] S. S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient Descent Provably Optimizes Overparameterized Neural Networks. In ICLR, 2019.
[15] F. He and T. Liu and D. Tao. Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence. In NeurIPS, 2019.
[16] Z. Fan and Z. Wang. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. In NeurIPS, 2020.
[17] E. Hairer, S. Norsett, and G. Wanner. Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, 2008.
[18] B. Hanin and M. Nica. Products of Many Large Random Matrices and Gradients in Deep Neural Networks. Communications in Mathematical Physics, 376:287-322, 2020.
[19] M. Hauser and A. Ray. Principles of riemannian geometry in neural networks. In NeurIPS, 2017.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Identity Mappings in Deep Residual Networks. In ECCV, 2016.
[21] S. Ioffe and C. Szegedy. Batch Normalization: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, page 448-456, 2015.
[22] A. Jacot, F. Gabriel, and C. Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In NeurIPS, pages 8571-8580, 2018.
[23] H. Karimi, J. Nutini, and M. Schmidt. Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition. In ECML PKDD, pages 795--811, 2016.
[24] N. Lee, T. Ajanthan, S. Gould, and P. H. S. Torr. A signal propagation perspective for pruning neural networks at initialization. In ICLR, 2020.
[25] G. Li, Y. Yang, K. Bhardwaj, and R. Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. In ICLR, 2023.
[26] Y. Li, C. Wei, and T. Ma. Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. In NeurIPS, 2019.
[27] M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Q, H. Li, and R. Jin. Zen-nas: A zero-shot nas for high-performance deep image recognition. In ICCV, 2021.
[28] C. Liu, L. Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. In NeurIPS, 2020.
[29] C. Liu, L. Zhu, and M. Belkin. Loss landscapes and optimization in over-parameterized nonlinear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-116, 2022.
[30] Y. Lu, C. Ma, Y. Lu, J. Lu, and L. Ying. A Mean Field Analysis Of Deep ResNet And Beyond: Towards Provably Optimization Via Overparameterization From Depth. In ICML, 2020.
[31] J. Martens, A. Ballard, G. Desjardins, G. Swirszcz, V. Dalibard, J. Sohl-Dickstein, and S. S. Schoenholz. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv:2110.01765, 2021.