[45] T. Jiang and Y. Qi. "Spectral radii of large non-Hermitian random matrices". In: Journal of Theoretical Probability 30.1 (2017), pp. 326-364.
[46] B. He, J. Martens, G. Zhang, A. Botev, A. Brock, S. L. Smith, and Y. W. Teh. Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation. 2023. arXiv: 2302.10322.
[47] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. 2023. arXiv: 2302.05442.
[48] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu. "On layer normalization in the transformer architecture". International Conference on Machine Learning. PMLR. 2020, pp. 10524-10533.
[49] J. L. Ba, J. R. Kiros, and G. E. Hinton. "Layer normalization". In: (2016). arXiv: 1607.06450.
[50] S. Hayou and G. Yang. Width and Depth Limits Commute in Residual Networks. 2023. arXiv: 2302.00453.
[51] S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau. "Stable resnet". International Conference on Artificial Intelligence and Statistics. PMLR. 2021, pp. 1324-1332.
[52] W. Tarnowski, P. Warchol, S. Jastrzebski, J. Tabor, and M. Nowak. "Dynamical isometry is achieved in residual networks in a universal way for any activation function". The 22nd International Conference on Artificial Intelligence and Statistics. PMLR. 2019, pp. 2221-2230.
[53] S. Hayou. On the infinite-depth limit of finite-width neural networks. 2022. arXiv: 2210.00688.
[54] G. Yang. "Tensor programs ii: Neural tangent kernel for any architecture". In: arXiv preprint arXiv:2006.14548 (2020).
[55] K. He, X. Zhang, S. Ren, and J. Sun. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification". Proceedings of the IEEE international conference on computer vision. 2015, pp. 1026-1034.
[56] A. Veit, M. J. Wilber, and S. Belongie. "Residual networks behave like ensembles of relatively shallow networks". In: Advances in neural information processing systems 29 (2016).
[57] S. Zhai, T. Likhomanenko, E. Littwin, D. Busbridge, J. Ramapuram, Y. Zhang, J. Gu, and J. Susskind. Stabilizing Transformer Training by Preventing Attention Entropy Collapse. 2023. arXiv: 2303.06296.
[58] I. Karatzas and S. Shreve. Brownian motion and stochastic calculus. Vol. 113. Springer Science & Business Media, 2012.
[59] J. Miller. Stochastic Calculus (Lecture Notes). http://www.statslab.cam.ac.uk/~jpm205/teaching/ent2016/lecture_notes.pdf.2015.
[60] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. "When do neural networks outperform kernel methods?" In: Advances in Neural Information Processing Systems 33 (2020), pp. 14820-14830.
[61] E. Abbe, E. B. Adsera, and T. Misiakiewicz. "The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks". Conference on Learning Theory. PMLR. 2022, pp. 4782-4887.
[62] J. Ba, M. A. Erdogdu, T. Suzuki, Z. Wang, D. Wu, and G. Yang. "High-dimensional asymptotics of feature learning: How one gradient step improves the representation". Int. Conf. Learning Representations (ICLR). 2022. URL: https://openreview.net/forum?id=akddwRG6EGi.
[63] A. Damian, J. Lee, and M. Soltanolkotabi. “Neural networks can learn representations with gradient descent”. Conference on Learning Theory. PMLR. 2022, pp. 5413–5452.
[64] A. Mousavi-Hosseini, S. Park, M. Girotti, I. Mitliagkas, and M. A. Erdogdu. Neural Networks Efficiently Learn Low-Dimensional Representations with SGD. 2022. arXiv: 2209.14863.
[65] E. Abbe, E. Boix-Adsera, and T. Misiakiewicz. SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics. 2023. arXiv: 2302.11055.
[66] R. Berthier, A. Montanari, and K. Zhou. Learning time-scales in two-layers neural networks. 2023. arXiv: 2303.00055.