[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. "Language models are few-shot learners". In: Advances in Neural Information Processing Systems 33 (2020), pp. 1877-1901.
[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 2017. arXiv: 1706.03762.
[3] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. 2017. arXiv: 1712.00409.
[4] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. 2020. arXiv: 2001.08361.
[5] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. 2022. arXiv: 2203.15556.
[6] Y. Tay, M. Dehghani, S. Abnar, H. W. Chung, W. Fedus, J. Rao, S. Narang, V. Q. Tran, D. Yogatama, and D. Metzler. Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? 2022. arXiv: 2207.10551.
[7] A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling Laws for Generative Mixed-Modal Language Models. 2023. arXiv: 2301.03728.
[8] Y. Dong, J.-B. Cordonnier, and A. Loukas. "Attention is not all you need: Pure attention loses rank doubly exponentially with depth". International Conference on Machine Learning. PMLR. 2021, pp. 2793-2803.
[9] L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Lucchi. Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse. 2022. arXiv: 2206.03126.
[10] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. "Exponential expressivity in deep neural networks through transient chaos". In: Advances in Neural Information Processing Systems 29 (2016).
[11] S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. "Deep Information Propagation". ICLR. 2017.
[12] G. Yang and S. S. Schoenholz. “Mean field residual networks: on the edge of chaos”. Advances in Neural Information Processing Systems. 2017, pp. 2865–2873.
[13] S. Hayou, A. Doucet, and J. Rousseau. "On the impact of the activation function on deep neural networks training". International Conference on Machine Learning. PMLR. 2019, pp. 2672-2680.
[14] M. Murray, V. Abrol, and J. Tanner. "Activation function design for deep networks: linearity and effective initialisation". In: Applied and Computational Harmonic Analysis 59 (2022), pp. 117-154.
[15] L. Xiao, J. Pennington, and S. Schoenholz. "Disentangling trainability and generalization in deep neural networks". International Conference on Machine Learning. PMLR. 2020, pp. 10462-10472.
[16] J. Martens, A. Ballard, G. Desjardins, G. Swirszcz, V. Dalibard, J. Sohl-Dickstein, and S. S. Schoenholz. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. 2021. arXiv: 2110.01765.
[17] G. Zhang, A. Botev, and J. Martens. Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers. 2022. arXiv: 2203.08120.
[18] M. Li, M. Nica, and D. Roy. "The neural covariance SDE: Shaped infinite depth-and-width networks at initialization". In: Advances in Neural Information Processing Systems 35 (2022), pp. 10795-10808.
[19] G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. 2022. arXiv: 2203.03466.
[20] R. M. Neal. "Bayesian learning for neural networks". PhD thesis. University of Toronto, 1995.