Defazio, A. and Bottou, L. (2021). Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networks. arXiv: 1906.04267.
Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R. (2001). Incorporating Second-Order Functional Knowledge for Better Option Pricing. In Advances in Neural Information Processing Systems, volume 13, pages 472-478, Vancouver, BC, Canada. MIT Press.
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9, pages 249-256.
Golikov, E. and Yang, G. (2022). Non-gaussian tensor programs. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems, volume 35.
Goodfellow, I. J., Mirza, M., Da, X., Courville, A. C., and Bengio, Y. (2014). An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks. In International Conference on Learning Representations, volume 2.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034.
Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Master's thesis, Technical University Munich, Munich, Germany.
Hoedt, P.-J., Hochreiter, S., and Klambauer, G. (2018). Characterising activation functions by their backward dynamics around forward fixed points. NeurIPS workshop on Critiquing and Correcting Trends in Machine Learning.
Hoedt, P.-J., Hochreiter, S., and Klambauer, G. (2022). Normalization is dead, long live normalization! In *ICLR Blog Track*. https://iclr-blog-track.github.io/2022/03/25/unnormalized-resnets/.
Huang, R., Xia, M., Nguyen, D.-T., Zhao, T., Sakamura, S., Zhao, J., Shahane, S. A., Rossoshek, A., and Simeonov, A. (2016). Tox21Challenge to Build Predictive Models of Nuclear Receptor and Stress Response Pathways as Mediated by Exposure to Environmental Chemicals and Drugs. Frontiers in Environmental Science, 3.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. (2017). Self-normalizing neural networks. Advances in neural information processing systems, 30.
Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images. Technical report, Canadian Institute for Advanced Research.
LeCun, Y., Bottou, L., Orr, G. B., and Müller, K. R. (1998). Efficient BackProp. In Orr, G. B. and Müller, K.-R., editors, Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, pages 9-50. Springer, Berlin, Heidelberg, 1 edition.
Maas, A. L., Hannun, A. Y., and Ng, A. Y. (2013). Rectifier Nonlinearities Improve Neural Network Acoustic Models. In Workshop for Deep Learning for Audio, Speech and Language Processing at ICML, Atlanta, GA, USA.
Makkuva, A., Taghvaei, A., Oh, S., and Lee, J. (2020). Optimal transport mapping via input convex neural networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 6672-6681. PMLR.
Martens, J., Ballard, A., Desjardins, G., Swirszcz, G., Dalibard, V., Sohl-Dickstein, J., and Schoenholz, S. S. (2021). Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping. arXiv: 2110.01765.
Mayr, A., Klambauer, G., Unterthiner, T., and Hochreiter, S. (2016). Deeptox: toxicity prediction using deep learning. Frontiers in Environmental Science, 3:80.
Mishkin, D. and Matas, J. (2016). All you need is a good init. In International Conference on Learning Representations, volume 4.