Title: Polynomial, trigonometric, and tropical activations
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Orthogonal function bases, Tropical polynomials, Polynomial mapping, Deep neural networks, ImageNet-1K, OpenWebText, Transformers, GPT2, Convolutional networks, ConvNeXt, Initialization scheme, PyTorch

Abstract: Which functions can be used as activations in deep neural networks? This article explores families of functions based on orthonormal bases, including the Hermite polynomial basis and the Fourier trigonometric basis, as well as a basis resulting from the tropicalization of a polynomial basis. Our study shows that, through a simple variance-preserving initialization and without additional clamping mechanisms, these activations can successfully be used to train deep models, such as GPT-2 for next-token prediction on OpenWebText and ConvNeXt for image classification on ImageNet. Our work addresses the issue of exploding and vanishing activations and gradients, particularly prevalent with polynomial activations, and opens the door for improving the efficiency of large-scale learning tasks. Furthermore, our approach provides insight into the structure of neural networks, revealing that networks with polynomial activations can be interpreted as multivariate polynomial mappings. Finally, using Hermite interpolation, we show that our activations can closely approximate classical ones in pre-trained models by matching both the function and its derivative, making them especially useful for fine-tuning tasks. These activations are available in the torchortho library.

TL;DR: New activation functions based on orthonormal bases (like Hermite and Fourier) and Tropical polynomials can train deep models effectively, avoid gradient issues, and approximate standard activations.