Polynomial Composition Activations: Unleashing the Dynamics of Large
  Language Models
Authors: name: Zhijian Zhuo, name: Ya Wang, name: Yutao Zeng, name: Xiaoqing Li, name: Xun Zhou, name: Jinwen Ma

Transformers have found extensive applications across various domains due to
the powerful fitting capabilities. This success can be partially attributed to
their inherent nonlinearity. Thus, in addition to the ReLU function employed in
the original transformer architecture, researchers have explored alternative
modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment
representational capacity. In this paper, we propose a novel category of
polynomial composition activations (PolyCom), designed to optimize the dynamics
of transformers. Theoretically, we provide a comprehensive mathematical
analysis of PolyCom, highlighting its enhanced expressivity and efficacy
relative to other activation functions. Notably, we demonstrate that networks
incorporating PolyCom achieve the optimal approximation rate,
indicating that PolyCom networks require minimal parameters to approximate
general smooth functions in Sobolev spaces. We conduct empirical experiments on
the pre-training configurations of large language models (LLMs), including both
dense and sparse architectures. By substituting conventional activation
functions with PolyCom, we enable LLMs to capture higher-order interactions
within the data, thus improving performance metrics in terms of accuracy and
convergence rates. Extensive experimental results demonstrate the effectiveness
of our method, showing substantial improvements over other activation
functions. Code is available at https://github.com/BryceZhuo/PolyCom.