FAN: Fourier Analysis Networks
Authors: name: Yihong Dong; name: Ge Li; name: Yongding Tao; name: Xue Jiang; name: Kechi Zhang; name: Jia Li; name: Jing Su; name: Jun Zhang; name: Jingjing Xu
Published: 2024-10-03 17:02:21

Abstract: Despite the remarkable success achieved by neural networks, particularly
those represented by MLP and Transformer, we reveal that they exhibit potential
flaws in the modeling and reasoning of periodicity, i.e., they tend to memorize
the periodic data rather than genuinely understanding the underlying principles
of periodicity. However, periodicity is a crucial trait in various forms of
reasoning and generalization, underpinning predictability across natural and
engineered systems through recurring patterns in observations. In this paper,
we propose FAN, a novel network architecture based on Fourier Analysis, which
empowers the ability to efficiently model and reason about periodic phenomena.
By introducing Fourier Series, the periodicity is naturally integrated into the
structure and computational processes of the neural network, thus achieving a
more accurate expression and prediction of periodic patterns. As a promising
substitute to multi-layer perceptron (MLP), FAN can seamlessly replace MLP in
various models with fewer parameters and FLOPs. Through extensive experiments,
we demonstrate the effectiveness of FAN in modeling and reasoning about
periodic functions, and the superiority and generalizability of FAN across a
range of real-world tasks, including symbolic formula representation, time
series forecasting, and language modeling.