Title: Is the Frequency Principle always valid?
Primary area: learning theory
Keywords: Neural Networks; Learning Theory; Frequency Principle

Abstract: We investigate the learning behaviors of shallow ReLU neural networks on the unit sphere $S^2\subset\mathbb{R}^3$ in polar coordinates $(\tau,\phi)$, comparing fixed versus trainable neuron weights $\{w_i\}$. Spherical harmonic analysis reveals that both cases exhibit low-frequency preference with coefficients decaying as $O(\ell^{5/2}/2^\ell)$ for fixed weights and $O(\ell^{7/2}/2^\ell)$ for trainable weights, typically leading to the Frequency Principle (FP) of lower-frequency-first learning. However, this principle can be violated under specific initial conditions or error distributions. Our numerical results show that trainable directions increase learning complexity while maintaining the low-frequency advantage, though they can also enable faster high-frequency emergence. This suggests the FP should be viewed as a tendency rather than a strict rule on curved domains like $S^2$.

TL;DR: In this paper, we explore when the frequency principle will holds or fail.