The Spectral Bias of Polynomial Neural Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) have demonstrated remarkable success in different domains [ 22 , 11 ] . DNNs can approximate complex functions or even datasets with randomized labels arbitrarily well [ 50 , 3 ] , which makes their ability to avoid over-fitting on real data surprising , since it seems to disagree with prior notions of model complexity . This has sparked the interest in investigating the notion of “ implicit bias ” in neural network training , which makes them favor low complexity solutions [ 43 , 21 , 26 ] . The spectral analysis of deep networks offers one perspective on this implicit bias . Deep neural networks demonstrate a learning bias towards low frequency functions - i.e . functions that vary globally without local fluctuations are learned faster when training neural networks via gradient descent [ 37 , 48 ] . The phenomenon , termed as the spectral bias of neural networks [ 37 ] , has been explored from the perspective of the Neural Tangent Kernel ( NTK ) [ 24 ] . The eigenvalues of the obtained kernel influence important characteristics such as the approximation properties and rate of learning [ 10 , 39 , 36 ] . For standard two-layer ReLU networks within the NTK regime , the analysis supports the idea of a spectral bias by showing faster error convergence for information in lower frequencies [ 12 ] . Recently , another class of models , Polynomial Neural Networks ( PNNs ) , that express high order polynomial expansions , have demonstrated state-of-the-art performance in the challenging tasks of image generation [ 28 ] and face recognition [ 17 ] . Both tasks rely on fine-grained details , which correspond to high-frequency information . More generally , multiplicative interactions have demonstrated strong empirical performance on image-based applications [ 44 , 47 , 4 ] . Jayakumar et al . [ 25 ] highlight how multiplicative interactions , which construct second degree polynomials , can enlarge the hypothesis space and lead to faster learning for certain classes of functions . To understand this success of PNNs , we conduct a spectral analysis , inspired by the analysis of DNNs . We focus on one instance of polynomial networks called Π-Nets [ 16 ] , where the output is a piece-wise polynomial function of the input obtained via multiplicative layers . The parameters of a Π-Net can be represented as high-order tensors , while polynomial expansions can offer increased representation power [ 17 ] . Our main contributions can be summarized as follows : 1 . We analyze two-layer polynomial networks in the NTK regime . By studying the spectral properties of the corresponding kernel , we prove a theoretical speed-up in learning higher frequencies over standard neural networks and validate the hypothesis in the approximate NTK regime , on the task of learning spherical harmonics . 2 . Beyond the NTK regime , we demonstrate this enhanced bias of Π-Nets towards higher frequencies in several experimental settings , beginning from synthetic learning tasks and then proceeding to state-of-art networks and inverse problems with 2D images . Aside from improving the understanding of polynomials in neural networks , our proposed analysis also sheds new light on the effect of multiplicative interactions in neural networks , prevalent in certain domains of machine learning including vision and natural language processing [ 29 , 7 ] . 2 RELATED WORK . Spectral Bias : Motivated by the empirical observations in [ 3 , 37 , 48 , 46 ] that deep networks first learn “ simple patterns ” , several papers [ 49 , 12 , 8 , 2 ] have conducted theoretical analyses to explain this bias towards lower frequencies . The work of [ 2 ] , aiming to understand why random labels take longer to learn than natural labels , showed that alignment of the labels with the eigenvectors of the NTK Gram matrix determines the learning speed . Extending this result , Cao et al . [ 12 ] provide an explanation for the spectral bias by analyzing the decay rate of eigenvalues of the NTK when the input data is uniformly distributed on the sphere . Under the same assumption , Basri et al . [ 8 ] study training dynamics in the NTK setting for 2-layer ReLU networks with a fixed outer layer and an explicitely included linear bias term in the ReLU . [ 9 ] further extends this work to account for non-uniform data distributions . All these findings show that DNNs learn lower frequency functions faster which prompted Tancik et al . [ 45 ] to propose methods to mitigate this bias . Our paper aims to establish similar results for PNNs , to explain their good performance at learning higher frequencies . Polynomial neural networks ( PNNs ) : The early papers that explore polynomials in the context of neural networks are mainly divided into two categories : 1 ) self-organizing networks with hard coded features [ 23 ] , 2 ) Pi-Sigma networks [ 41 , 34 ] . In both cases , the constructions did not scale well to higher dimensional inputs , and were not used for high-dimensional signals , such as images . More recent papers have used the Hadamard product to capture correlations between different branches of an architecture [ 4 , 44 , 47 , 17 ] . Our goal is to analyze the properties of these polynomial neural networks that have shown sucess in practice . Polynomial activation functions ( PAFs ) : It is important to note the distinction between PNNs and Polynomial activation functions . PAFs expand ( element-wise ) each feature to an rth degree , i.e. , they assume a ( deep ) neural network where the element-wise activation functions are rth degree polynomials . This is substantially different from capturing higher-order correlations across input ( or feature ) elements like PNNs especially in the presence of non-linear activations . Theoretical work on over-parametrization [ 18 ] , expressive power [ 30 ] and generalization of shallow nets [ 35 ] have emerged for PAFs . The aforementioned papers , however , do not conduct a spectral analysis and do not exhibit the benefits of PNNs for learning high-frequency information . 3 ANALYSIS OF POLYNOMIAL NEURAL NETWORKS IN THE NTK REGIME . In this section , we conduct a careful analysis of the kernel approximation of polynomial neural networks ( PNNs ) to gain insight on the effect of the multiplicative interactions . PNNs include multiplicative interactions and express high-degree polynomial expansions . The recent parametrization of the Π-Net family [ 16 ] , which we summarize below , is used as a representative PNN . We review the tangent kernel approximation of neural networks and then we derive the tangent kernel of two-layer Π-Nets to study the spectral bias of Π-Nets . Notation : We denote by x¨ , ¨y the standard inner-product on Rd . For two vectors x , y P Rd , x ˚ y denotes the element-wise or Hadamard product . We use ˆm to denote the mode-m vector product1 . 1The reader may refer to the appendix for more details on the mode-m product . We define the asymptotic notations Ωp¨q and Ω̃p¨q as follows : Let an and bn be two sequences . We write an “ Ωpbnq if lim infnÑ8 |an { bn| ą 0 . We use Ω̃p¨q to hide the logarithmic factors in Ωp¨q . 3.1 Π-NET FORMULATION . A polynomial expansion of the input vector z P Rδ can be used to express the output x P Ro as an N th degree polynomial expansion as follows : x “ N ÿ n “ 1 ˆ W rns n ` 1 ź j “ 2 ˆjz ˙ ` β , ( 1 ) where β P Ro and W rns P Roˆ śn m “ 1 ˆmd ( N n “ 1 are the learnable parameters , and ˆm is the modem vector product . Since the number of tensor parameters W rns grow exponentially with the degree of the polynomial , a coupled tensor decomposition with factor sharing can be used . The idea in Π-Nets is to propose such decompositions that can capture higher order correlations , and can be implemented in standard deep learning frameworks . One such decomposition uses the recursive formulation xn “ ´ ATrnsz ¯ ˚ ´ STrnsxn´1 ` B T rnsbrns ¯ for n “ 1 , . . . , N and expresses the output x as x “ CxN ` β . The term in the rightmost parenthesis is exactly the recursive form of a standard neural network ( without activations ) . Therefore , Π-Nets augment standard neural network with multiplicative interactions via the Hadamard product . While Π-Nets can approximate the target function without activations , they achieve state-of-art performance with activation functions , wherein the output is a piece-wise polynomial . We include more details in the Appendix . 3.2 THE NEURAL TANGENT KERNEL . Consider the following two-layer ReLU neural network ( without bias parameters ) with width m that assumes the following form ( defined as in [ 10 ] ) : fW pxq “ b 2 mW2σpW1xq , where W1 P Rmˆpd ` 1q , W2 P R1ˆm and we assume inputs txuni “ 1 follow some distribution τ on the unit sphere Sd P Rd ` 1 ; σp¨q denotes the element-wise ReLU operator . As the width of the network m goes to infinity , if the weights at initialization W p0q are independent and each follow the standard normal distribution , the inner product of the network gradient at initialization gives rise to a limiting kernel , namely the Neural Tangent Kernel ( NTK ) [ 24 ] κ defined as : κpx , x1q “ lim mÑ8 x∇W fW p0qpxq , ∇W fW p0qpx1qy . ( 2 ) This kernel has been used to characterize the behavior of sufficiently wide networks fW during training . For instance , if the network is trained to minimize the ` 2 loss , then its training dynamics closely track those of kernel regression under κ . This holds in a particular training regime , referred to as “ lazy training ” [ 14 ] , where the parameters hardly vary after initialization and the network can be approximated by its first-order Taylor expansion at initialization as : fW pxq « fW p0qpxq ` x∇W fW p0qpxq , W ´W p0qy . In practice however , the conditions of lazy training are often violated ( notably , the requirement that the weights do not move ) within the first few steps of gradient descent . Nevertheless , the NTK remains a useful theoretical tool for analyzing the neural network behavior as some of its predictions have been shown to hold in practice [ 12 , 32 ] . The NTK for the two-layer ReLU network fW pxq takes the following form [ 10 , 14 , 19 ] κpx , x1q “ 2xx , x1yκ1px , x1q ` 2κ2px , x1q , ( 3 ) where the kernels κ1 and κ2 are defined as follows : κ1px , x1q “ Ew „ Np0 , Iqrσ1pxw , xyqσ1pxw , x1yqs , κ2px , x1q “ Ew „ Np0 , Iqrσpxw , xyqσpxw , x1yqs , ( 4 ) where σp¨q denotes the ReLU operator and σ1p¨q denotes the indicator function 1r¨ ě 0s . The kernels κ1 , κ2 have been explicitly computed for the ReLU activation in [ 38 , 15 , 10 ] where they were shown to be positive semi-definite dot-product kernels . The function value depends only on the value of the dot-product of the arguments or more formally , κ1px , x1q “ g1pxx , x1yq and κ2px , x1q “ g2pxx , x1yq for some functions g1 , g2 : R Ñ R. These kernels will serve as building blocks for the tangent kernel of the Π-Net architecture studied in the next section .

Summary: The paper gives an analysis of polynomial networks in the NTK regime. They show a theoretical speed-up in learning higher frequencies when using polynomial networks. They also have some experiments that verify these properties.