Eigenspace Restructuring:  a Principle of Space and Frequency in Neural Networks

1 INTRODUCTION . Learning in high dimensions is commonly believed to suffer from the curse of dimensionality , in which the number of samples required to solve the problem grows rapidly ( often polynomially ) with the dimensionality of the input . Nevertheless , modern neural networks often exhibit an astonishing power to tackle a wide range of highly complex and high-dimensional real-world problems , many of which were thought to be out-of-scope of known methods ( Krizhevsky et al. , 2012 ; Vaswani et al. , 2017 ; Devlin et al. , 2018 ; Silver et al. , 2016 ; Senior et al. , 2020 ; Kaplan et al. , 2020 ) . What are the mathematical principles that govern the astonishing power of neural networks ? This question perhaps is the most crucial research question in the theory of deep learning because such principles are also the keys to resolve fundamental questions in the practice of machine learning such as ( outof-distribution ) generalization ( Zhang et al. , 2021 ) , calibration ( Ovadia et al. , 2019 ) , interpretability ( Montavon et al. , 2018 ) , robustness ( Goodfellow et al. , 2014 ) . Unarguably , there can be more than one of such principles . They are related to one or more of the three basic ingredients of machine learning methods : the data , the model and the inference algorithm . Among them , the models , a.k.a . architectures of neural networks are the most crucial innovation in deep learning that set it apart from classical machine learning methods . More importantly , the current revolution in machine learning is initialized by the ( re- ) introduction of convolution-based architectures ( Krizhevsky et al. , 2012 ; Lecun , 1989 ) , and subsequent breakthroughs are often driven by the discovery or application of novel architectures ( Vaswani et al . ( 2017 ) ; Devlin et al . ( 2018 ) ) . As such , identifying and understanding fundamental roles of architectures are of great importance . In this paper , we take a step forwards by leveraging recent developments in overparameterized networks ( Poole et al . ( 2016 ) ; Daniely et al . ( 2016 ) ; Schoenholz et al . ( 2017 ) ; Lee et al . ( 2018 ) ; Matthews et al . ( 2018 ) ; Xiao et al . ( 2018 ) ; Jacot et al . ( 2018 ) ; Du et al . ( 2018 ) ; Novak et al . ( 2019a ) ; Lee et al . ( 2019 ) and many others . ) These developments have discovered an important connection between neural networks and kernel machines : the Neural Network Gaussian Process ( NNGP ) kernels and the neural tangent kernels ( NTKs ) . Under certain scaling limits , the former describes the distribution of the outputs of a randomly initialized network ( a.k.a . prior ) , and the latter can describe the network ’ s gradient descent dynamics . Although recent work ( Ghorbani et al. , 2019 ; Yang & Hu , 2020 ) has identified several limitations of using them in studying the feature learning dynamics of practical networks , we show that they do capture several crucial aspects of the architectural inductive biases . Our main contribution is an eigenspace restructuring theorem . It characterizes a mathematical connection between a network ’ s architecture and its learnability through a trade-off between space and frequency , providing novel insights behind the mystery power of deep CNNs ( more generally , hierarchical locality ( Deza et al. , 2020 ; Vasilescu et al. , 2021 ) . ) By frequency , we mean the degree ( order ) of the eigenfunction and by space , we mean the spatial distance among the eigenfunction ’ s interaction terms . We summarize our main contribution below ; see Fig . 1 . 1 . The learning order ( see Green Arrow in Fig . 1 ) of eigen-functions is governed by the learning index ( LI ) , the sum of the frequency index ( FI ) and the spatial index ( SI ) , which can be characterized precisely by the network ’ s topology . 2 . There is a trade-off between space and frequency : within a fixed ( compute/data ) budget , it is impossible to model generic Long-Range-High-Frequency interactions . MLPs can model Long-Range-Low-Frequency ( LRHF ) interactions but fail to model Short-Range-HighFrequency ( SRHF ) , while shallow CNNs ( S-CNNs ) are the opposite . Remarkably , deep CNNs ( D-CNNs ) can simultaneously model both and various interpolating interactions between them ( e.g. , Median-Range-Median-Frequency ( MRMF ) . ) 3 . In addition , high-resolution CNNs ( HS-CNNs , EfficientNet-type of model scaling ) further broaden the class of learnable functions to contain ( 1 ) extrapolation : Ultra-Long-RangeUltra-Low-Frequency and the dual interactions and ( 2 ) finer interpolations interactions . 4 . Finally , we verify the above claims empirically for neural kernel methods and finite-width networks using SGD + Momentum for dataset and networks of practical sizes . 2 LINEAR AND LINEARIZED MODELS . As a warm up exercise , we briefly go through the training dynamics of linear models . Let ( X , Y ) denote the inputs and labels , where X ⊆ Rd and Y ⊆ R. Assume J : Rd → Rn is a feature map and the task is to learn a linear function f ( x , θ ) = J ( x ) θ to minimize the MSE objective 1 2 ∑ ( x , y ) ∈ ( X , Y ) |f ( x , θ ) − y|2 . Let ℛ ( X , θ ) = f ( X , θ ) −Y be the residual of the predictions of X . Then the gradient flow dynamics can be written as d dt ℛ ( X , θ ) = −J ( X ) JT ( X ) ℛ ( X , θ ) ≡ −K ( X , X ) ℛ ( X , θ ) ( 1 ) Since the feature kernel K ( X , X ) = J ( X ) JT ( X ) is constant in time , the above ODE can be solved in closed form . Let m = |X | the cardinality of X and K̂ ( j ) /uj be the j-th eigenvalue/eigenvector of K ( X , X ) in descending order . By initializing θ = 0 at time t = 0 and denoting the projection by ηj = u T j ℛ ( X , 0 ) , the dynamics of the residual and the loss can be reduced to ℛ ( X , θt ) = ∑ j∈ [ m ] e−K̂ ( j ) tηjuj , L ( θt ) = 1 2 ∑ j∈ [ m ] e−2K̂ ( j ) tη2j ( 2 ) Therefore , to make the residual in uj smaller than some > 0 , the amount of time needed is t ≥ K̂ ( j ) −1 log 2 η2j /2 . The larger K̂ ( j ) is , the shorter amount of time it takes to learn uj . Although simple , linear models provide us with the most useful intuition behind the relation between “ eigenstructures '' and learning dynamics . 2.1 LINEARIZED NEURAL NETWORKS : NNGP KERNELS AND NT KERNELS . Let f ( θ , x ) be a general function , e.g . f is neural network parameterized by θ . Similarly , d dt ℛ ( X , θ ) = −J ( X ; θ ) JT ( X ; θ ) ℛ ( X , θ ) ≡ −K ( X , X ; θ ) ℛ ( X , θ ) . ( 3 ) However , the kernelK ( X , X ; θ ) depends on θ via the Jacobian J ( X ; θ ) of f ( X ; θ ) and evolves with time . The above system is unsolvable in general . However , under certain parameterization methods ( e.g . Sohl-Dickstein et al . ( 2020 ) ) and when the network is sufficient wide , this kernel does not change much during training and converges to a deterministic kernel called the NTK ( Jacot et al. , 2018 ) , K ( X , X ; θ ) → Θ ( X , X ) as width→∞ . ( 4 ) The residual dynamics becomes a constant coefficient ODE again ℛ̇ ( X , θ ) = −Θ ( X , X ) ℛ ( X , θ ) . To solve this system , we need the initial value of ℛ ( X , θ ) . Since the parameters θ are often initialized using iid standard Gaussian variables , as the width approach infinity , the logits f ( X ; θ ) converge to a Gaussian process ( GP ) , known as the neural network Gaussian process ( NNGP ) . Specifically , f ( X ; θ ) ∼ N ( 0 ; K ( X , X ) ) , where K is the NNGP kernel . Note that one can also treat infinite-width networks as Bayesian models , a.k.a . Bayesian Neural Networks , and apply Bayesian inference to compute the posteriors . This approach is equivalent to training only the network ’ s classification layer ( Lee et al. , 2019 ) and the gradient descent dynamics is described by the kernel K. As such , there are two natural kernels , the NTK Θ and the NNGP kernel K , associated to infinitewidth networks , whose training dynamics are governed by constant coefficient ODEs . To make progress , it is tempting to apply Mercer ’ s Theorem to eigendecompose Θ and K , e.g. , K ( x , x̄ ) = ∑ K̂ ( j ) φj ( x ) φj ( x̄ ) and Θ ( x , x̄ ) = ∑ Θ̂ ( j ) ψj ( x ) ψj ( x̄ ) ( 5 ) One advantage of applying this decomposition is that it has almost no constraint on the kernels and the inputs . However , this decomposition is too coarse to be useful since it can hardly provide fine-grained information about the eigenstructures . E.g , it is not clear what are the corrections to Eq . ( 5 ) when changing the architecture from a 2-layer CNN to a 4-layer CNNs . For this reason , we choose to work on the product space of hyperspheres , which has richer mathematical structures . Our primary goal is to characterize the analytical dependence of the decomposition Eq . ( 5 ) on the network ’ s topology in the high-dimensional limit . 3 NEURAL COMPUTATIONS ON DAGS . Notations will become heavier starting from this section . In particular , we rely crucially on the directed acyclic graphs ( DAGs ) and minimal spanning trees ( MSTs ) to define the spatial complexity of eigenfunctions . In Sec . B , we provide a toy example to help understand the motivation . For a positive integer p , let Sp−1 denote the unit sphere in Rp and Sp−1 = √ pSp−1 , the sphere of radius √ p in Rp . We introduce the normalized sum ( integral ) x∈X f ( x ) ≡ |X|−1 ∑ x∈X f ( x ) ( x∈X f ( x ) ≡ µ ( X ) −1 ˆ x∈X f ( x ) µ ( dx ) ) ( 6 ) where X is a finite set ( a measurable set with a finite positive measure µ ) . We find it more convenient to express the computations in neural networks , and in neural kernels via DAGs ( Daniely et al. , 2016 ) , as both computations are of recursive nature . The associated DAG of a network can be thought of as the same network by setting all its widths ( or the number of channels for CNNs ) to 1 . Let G = ( N , E ) denote a DAG , where N and E are the nodes and edges , resp . We always assume the graph to have a unique output node oG and is an ancestor of all other nodes . Denote N0 ⊆ N the set of input nodes ( leaves ) of G , i.e. , the collection nodes without a child . Each node u ∈ N is associated with a pointwise function φu : R → R , which is normalized in the sense Ez∈N ( 0,1 ) φ2u ( z ) = 1 . It induces a function φ∗u : I ≡ [ −1 , 1 ] → I defined to be φ∗u ( t ) = E ( z1 , z2 ) ∈Ntφu ( z1 ) φu ( z2 ) . Here Nt denotes a pair of standard Gaussians with correlation t. We associate each u ∈ N a finite-dimensional Hilbert space Hu , and each uv ∈ E a bounded linear operator Luv : Hv → Hu . Let X ≡ ∏ u∈N0 X u ≡ ∏ u∈N0 Sdim ( Hu ) −1 ⊆ ∏ u∈N0 Hu and I = I |N0| be the input tensors and the input correlations to the graph G , resp . We associate two types of computations to a DAG : finite-width neural network computation and kernel computation , NG : X → HoG and KG : I → I , ( 7 ) resp . They are defined recursively as follows Nu ( x ) = φu ( ∑ v : uv∈E Luv ( Nv ( x ) ) ) if u /∈ N0 else Nu ( x ) = xu ( 8 ) Ku ( t ) = φ∗u ( v : uv∈E Kv ( t ) ) if u /∈ N0 else Ku ( t ) = tu ( 9 ) where x ∈ X and t ∈ I . The outputs of the computations are NG ( x ) = NoG ( x ) and KG ( t ) = KoG ( t ) . Note that KG is indeed the NNGP kernel . The NTK can also be written recursively as Θu ( t ) = φ̇ ∗ u ( v : uv∈E Kv ( t ) ) v : uv∈E ( Kv ( t ) + Θv ( t ) ) with ΘG = ΘoG . ( 10 ) Here , Θu = 0 if u ∈ N0 and φ̇∗u is the derivative of φ∗u .

Summary: The paper categorizes architectural biases from a frequency point of view through the lens of NTKs and NNGPs. The two main quantities are the frequency index (FI) and the spatial index (SI) which are calculated from the DAGs. The authors claim that these two quantities capture what type of functions (in terms of frequency and space) can be learned by specific architectures: i.e., MLPs learn LRLF functions, s-CNNs learn SRHF functions, and so on. The main theoretical contribution is Theorem 1: the authors give an architecture-dependent spectral decomposition formula for the NTK and NNGP Kernels using DAGs.