Neural tangent kernels, transportation mappings, and universal approximation

1 MAIN RESULT AND OVERVIEW . Consider functions computed by a single ReLU layer , meaning x 7→ m∑ j=1 sjσ ( 〈 wj , x 〉 + bj ) , ( 1.1 ) where σ ( z ) : = max { 0 , z } . While shallow networks are celebrated as being universal approximators ( Cybenko , 1989 ; Funahashi , 1989 ; Hornik et al. , 1989 ) — they approximate continuous functions arbitrarily well over compact sets — what is more shocking is that gradient descent can learn the parameters to these networks , and they generalize ( Zhang et al. , 2016 ) . Working towards an understanding of gradient descent on shallow ( and deep ! ) networks , researchers began investigating the neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ; Du et al. , 2018 ; Allen-Zhu et al. , 2018 ) , which replaces a network with its linearization at initialization , meaning x 7→ √ m m∑ j=1 sj 〈 τj , x̃ 〉 σ′ ( 〈 w̃j , x̃ 〉 ) , where x̃ = ( x , 1 ) ∈ Rd+1 , w̃ = ( w , b ) ∈ Rd+1 ; ( 1.2 ) here each w̃j = ( wj , bj ) is frozen at Gaussian initialization ( henceforth the bias is collapsed in for convenience ) , and each transported weight τj is microscopically close to the corresponding initial weight w̃j , concretely ‖τj − w̃j‖ = O ( 1/ √m ) , where > 0 is a parameter and the scaling /√m is conventional in this literature ( Allen-Zhu et al. , 2018 ) . As eq . ( 1.2 ) is merely affine in the parameters , it is not outlandish that gradient descent can be analyzed . What is outlandish is firstly that gradient descent on eq . ( 1.1 ) with small learning rate will track the behavior of eq . ( 1.2 ) , and secondly the weights hardly change as a function ofm , specifically ‖τj − w̃j‖2 = O ( 1/ √m ) . Contributions . This work provides rates of function approximation for the NTK as defined in eq . ( 1.2 ) , moreover in the “ NTK setting ” : the transported weights must be near initialization , meaning ‖τj − w̃j‖ = Õ ( 1/ √m ) . In more detail : Continuous functions ( cf . Theorem 1.5 ) . The main theorem packages the primary tools here to say : the NTK can approximate continuous functions arbitrarily well , so long as the width is at least 1/δ10d , where δ depends on continuity properties of the target function ; moreover , the transports satisfy ‖τj − w̃j‖ = O ( 1/ √m ) , and the ReLU network ( eq . ( 1.1 ) ) and its linearization ( eq . ( 1.2 ) ) stay close . Re-using many parts of the proof , a nearly-optimal rate 1/δ2d is exhibited for ReLU networks in Theorem E.1 ; this gap between ReLU networks and their NTK poses a tantalizing gap for future work . Approximation via sampling of transport mappings . The first component of the proof of Theorem 1.5 , detailed in Section 2 , is a procedure which starts with an infinite width network , and describes how sampling introduces redundancy in the weights , and automatically leads to the desired microscopic transports ‖τj − w̃j‖ = O ( 1/ √m ) . As detailed in Theorem 2.1 , the error between the infinite width and sampled networks is Õ ( + 1/√m ) . In this way , the analysis provides another perspective on the scaling behavior and small weight changes of the NTK . Construction of transport mappings . The second component of the proof of Theorem 1.5 , detailed in Section 3 , is to construct explicit transport mappings for various types of functions . As detailed in Lemma 3.3 , approximating continuous functions proceeds by constructing an infinite width network not directly for the target function f , but instead its convolution f ∗Gα with a Gaussian Gα with tiny variance α2 . Care is needed in order to obtain a rate of the form 1/δO ( d ) , rather than , say , 1/δO ( d/δ ) . The main constructions are based on Fourier transforms . Rounding out the organization of this paper : this introduction will state the main summarizing result and its intuition , and then close with related work ; Section 4 will describe certain odds and ends for approximating continuous functions which were left out from the main tools in Section 2 and Section 3 ; Section 5 sketches abstract approaches to constructing transport mappings , including ones based on a corresponding Hilbert space ; Section 6 will conclude with open problems and related discussion . Proofs are sketched in the paper body , but details are deferred to the appendices . 1.1 BASIC NOTATION , INTUITION , AND MAIN RESULT . The NTK views networks as finite width realizations of intrinsically infinite width objects . In order to transport an infinite number of parameters away from their initialization , one option is to use a transport mapping T : Rd+1 → Rd+1 to show where weights should go : x 7→ Ew̃ 〈 T ( w̃ ) , Φ ( x ; w̃ ) 〉 = Ew̃ 〈 T ( w̃ ) , x̃ 〉 σ′ ( 〈w̃ , x̃〉 ) = Ew̃ 〈 T ( w̃ ) , x̃ 〉 1 [ 〈w̃ , x̃〉 ≥ 0 ] , where Φ ( x ; w̃ ) = x̃σ′ ( 〈w̃ , x̃〉 ) is a random feature representation of x ( Rahimi & Recht , 2008 ) . This abstracts the individual transported weights ( τj ) mj=1 from before into transported weights defined over arbitrary weights w̃ ∈ Rd+1 . These ( augmented ) weights w̃ = ( w , b ) ( with weight w ∈ Rd and bias b ∈ R ) will always be distributed according to a standard Gaussian with identity covariance , with G denoting the density and probability law simultaneously . A key message of this work , developed in Section 2 , is ( a ) the infinite width network can be sampled to give rates of approximation by finite width networks , ( b ) the microscopic adjustments of the NTK setting arise naturally from the sampling process ! Indeed , letting s ∈ { −1 , +1 } denote a uniformly distributed random sign , Ẽ w 〈 T ( w̃ ) , Φ ( x ; w̃ ) 〉 = Ẽ w , s 〈 s2T ( w̃ ) + sw̃ √ m , Φ ( x ; w̃ ) 〉 ∵ E s2 = 1 , E s = 0 ≈ 1 m m∑ j=1 〈 s2jT ( w̃j ) + sjw̃j √ m , Φ ( x ; w̃j ) 〉 sampling ( wj , sj ) = √ m m∑ j=1 〈 sjT ( w̃j ) √ m + w̃j , sjΦ ( x ; w̃j ) 〉 . ( 1.3 ) As highlighted by the bolded terms : increasing the width m corresponds to resampling , and allows the transported weights to be scaled down ! Indeed , the distance moved is O ( 1/ √m ) by construction . To this end , for convenience define τj : = T ( w̃j , sj ) : = sjT ( w̃j ) √ m + w̃j1 [ ‖w̃j‖ ≤ R ] , φj ( x ) : = Φ ( x ; w̃j , sj ) : = sj√ m Φ ( x ; w̃j ) = sj√ m x̃σ′ ( 〈 w̃j , x̃ 〉 ) , ( 1.4 ) where R is a truncation radius included for purely technical reasons . The transport mappings constructed in Section 3 satisfy B : = supw̃ ‖T ( w̃ ) ‖ < ∞ , and thus maxj ‖τj − w̃j‖ ≤ B/ √ m by construction as promised ( with high probability ) . The key message of Section 2 is to control the deviations of this process , culminating in Theorem 2.1 and also Theorem 1.5 below , which yields upper bounds on the width necessary to approximate infinite width networks . The notion of approximation here will follow ( Barron , 1993 ) and use the L2 ( P ) metric , where P is a probability measure on the ball { x ∈ Rd : ‖x‖ ≤ 1 } : ‖h‖L2 ( P ) = √∫ h ( x ) 2 dP ( x ) . Additionally ‖h‖L2 = √∫ h ( x ) 2 dx and ‖h‖L1 = ∫ |h ( x ) |dx will respectively denote the usual L2 and L1 metrics over functions on Rd . Theorem 1.5 ( Simplification of Theorems 2.1 and 4.3 ) . Let continuous function f : Rd → R be given , along with δ ∈ ( 0 , 1 ] so that |f ( x ) −f ( x′ ) | ≤ whenever ‖x−x′‖2 ≤ δ and max { ‖x‖ , ‖x′‖ } ≤ 1+δ . Let P be any probability distribution over ‖x‖ ≤ 1 . Then there exists a transport mapping T ( defining T and τj as in eq . ( 1.4 ) ) and associated scalars B : = sup w̃ ‖T ( w̃ ) ‖2 = Õ ( M5d ( 5d+9 ) /2 4δ5 ( d+1 ) ) , where M : = sup ‖x‖≤1+δ |f ( x ) | , so that with probability at least 1− 3η over Gaussian weights ( w̃j ) mj=1 and uniform signs ( sj ) mj=1 , then maxj ‖τj − w̃j‖ ≤ B/ √m , and∥∥∥∥∥∥f − m∑ j=1 〈 τj , φj ( · ) 〉∥∥∥∥∥∥ L2 ( P ) ≤ Õ ( [ B√ m + √ d ] √ ln ( 1/η ) ) , ∥∥∥∥∥∥ √m m∑ j=1 sjσ ( 〈 τj , x̃ 〉 ) − m∑ j=1 〈 τj , φj ( · ) 〉∥∥∥∥∥∥ L2 ( P ) ≤ Õ [ B2 m3/2 + B √ d m + B√ m + √ d ] √ ln ( 1/η ) . In words : given an arbitrary target function f and associated continuity parameter δ , width ( B/ ) 2 = Õ ( d5d+9/ 10δ10 ( d+1 ) ) suffices for error Õ ( ) , parameters are close to initialization , and the NTK and the original network behave similarly . The randomized construction does not merely give existence , but holds with high probability : the sampling process is thus in a sense robust , and may be used algorithmically ! As provided in Theorem 4.5 , elements of the proof of Theorem 1.5 can be extracted and converted into a direct approximation rate of continuous functions by ReLU networks , and the rate becomes Õ ( dd+2/ 2δ2d+2 ) . Since this rate is nearly tight , together these rates pose an interesting question : is there a purely approximation-theoretic gap between shallow ReLU networks and their NTK ? 1.2 RELATED WORK . Optimization literature ; the NTK . This work is motivated and inspired by the optimization literature , which introduced the NTK to study gradient descent in a variety of nearly-parallel works ( Jacot et al. , 2018 ; Du et al. , 2018 ; Du et al. , 2018 ; Allen-Zhu et al. , 2018 ; Arora et al. , 2019 ; Oymak & Soltanolkotabi , 2019 ; Li & Liang , 2018 ; Cao & Gu , 2019 ) . These works require the network width to be polynomial in n , the size of the training set ; by contrast , the analysis here studies closeness in function space , and the width instead scales with properties of the target function . One close relative to the present work is that of ( Chizat & Bach , 2019 ) , which provides an abstract proof scheme following the preceding works , and explains the microscopic change of the weights as a consequence of the scaling /√m . This is consistent with the resampling perspective here , as summarized in eq . ( 1.3 ) . Random features and the mean-field perspective . The random features perspective ( Rahimi & Recht , 2008 ) studies a related convex problem : similarly to the NTK , the activations σ′ ( 〈 w̃j , x̃ 〉 ) are held fixed , and what are trained are scalar weights aj ∈ R on each feature . The Fourier transport map construction used both for the NTK here in Theorem 1.5 and for shallow networks in Theorem E.1 proceeds by constructing exactly such a reweighting , and thus the present work also establishes universal approximation properties of random features . A related perspective is presented in the mean-field literature , which relate gradient descent on ( w̃j ) mj=1 to a Wasserstein flow in the space of distributions on these features ( Chizat & Bach ; Mei et al. , 2018 ) . The analysis here does not have any explicit ties to the mean-field literature , however it is interesting and suggestive that transport mappings appear in both . Approximation literature . The closest prior work is due to Barron ( 1993 ) , who gave good rates of approximation for functions f : Rd → R when the associated quantity ∫ ‖w‖ · |f̂ ( w ) |dw is small , where f̂ denotes the Fourier transform of f . The proofs in Section 3 will use elements from the proofs in ( Barron , 1993 ) , but with many distinct components , and thus it is interesting that the same quantity ∫ ‖w‖ · |f̂ ( w ) |dw arises once again . Like the work of ( Barron , 1993 ) , the present work also chooses to approximate in the L2 ( P ) metric . Standard classical works in this literature are general universal approximation guarantees without rates or attention to the weight magnitudes ( Cybenko , 1989 ; Hornik et al. , 1989 ; Funahashi , 1989 ; Leshno et al. , 1993 ) . The rate given here of roughly 1/δ2d in Theorem 4.5 does not seem to appear rigorously in prior work , though it is mentioned as a consequence of a proof in ( Mhaskar & Micchelli , 1992 ) , who also take the approach of approximation via Gaussian convolutions ; the use of convolutions is not only standard ( Wendland , 2004 ) , it is moreover classical , having been used in Weierstrass ’ s original proof ( Weierstrass , 1885 ) . Many related works use a RKHSes directly . Sun et al . ( 2018 ) prove universal approximation ( with rates ) via an RKHS , however they do not consider the NTK ( or the NTK setting of small weight changes ) . Bach ( 2017a ) ( see also ( Bach , 2017b ; Basri et al. , 2019 ; Bietti & Mairal , 2019 ) ) studies a variety of questions related to function fitting with the random features model , including establishing rates of approximation for Lipschitz functions on the surface of the sphere ( with a few further conditions ) ; the rates are better than those here ( roughly Θ ( 1/δd/2 ) ) , however they do not consider the NTK setting , meaning either the setting of small changes from initialization nor the linearization . Another close parallel work studies exact representation power of infinite width networks , developing representations for functions with Ω ( d ) derivatives ( Ongie et al. , 2019 ) ; similarly , the constructions here use an exact representation result for Gaussian convolutions , as developed in Section 3 . Regarding lower bounds from the literature , there are two lower bounds of the form 1/δd/2 for general shallow networks , not necessarily in the NTK setting ( Yarotsky , 2016 ; Bach , 2017a ) . Interestingly , Yarotsky ( 2016 ) also presents a lower bound of 1/δd for approximations whose parameters vary continuously with the target function ; this seems to hold for the Fourier constructions here in Section 3 , though an argument needs to be made for the sampling step .

Summary: the paper consider representational aspects of neural tangent kernels (NTKs). More precisely, recent literature on overparametrized neural networks has identified NTKs as a way to characterize the behavior of gradient descent on wide neural networks as fitting these types of kernels. This paper focuses on the representational aspect: namely that functions of appropriate "complexity" can be written as an NTK with parameters close to initialization (comparably close to what results on gradient descent get).