Tighter Sparse Approximation Bounds for ReLU Neural Networks

A well-known line of work ( Barron , 1993 ; Breiman , 1993 ; Klusowski & Barron , 2018 ) provides bounds on the width n of a ReLU two-layer neural network needed to approximate a function f over the ball BR ( Rd ) up to error , when the Fourier based quantityCf = ∫ Rd ‖ξ‖ 2|f̂ ( ξ ) | dξ is finite . More recently Ongie et al . ( 2019 ) used the Radon transform as a tool for analysis of infinite-width ReLU two-layer networks . In particular , they introduce the concept of Radon-based R-norms and show that a function defined on Rd can be represented as an infinite-width twolayer neural network if and only if its R-norm is finite . In this work , we extend the framework of ( Ongie et al. , 2019 ) and define similar Radon-based semi-norms ( R , U-norms ) such that a function admits an infinite-width neural network representation on a bounded open set U ⊆ Rd when itsR , U-norm is finite . Building on this , we derive sparse ( finite-width ) neural network approximation bounds that refine those of Breiman ( 1993 ) ; Klusowski & Barron ( 2018 ) . Finally , we show that infinite-width neural network representations on bounded open sets are not unique and study their structure , providing a functional view of mode connectivity . 1 INTRODUCTION . Extensive work has shown that for a neural network to be able to generalize , the size or magnitude of the parameters is more important than the size of the network , when the latter is large enough ( Bartlett , 1997 ; Neyshabur et al. , 2015 ; Zhang et al. , 2016 ) . Under certain regimes , the size of the neural networks used in practice is so large that the training data is fit perfectly and an infinite-width approximation is appropriate . In this setting , what matters to obtain good generalization is to fit the data using the right inductive bias , which is specified by how network parameters are controlled ( Wei et al. , 2020 ) together with the training algorithm used ( Lyu & Li , 2020 ) . The infinite-width two-layer neural network model has been studied from several perspectives due to its simplicity . One can replace the finite-width ReLU network 1n ∑n i=1 ai ( 〈ωi , x〉 − bi ) + by an integral over the parameter space with respect to a signed Radon measure : ∫ ( 〈ω , x〉− b ) + dα ( ω , b ) . Thus , controlling the magnitude of the neural network parameters is akin to controlling the measure α according to a certain norm . Bach ( 2017 ) introduced the F1-space , which is the infinitewidth neural network space with norm inf { ∫ |b| d|α| ( ω , b ) } , derived from the finite-width regular- izer 1n ∑n i=1 |ai|‖ ( ωi , bi ) ‖2 ( the infimum is over all the measures α which represent the function at hand ) . A different line of work ( Savarese et al. , 2019 ; Ongie et al. , 2019 ) consider the infinite-width spaces with norm inf { ‖α‖TV = ∫ d|α| ( ω , b ) } , which is derived from the finite-width regularizer 1 n ∑n i=1 |ai|‖ωi‖2 ( i.e . omitting the bias term ) . Both of these works seek to find expressions for this norm , leading to characterizations of the functions that are representable by infinite-width networks . Savarese et al . ( 2019 ) solves the problem in the one-dimensional case : they show that for a function f on R , this norm takes value max { ∫ R |f ′′ ( x ) | dx , |f ′ ( −∞ ) + f ′ ( ∞ ) | } . Ongie et al . ( 2019 ) give an expression for this norm ( the R-norm ) for functions on Rd , making use of Radon transforms ( see Subsec . 2.3 ) . Although we mentioned in the first paragraph that in many occasions the network size is large enough that the specific number of neurons is irrelevant , when the target function is hard to approx- imate it is interesting to have an idea of how many neurons one needs to approximate it . The first contribution in this direction was by Cybenko ( 1989 ) ; Hornik et al . ( 1989 ) , which show that twolayer neural networks with enough neurons can approximate any reasonable function on bounded sets in the uniform convergence topology . Later on , Barron ( 1993 ) ; Breiman ( 1993 ) provided sparse approximation bounds stating that if a function f is such that a certain quantity Cf constructed from the Fourier transform f̂ is finite , then there exists a neural network of width n such that the L2 approximation error with respect to a distribution of bounded support is lower than O ( Cf/n ) . More recently , Klusowski & Barron ( 2018 ) provided alternative sparse approximation bounds of Breiman ( 1993 ) by restricting to networks with bounded weights and a slightly better dependency on n at the expense of a constant factor increasing with d ( see Subsec . 2.2 ) . Contributions . In our work , we seek to characterize the functions that coincide with an infinitewidth two-layer neural network on a fixed bounded open set . This endeavor is interesting in itself because in practice , we want to learn target functions for which we know samples on a bounded set , and we are typically unconcerned with the values that the learned functions take at infinity . Moreover , the tools that we develop allow us to derive state-of-the-art sparse approximation bounds . Our main contributions are the following : • In the spirit of theR-norm introduced by Ongie et al . ( 2019 ) , for any bounded open set U ⊆ Rd we define theR , U-norm of a function on Rd , and show that when theR , U-norm of f is finite , f ( x ) can admits a representation of the form ∫ Sd−1×R ( 〈ω , x〉 − b ) + dα ( ω , b ) + 〈v , x〉 + c for x ∈ U , where v ∈ Rd , c ∈ R and α is an even signed Radon measure . • Using theR , U-norm , we derive function approximation bounds for neural networks with a fixed finite width . We compute theR , U-norm of a function in terms of its Fourier representation , and show that it admits an upper bound by the quantity Cf . This shows that our approximation bound is tighter than the previous bound by Breiman ( 1993 ) , and meaningful in more instances ( e.g . for finite-width neural networks ) . We also showR , U-norm-based bounds analogous to the ones of Klusowski & Barron ( 2018 ) . • Setting U as the open unit ball of radius R , we show that neural network representations of f on U hold for multiple even Radon measures , which contrasts with the uniqueness result provided by Ongie et al . ( 2019 ) for the case of Rd . We study the structure of the sets of Radon measures which give rise to the same function on U . The non-uniqueness of the measure representing a measure could be linked to the phenomenon of mode connectivity . Additional related work . There have been other recent works which have used the Radon transform to study neural networks in settings different from ours ( Parhi & Nowak , 2021a ; Bartolucci et al. , 2021 ) . These two works consider the R-norm as a regularizer for an inverse problem , and proceed to prove representer theorems : there exists a solution of the regularized problem which is a twolayer neural network equal to the number of datapoints . Regarding infinite-width network spaces , E & Wojtowytsch ( 2020 ) present several equivalent definitions and provides a review . A well-known line of work ( Mei et al. , 2018 ; Chizat & Bach , 2018 ; Rotskoff & Vanden-Eijnden , 2018 ) studies the convergence of gradient descent for infinite-width two-layer neural networks . 2 FRAMEWORK . 2.1 NOTATION . Sd−1 denotes the ( d − 1 ) -dimensional hypersphere ( as a submanifold of Rd ) and BR ( Rd ) is the Euclidean open ball of radius R. For U ⊆ Rd measurable , the space C0 ( U ) of functions vanishing at infinity contains the continuous functions f such that for any > 0 , there exists compact K ⊆ U depending on f such that |f ( x ) | < for x ∈ U \ K. P ( U ) is the set of Borel probability measures , M ( U ) is the space of finite signed Radon measures ( which may be seen as the dual of C0 ( U ) ) . Throughout the paper , the term Radon measure refers to a finite signed Radon measure for shortness . If γ ∈ M ( U ) , then ‖γ‖TV is the total variation ( TV ) norm of γ. MC ( U ) denotes the space of complex-valued finite signed Radon measures , defined as the dual space of C0 ( U , C ) ( the space of complex-valued functions vanishing at infinity ) . We denote by S ( Rd ) the space of Schwartz functions , which contains the functions in C∞ ( Rd ) whose derivatives of any order decay faster than polynomials of all orders , i.e . for all k , p ∈ ( N0 ) d , supx∈Rd |xk∂ ( p ) ϕ ( x ) | < +∞ . For f ∈ L1 ( Rd ) , we use f̂ to denote the unitary Fourier transforms with angular frequency , defined as f̂ ( ξ ) = 1 ( 2π ) d/2 ∫ Rd f ( x ) e −i〈ξ , x〉dx . If f̂ ∈ L1 ( Rd ) as well , we have the inversion formula f ( x ) = 1 ( 2π ) d/2 ∫ Rd f̂ ( ξ ) e i〈ξ , x〉dx . The Fourier transform is a continuous automorphism on S ( Rd ) . 2.2 EXISTING SPARSE APPROXIMATION BOUNDS . One of the classical results of the theory of two-layer neural networks ( Breiman ( 1993 ) , building on ( Barron , 1993 ) ) states that given a probability measure p ∈ P ( BR ( Rd ) ) and a function f : BR ( Rd ) → R admitting a Fourier representation of the form f ( x ) = 1 ( 2π ) d/2 ∫ Rd e i〈ξ , x〉df̂ ( ξ ) , where f̂ ∈MC ( Rd ) is a complex-valued Radon measure such that Cf = 1 ( 2π ) d/2 ∫ Rd ‖ξ‖ 2 2 d|f̂ | ( ξ ) < +∞ , there exists a two-layer neural network f̃ ( x ) = 1n ∑n i=1 ai ( 〈x , ωi〉 − bi ) + such that∫ BR ( Rd ) ( f ( x ) − f̃ ( x ) ) 2 dx ≤ ( 2R ) 4C2f n . ( 1 ) These classical results do not provide bounds on the magnitude of the neural network weights . More recently , Klusowski & Barron ( 2018 ) showed similar approximation bounds for two-layer ReLU networks under additional l1 and l0 bounds on the weights ai , ωi . Namely , if C̃f = 1 ( 2π ) d/2 ∫ Rd ‖ξ‖ 2 1 d|f̂ | ( ξ ) < +∞ there exists a two-layer neural network f̃ ( x ) = a0 + 〈ω0 , x〉 + κ n ∑n i=1 ai ( 〈wi , x〉 − bi ) + with |ai| ≤ 1 , ‖ωi‖ ≤ 1 , bi ∈ [ 0 , 1 ] , and κ ≤ 2C̃f , and sup x∈ [ −1,1 ] d |f ( x ) − f̃ ( x ) | ≤ c C̃f √ d+ log n n−1/2−1/d , ( 2 ) where c is a universal constant . 2.3 REPRESENTATION RESULTS ON Rd BASED ON THE RADON TRANSFORM One defines Pd denotes the space of hyperplanes on Rd , whose elements may be represented by points in Sd−1 × R by identifying { x|〈ω , x〉 = b } with both ( ω , b ) and ( −ω , −b ) . Thus , functions on Pd are even functions on Sd−1 × R and we will use both notions interchangeably1 . The Radon transform and the dual Radon transform . If f : Rd → R is a function which is integrable over all the hyperplanes of Rd , we may define the Radon transformRf : Pd → R as Rf ( ω , b ) = ∫ { x|〈ω , x〉=b } f ( x ) dx , ∀ ( ω , b ) ∈ Sd−1 × R. That is , one integrates the function f over the hyperplane ( ω , b ) . If Φ : Pd → R is a continuous function , the dual Radon transformR∗Φ : Rd → R is defined as R∗Φ ( x ) = ∫ Sd−1 Φ ( ω , 〈ω , x〉 ) dω , ∀x ∈ Rd , where the integral is with respect to the Hausdorff measure over Sd−1 . R and R∗ are adjoint operators in the appropriate domains ( see Lemma 13 ) . The Radon inversion formula . When f ∈ C∞ ( Rd ) , one has that ( Theorem 3.1 , Helgason ( 2011 ) ) f = cd ( −∆ ) ( d−1 ) /2R∗Rf ( 3 ) where cd = 12 ( 2π ) d−1 and ( −∆ ) s/2 denotes the ( negative ) fractional Laplacian , defined via its Fourier transform as ̂ ( −∆ ) s/2f ( ξ ) = ‖ξ‖sf̂ ( ξ ) . 1Similarly , the space M ( Pd ) of Radon measures over Pd contains the even measures in M ( Sd−1 × R ) . If α ∈ M ( Pd ) , ∫ Sd−1×R ϕ ( ω , b ) dα ( ω , b ) is well defined for any measurable function ϕ on S d−1 × R , but∫ Pd ϕ ( ω , b ) dα ( ω , b ) is only defined for even ϕ. TheR-norm . Given a function f : Rd → R , Ongie et al . ( 2019 ) introduce the quantity ‖f‖R = { sup { −cd〈f , ( −∆ ) ( d+1 ) /2R∗ψ〉 | ψ ∈ S ( Sd−1 × R ) , ψ even , ‖ψ‖∞ ≤ 1 } if f Lipschitz +∞ otherwise . ( 4 ) They call it the R-norm of f , although it is formally a semi-norm . Here , the space S ( Sd−1 × R ) of Schwartz functions on Sd−1 × R is defined , in analogy with S ( Rd ) , as the space of C∞ functions ψ on Sd−1 × R which for any integers k , l ≥ 0 and any differential operator D on Sd−1 satisfy sup ( ω , b ) ∈Sd−1×R | ( 1 + |b|k ) ∂kb ( Dψ ) ( ω , b ) | < +∞ ( Helgason ( 2011 ) , p. 5 ) . Moreover , S ( Pd ) = { ψ ∈ S ( Sd−1 × R ) | ψ even } , which means the conditions on ψ in ( 4 ) can be written as ψ ∈ S ( Pd ) , ‖ψ‖∞ ≤ 1 . The finiteness of the R-norm indicates whether a function on Rd admits an exact representation as an infinitely wide neural network . Namely , Ongie et al . ( 2019 ) in their Lemma 10 show that ‖f‖R is finite if and only if there exists a ( unique ) even measure α ∈M ( Sd−1×R ) and ( unique ) v ∈ Rd , c ∈ R such that for any x ∈ Rd , f ( x ) = ∫ Sd−1×R ( 〈ω , x〉 − b ) + dα ( ω , b ) + 〈v , x〉+ c , ( 5 ) in which case , ‖f‖R = ‖α‖TV . Remark the following differences between this result and the bounds by ( Breiman , 1993 ; Klusowski & Barron , 2018 ) shown in equations ( 1 ) and ( 2 ) : ( i ) in ( 5 ) we have an exact representation with infinite-width neural networks instead of an approximation result with finite-width , ( ii ) in ( 5 ) the representation holds on Rd instead of a bounded domain . In our work , we derive representation results similar to the ones of Ongie et al . ( 2019 ) for functions defined on bounded open sets , which naturally give rise to sparse approximation results that refine those of ( Breiman , 1993 ; Klusowski & Barron , 2018 ) . One property that makes the Radon transform and its dual useful to analyze neural networks can be understood at a very high level via the following argument : if f ( x ) = ∫ Sd−1×R ( 〈ω , x〉 − b ) +ρ ( ω , b ) d ( ω , b ) + 〈v , x〉 + c for some smooth rapidly decreasing function ρ , then ∆f ( x ) =∫ Sd−1×R δ〈ω , x〉=bρ ( ω , b ) d ( ω , b ) = ∫ Sd−1 ρ ( ω , 〈ω , x〉 ) dω = ( R ∗ρ ) ( x ) . For a general function f of the form ( 5 ) , one has similarly that 〈∆f , ϕ〉 = 〈α , Rϕ〉 for any ϕ ∈ S ( Rd ) . This property relates the evaluations of the measure α to the function ∆f via the Radon transform , and is the main ingredient in the proof of Lemma 10 of Ongie et al . ( 2019 ) . While we also rely on it , we need many additional tools to deal with the case of bounded open sets .

Summary: This paper studies the approximation bounds for the 2-layer  ReLU network.  The authors extend the analysis framework of Ongie et al. (2019) and show the approximation bounds for the infinite-width network on a bounded open set as well as the bounds for sparse (i.e., finite-width) network that refine the similar results in the literature. At last, the authors show that the infinite-width neural network representations may not be unique on bounded open sets and provide a functional view of the mode connectivity. In particular, the authors refine the R-norm introduced by Ongie et al. (2019) to R, U-norm to tickle the bounded open set case.