A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case

1 INTRODUCTION . It has been argued for a while , and is becoming increasingly apparent in recent years , that in terms of complexity control and generalization in neural network training , “ the size [ magnitude ] of the weights is more important then the size [ number of weights or parameters ] of the network ” ( Bartlett , 1997 ; Neyshabur et al. , 2014 ; Zhang et al. , 2016 ) . That is , inductive bias and generalization are not achieved by limiting the size of the network , but rather by explicitly ( Wei et al. , 2019 ) or implicitly ( Nacson et al. , 2019 ; Lyu & Li , 2019 ) controlling the magnitude of the weights . In fact , since networks used in practice are often so large that they can fit any function ( any labels ) over the training data , it is reasonable to think of the network as virtually infinite-sized , and thus able to represent essentially all functions . Training and generalization ability then rests on fitting the training data while controlling , either explicitly or implicitly , the magnitude of the weights . That is , training searches over all functions , but seeks functions with small representational cost , given by the minimal weight norm required to represent the function . This “ representational cost of a function ” is the actual inductive bias of learning—the quantity that defines our true model class , and the functional we are actually minimizing in order to learn . Understanding learning with overparameterized ( virtually infinite ) networks thus rests on understanding this “ representational cost ” , which is the subject of our paper . Representational cost appears to play an important role in generalization performance ; indeed Mei & Montanari ( 2019 ) show that minimum norm solutions are optimal for generalization in certain simple cases , and recent work on “ double descent ” curves is an example of this phenomenon ( Belkin et al. , 2019 ; Hastie et al. , 2019 ) . We can also think of understanding the representational cost as asking an approximation theory question : what functions can we represent , or approximate , with our de facto model class , namely the class of functions representable with small magnitude weights ? There has been much celebrated work studying approximation in terms of the network size , i.e. , asking how many units are necessary in order to approximate a target function ( Hornik et al. , 1989 ; Cybenko , 1989 ; Barron , 1993 ; Pinkus , 1999 ) . But if complexity is actually controlled by the norm of the weights , and thus our true model class is defined by the magnitude of the weights , we should instead ask how large a norm is necessary in order to capture a target function . This revised view of approximation theory should also change how we view issues such as depth separation : rather then asking how increasing depth can reduce the number of units required to fit a function , we should instead ask how increasing depth can reduce the norm required , i.e. , how the representational cost we study changes with depth . Our discussion above directly follows that of Savarese et al . ( 2019 ) , who initiated the study of the representational cost in term of weight magnitude . Savarese et al . considered two-layer ( i.e. , single hidden layer ) ReLU networks , with an unbounded ( essentially infinite ) number of units , and where the overall Euclidean norm ( sum of squares of all the weights ) is controlled . ( Infinite width networks of this sort have been studied from various perspectives by e.g. , Bengio et al . ( 2006 ) ; Neyshabur et al . ( 2015 ) ; Bach ( 2017 ) ; Mei et al . ( 2018 ) ) . For univariate functions f : R → R , corresponding to networks with a single one-dimensional input and a single output , Savarese et al . obtained a crisp and precise characterization of the representational cost , showing that minimizing overall Euclidean norm of the weights is equivalent to fitting a function by controlling : max ( ∫ |f ′′ ( x ) |dx , |f ′ ( −∞ ) + f ′ ( +∞ ) | ) . ( 1 ) While this is an important first step , we are of course interested also in more than a single onedimensional input . In this paper we derive the representational cost for any function f : Rd → R in any dimension d. Roughly speaking , the cost is captured by : ‖f‖R ≈̇‖R { ∆ ( d+1 ) /2f } ‖1 ≈ ‖∂d+1b R { f } ‖1 ( 2 ) whereR is the Radon transform , ∆ is the Laplacian , and ∂b is a partial derivative w.r.t . the offset in the Radon transform ( see Section 3 for an explanation of the Radon transform ) . This characterization is rigorous for odd dimensions d and for functions where the above expressions are classically welldefined ( i.e. , smooth enough such that all derivatives are finite , and the integrand in the Radon transform is integrable ) . But for many functions of interest these quantities are not well-defined classically . Instead , in Definition 1 , we use duality to rigorously define a semi-norm ‖f‖R that captures the essence of the above quantities and is well-defined ( though possibly infinite ) for any f in any dimension . We show that ‖f‖R precisely captures the representational cost of f , and in particular is finite if and only if f can be approximated arbitrarily well by a bounded norm , but possibly unbounded width , ReLU network . Our precise characterization applies to an architecture with unregularized bias terms ( as in Savarese et al . ( 2019 ) ) and a single unregularized linear unit— otherwise a correction accounting for a linear component is necessary , similar but more complex than the term |f ′ ( −∞ ) + f ′ ( +∞ ) | in the univariate case , i.e. , ( 1 ) . As we uncover , the characterization of the representational cost for multivariate functions is unfortunately not as simple as the characterization ( 1 ) in the univariate case , where the Radon transform degenerates . Nevertheless , it is often easy to evaluate , and is a powerful tool for studying the representational power of bounded norm ReLU networks . Furthermore , as detailed in Section 5.5 , there is no kernel function for which the associated RKHS norm is the same as ( 2 ) ; i.e. , training bounded norm neural networks is fundamentally different from kernel learning . In particular , using our characterization we show the following : • All sufficiently smooth functions have finite representational cost , but the necessary degree of smoothness depends on the dimension . In particular , all functions in the Sobolev space W d+1,1 ( Rd ) , i.e. , when all derivatives up to order d + 1 are L1-bounded , have finite representational cost , and this cost can be bounded using the Sobolev norm . ( Section 5.1 ) • We calculate the representational cost of radial “ bumps ” , and show there are bumps with finite support that have finite representational cost in all dimensions . The representational cost increases as 1/ε for “ sharp ” bumps of radius ε ( and fixed height ) . ( Section 5.2 ) • In dimensions greater than one , we show a general piecewise linear function with bounded support has infinite representational cost ( i.e. , can not be represented with a bounded norm , even with infinite networks ) . ( Section 5.3 ) • We obtain a depth separation in terms of norm : we demonstrate a function that is representable using a depth three ReLU network ( i.e. , with two hidden layers ) with small norm weights , but can not be represented by any bounded-norm depth two ( single hidden layer ) ReLU network . As far as we are aware , this is the first depth separation result in terms of the norm required for representation . ( Section 5.4 ) Related Work Although the focus of most previous work on approximation theory for neural networks was on the number of units , the norm of the weights was often used as an intermediate step . However , this use does not provide an exact characterization of the representational cost , only a ( often very loose ) upper bound , and in particular does not allow for depth separation results where a lower bound is needed . See Savarese et al . ( 2019 ) for a detailed discussion , e.g. , contrasting with the work of Barron ( 1993 ; 1994 ) . The connection between the Radon transform and two-layer neural networks was previously made by Carroll & Dickinson ( 1989 ) and Ito ( 1991 ) , who used it to obtain constructive approximations when studying approximation theory in terms of network size ( number of units ) for threshold and sigmoidal networks . This connection also forms the foundation of ridgelet transform analysis of functions Candès & Donoho ( 1999 ) ; Candès ( 1999 ) . More recently , Sonoda & Murata ( 2017 ) used ridgelet transform analysis to study the approximation properties of two-layer neural networks with unbounded activation functions , including the ReLU . While working on this manuscript , we learned through discussions with Matus Telgarsky of his related parallel work . In particular , Bailey et al . ( 2019 ) ; Ji et al . ( 2019 ) obtained a calculation formula for the norm required to represent a radial function , paralleling our calculations in Section 5.2 , and used it to show that sufficiently smooth radial functions have finite norm in any dimension , and studied how this norm changes with dimension . 2 INFINITE WIDTH RELU NETWORKS . We repeat here the discussion of Savarese et al . ( 2019 ) defining the representational cost of infinitewidth ReLU networks , with some corrections and changes that we highlight . Consider the collection of all two-layer networks having an unbounded number of rectified linear units ( ReLUs ) , i.e. , all gθ : Rd → R defined by gθ ( x ) = k∑ i=1 ai [ w > i x− bi ] + + c , for all x ∈ Rd ( 3 ) with parameters θ = ( k , W = [ w1 , ... , wk ] , b = [ b1 , ... , bk ] > , a = [ a1 , ... , ak ] > , c ) , where the width k ∈ N is unbounded . Let Θ be the collection of all such parameter vectors θ . For any θ ∈ Θ we let C ( θ ) be the sum of the squared Euclidean norm of the weights in the network excluding the bias terms , i.e. , C ( θ ) = 1 2 ( ‖W ‖2F + ‖a‖2 ) = 1 2 k∑ i=1 ( ‖wi‖22 + |ai|2 ) , ( 4 ) and consider the minimal representation cost necessary to exactly represent a function f ∈ Rd → R R ( f ) : = inf θ∈Θ C ( θ ) s.t . f = gθ . ( 5 ) By the 1-homogeneity of the ReLU , it is shown in Neyshabur et al . ( 2014 ) ( see also Appendix A of Savarese et al . ( 2019 ) ) that minimizing C ( θ ) is the same as constraining the inner layer weight vectors { wi } ki=1 to be unit norm while minimizing the ` 1-norm of the outer layer weights a . Therefore , letting Θ′ be the collection of all θ ∈ Θ with each wi constrained to the unit sphere Sd−1 : = { w ∈ Rd : ‖w‖ = 1 } , we have R ( f ) = inf θ∈Θ′ ‖a‖1 s.t . f = gθ . ( 6 ) However , we see R ( f ) is finite only if f is exactly realizable as a finite-width two layer ReLU network , i.e. , f must be a continuous piecewise linear function with finitely many pieces . Yet , we know that any continuous function can be approximated uniformly on compact sets by allowing the number of ReLU units to grow to infinity . Since we are not concerned with the number of units , only their norm , we modify our definition of representation cost to capture this larger space of functions , and define1 R ( f ) : = lim ε→0 ( inf θ∈Θ′ C ( θ ) s.t . |gθ ( x ) − f ( x ) | ≤ ε ∀ ‖x‖ ≤ 1/ε and gθ ( 0 ) = f ( 0 ) ) ( 7 ) 1Our definition ofR ( f ) differs from the one given in Savarese et al . ( 2019 ) . We require |gθ ( x ) −f ( x ) | ≤ ε on the ball of radius 1/ε rather than all of Rd , and we additionally require gθ ( 0 ) = f ( 0 ) . These modifications are needed to ensure ( 7 ) and ( 9 ) are equivalent . Also , we note the choice of zero in the condition gθ ( 0 ) = f ( 0 ) is arbitrary and can be replaced with any point x0 ∈ Rd . In words , R ( f ) is the minimal limiting representational cost among all sequences of networks converging to f uniformly ( while agreeing with f at zero ) . Intuitively , if R ( f ) is finite this means f is expressible as an “ infinite-width ” two layer ReLU network whose outer-most weights are described by a density α ( w , b ) over all weight and bias pairs ( w , b ) ∈ Sd−1 × R. To make this intuition precise , let M ( Sd−1 × R ) denote the space of signed measures α defined on ( w , b ) ∈ Sd−1 × R with finite total variation norm ‖α‖1 = ∫ Sd−1×R d|α| ( i.e. , the analog of the L1-norm for measures ) , and let c ∈ R. Then we define the infinite-width two-layer ReLU network hα , c ( or “ infinite-width net ” for short ) by2 hα , c ( x ) : = ∫ Sd−1×R ( [ w > x− b ] + − [ −b ] + ) dα ( w , b ) + c ( 8 ) We prove in Appendix C that R ( f ) is equivalent to R ( f ) = min α∈M ( Sd−1×R ) , c∈R ‖α‖1 s.t . f = hα , c . ( 9 ) Hence , learning an unbounded width ReLU network gθ by fitting some loss functional L ( · ) while controlling the Euclidean norm of the weights C ( θ ) by minimizing inf θ∈Θ L ( gθ ) + λC ( θ ) ( 10 ) is effectively the same as learning a function f by controlling R ( f ) : min f : Rd→R L ( f ) + λR ( f ) . ( 11 ) In other words , R ( f ) captures the true inductive bias of learning with unbounded width ReLU networks having regularized weights . Our goal is then to calculate R ( f ) for any function f : Rd → R , and in particular characterize when it is finite in order to understand what functions can be approximated arbitrarily well with bounded norm but unbounded width ReLU networks .

Summary: In this paper, the author analysis the (approximate) function class generated by an infinite-width network when the Euclidean norm is bounded. They extend the work of Savarese et al. on the univariable function by introducing the Randon Transform and R-norm to this problem.  The authors finally prove that any function in Sobolev space could be (approximately) obtained by a bounded network. The results achieved implies some generalization performance analysis and the induction error. Also, according to the authors, the difference between R-norm and RKHS norm might lead to the distinct from neural networks and kernel methods.