The Universal Approximation Power of Finite-Width Deep ReLU Networks

We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation ( Bölcskei et al. , 2018 ) of a wide class of functions , including polynomials , windowed sinusoidal functions , one-dimensional oscillatory textures , and the Weierstrass function , a fractal function which is continuous but nowhere differentiable . Together with the recently established universal approximation result for affine function systems ( Bölcskei et al. , 2018 ) , this demonstrates that deep neural networks approximate vastly different signal structures generated by the affine group , the Weyl-Heisenberg group , or through warping , and even certain fractals , all with approximation error decaying exponentially in the number of neurons . We also prove that in the approximation of sufficiently smooth functions finite-width deep networks require strictly fewer neurons than finite-depth wide networks . 1 INTRODUCTION . A theory establishing a link between the complexity of a neural network and the complexity of the function class to be approximated by the network was recently developed in Bölcskei et al . ( 2018 ) . Based on this framework , it was shown ( Bölcskei et al. , 2018 ) that all affine function classes are optimally representable by neural networks in the sense of Kolmogorov rate-distortion theory ( Donoho , 1993 ; Grohs , 2015 ) . Equivalently , this means that the approximation error decays exponentially in the number of neurons employed in the approximation . The present paper explores the question of whether the universality result established in Bölcskei et al . ( 2018 ) extends beyond affine function classes and answers it in the affirmative . Specifically , we consider the approximation of polynomials , windowed sinusoidal functions ( Gröchenig & Samarah , 2000 ; Gröchenig , 2001 ) , one-dimensional oscillatory textures according to Demanet & Ying ( 2007 ) , and the Weierstrass function , a fractal function which is continuous everywhere and differentiable nowhere . The central conclusion of this paper is that finite-width ReLU networks of depth scaling poly-logarithmically in the inverse of the approximation error lead to exponentially decaying approximation error for all these different signal structures . This result is established by building on a recent breakthrough in Yarotsky ( 2016 ) and recognizing that the width of networks approximating polynomials need not scale linearly in the degree of the polynomial , but can actually be finite independently of the degree . This insight will also allow a sharp statement on the benefit of depth ; specifically , we prove that in the approximation of sufficiently smooth functions finite-width deep networks require strictly fewer neurons than finite-depth wide networks . Notation . For the function f ( x ) : Rd → R , we define ‖f‖L∞ ( Ω ) : = inf { C ≥ 0 : |f ( x ) | ≤ C , for all x ∈ Ω } . For a vector b ∈ Rd , we let ‖b‖∞ : = maxi=1 , ... , d |bi| , similarly we write ‖A‖∞ : = maxi , j |Ai , j | for the matrix A ∈ Rm×n . We denote the identity matrix of size n × n by In . Throughout , log stands for the logarithm to base 2 . 2 SETUP AND BASIC RELU CALCULUS . We start by defining ReLU neural networks . Definition 2.1 . Let L , N0 , N1 , . . . , NL ∈ N. A map Φ : RN0 → RNL given by Φ ( x ) = { W2 ( ρ ( W1 ( x ) ) ) , L = 2 WL ( ρ ( WL−1 ( ρ ( . . . ρ ( W1 ( x ) ) ) ) ) ) , L ≥ 3 , ( 1 ) with affine linear maps W ` : RN ` −1 → RN ` , ` ∈ { 1 , 2 , . . . , L } , and the ReLU activation function ρ ( x ) = max ( x , 0 ) , x ∈ R , acting component-wise ( i.e. , ρ ( x1 , . . . , xN ) : = ( ρ ( x1 ) , . . . , ρ ( xN ) ) ) is called a ReLU neural network . The mapW ` corresponding to layer ` is given byW ` ( x ) = A ` x+b ` , with A ` ∈ RN ` ×N ` −1 and b ` ∈ RN ` . We define the network connectivity as the total number of nonzero entries in the matrices A ` , ` ∈ { 1 , 2 , . . . , L } , and the vectors b ` , ` ∈ { 1 , 2 , . . . , L } . The depth of the network or , equivalently , the number of layers is L ( Φ ) : = L and its width W ( Φ ) : = max ` =0 , ... , LN ` . We further denote by B ( Φ ) : = max ` =1 , ... , L max { ‖A ` ‖∞ , ‖b ` ‖∞ } the maximum absolute value of the weights in the network . We designate the class of ReLU networks Φ : Rd → RNL with no more than L layers , width no more than M , input dimension d , and output dimension NL by NNL , M , d , NL . Note that the connectivity of Φ is upper-bounded by LM ( M + 1 ) . For later use we record three technical results . We first record a technical lemma on the composition of neural networks.1 Lemma 2.2 . Let L1 , L2 , M1 , M2 , d1 , d2 , NL1 , NL2 ∈ N , Φ1 ∈ NNL1 , M1 , d1 , NL1 , and Φ2 ∈ NNL2 , M2 , d2 , NL2 with NL1 = d2 . Then , there exists a network Ψ ∈ NNL1+L2 , max { 2NL1 , M1 , M2 } , d1 , NL2 with B ( Ψ ) = max { B ( Φ1 ) , B ( Φ2 ) } , satisfying Ψ ( x ) = Φ2 ( Φ1 ( x ) ) , for all x ∈ Rd1 . Before we can formalize the concept of a linear combination of neural networks , we need a result that shows how to augment network depth while retaining the networks input-output relation Lemma 2.3 . Let L , M , K , d ∈ N , Φ1 ∈ NNL , M , d,1 , and K > L. Then , there exists a corresponding network Φ2 ∈ NNK , max { 2 , M } , d,1 such that Φ2 ( x ) = Φ1 ( x ) for all x ∈ Rd . Moreover , the weights of Φ2 consist of the weights of Φ1 and ±1 ’ s . The next result formalizes the concept of a linear combination of neural networks . Lemma 2.4 . Let N , Li , Mi , di ∈ N , ai ∈ R , Φi ∈ NNLi , Mi , di,1 , i = 1 , 2 , . . . , N , d = ∑N i=1 di . Then , there exist networks Φ1 ∈ NNL , M , d , N and Φ2 ∈ NNL , M , d,1 with L = maxi Li , and M ≤ ∑N i=1 max { 2 , Mi } satisfying Φ1 ( x ) = ( Φ1 ( x1 ) Φ2 ( x2 ) . . . ΦN ( xN ) ) T and Φ2 ( x ) = N∑ i=1 aiΦi ( xi ) , for all x = ( xT1 , x T 2 , . . . , x T N ) T ∈ Rd with xi ∈ Rdi , i = 1 , 2 , . . . , N . Moreover , the weights of Φ1 consist of the weights of the networks Φi , i = 1 , 2 , . . . , N , and ±1 ’ s . The weights of Φ2 consist of the weights of Φ1 and { a1 , a2 , ... , aN } . Remark 2.5 . Note that if the networks Φi in Lemma 2.4 have shared inputs , the resulting networks Φ1 and Φ2 will have fewer than d = ∑N i=1 di inputs . 3 APPROXIMATION OF POLYNOMIALS . This section shows how the multiplication operation and polynomials can be approximated to within error with ReLU networks of finite width and of depth poly-logarithmic in 1/ . We also note that the approximation results throughout the paper guarantee that the magnitude of the weights in the network does not grow faster than polynomially in the size of the domain over which approximation takes place . Although not shown here for space constraints , the combination of finite width , depth 1The proofs of the following two lemmata , along with all other proofs not provided in the paper , can be found in the supplement . scaling poly-logarithmically in 1/ , and weights growing no faster than polynomially guarantees rate-distortion optimality in the sense of Bölcskei et al . ( 2018 ) and hence exponential error decay . Previous results on the approximation of the multiplication operation and of polynomials through finite-width ReLU networks reported in Hanin & Sellke ( 2017 ) ; Yarotsky ( 2018 ) do not come with bounds on the network weights , have depth scaling polynomially in 1/ , and therefore do not allow to conclude rate-distortion optimality . The proof ideas for the results in this section are inspired by Yarotsky ( 2016 ) and by the “ sawtooth ” construction of Telgarsky ( 2015 ) . In contrast to Yarotsky ( 2016 ) , we consider networks without “ skip connections ” and of finite and explicitly specified width . Before starting with the approximation of x2 , we note that all our results apply to the multivariate case as well , but we restrict ourselves to the univariate case for simplicity of exposition . Proposition 3.1 . There exists a constant C > 0 such that for all ∈ ( 0 , 1/2 ) there is a network Φ ∈ NN∞,4,1,1 satisfying L ( Φ ) ≤ C log ( −1 ) , B ( Φ ) ≤ 4 , Φ ( 0 ) = 0 , and ‖Φ ( x ) − x2‖L∞ ( [ 0,1 ] ) ≤ . ( 2 ) With Proposition 3.1 we are now ready to show how ReLU networks can approximate the multiplication operation , which will then lead us to the approximation of arbitrary powers of x . Proposition 3.2 . There exists a constant C > 0 such that for allD ∈ R+ and ∈ ( 0 , 1/2 ) there is a network ΦD , ∈ NN∞,12,2,1 satisfying L ( ΦD , ) ≤ C log ( dDe2 −1 ) , B ( ΦD , ) ≤ max { 4 , 2dDe2 } , ‖ΦD , ( x , y ) − xy‖L∞ ( [ −D , D ] 2 ) ≤ , ( 3 ) and ΦD , ( 0 , x ) = ΦD , ( x , 0 ) = 0 , for all x ∈ R. The next result establishes that arbitrary polynomials can be approximated by ReLU networks of finite and explicitly specified width and of depth growing logarithmically in the inverse of the approximation error . In particular , the width of the approximating network does not grow with the degree of the polynomial as is the case in Yarotsky ( 2016 ) , Ding et al . ( 2018 ) , Liang & Srikant ( 2017 ) . This finite-width aspect is central to the approximation of sinusoidal functions by ReLU networks as described in the next section . Proposition 3.3 . There exists a constant C > 0 such that for all m ∈ N , A ∈ R+ , pm ( x ) =∑m i=0 aix i with maxi=0 , ... , m |ai| = A , D ∈ R+ , and ∈ ( 0 , 1/2 ) , there is a network Φpm , D , ∈ NN∞,16,1,1 satisfying L ( Φpm , D , ) ≤ Cm ( log ( dAe ) + log ( −1 ) + m log ( dDe ) + log ( m ) ) , B ( Φpm , D , ) ≤ max { A , 8dDe2m−2 } , and ‖Φpm , D , − pm‖L∞ ( [ −D , D ] ) ≤ . ( 4 ) Proof . We start by noting that for m = 1 the resulting affine function p1 ( x ) = a0 + a1x can be realized exactly , i.e. , with = 0 , by a network of depth L = 2 with W1 ( x ) = ( a1 −a1 ) x+ ( a0 −a0 ) andA2 = ( 1 −1 ) , b2 = 0 . The proof form ≥ 2 will be effected by realizing the monomials xk , k ≥ 2 , through iterative composition of multiplication networks and combining this with a construction which uses the network realizing xk not only as a building block in the network realizing xk+1 but also to construct the network approximating the partial sum ∑k i=0 aix i in parallel . We start by setting HkD , η : = dDek + η ∑k−2 s=0dDes , k ∈ N , and let ΦHkD , η , η , D ∈ R+ , k ∈ N , η ∈ ( 0 , 1/2 ) , be multiplication networks according to Proposition 3.2 . For D ∈ R+ , k ∈ N , η ∈ ( 0 , 1/2 ) , we then recursively define ΨkD , η according to Ψ0D , η ( x ) = 1 , Ψ1D , η ( x ) = x , and ΨkD , η ( x ) = ΦHk−1D , η , η ( x , Ψk−1D , η ( x ) ) , k ≥ 2 . Note that ΨkD , η ( x ) can be realized through a neural network for all k ∈ N thanks to Lemma 2.2 and the fact that , as already noted above for the case m = 1 , any affine function can be realized through a neural network . We first show by induction that ‖ΨkD , η ( x ) − xk‖L∞ ( [ −D , D ] ) ≤ η k−2∑ s=0 dDes , ( 5 ) for all η ∈ ( 0 , 1/2 ) , k ≥ 2 . The base case k = 2 follows from ‖Ψ2D , η ( x ) − x2‖L∞ ( [ −D , D ] ) = ‖ΦH1D , η , η ( x , x ) − x 2‖L∞ ( [ −D , D ] ) ≤ η . We proceed to establishing the induction step ( k − 1 ) → k. The induction assumption is ‖Ψk−1D , η ( x ) − x k−1‖L∞ ( [ −D , D ] ) ≤ η k−3∑ s=0 dDes . ( 6 ) Since ‖Ψk−1D , η ‖L∞ ( [ −D , D ] ) ≤ ‖xk−1‖L∞ ( [ −D , D ] ) +‖Ψ k−1 D , η ( x ) −xk−1‖L∞ ( [ −D , D ] ) ≤ H k−1 D , η , Proposition 3.2 implies that ‖ΨkD , η ( x ) − xk‖L∞ ( [ −D , D ] ) ≤ ‖ΦHk−1D , η , η ( x , Ψ k−1 D , η ( x ) ) − xΨ k−1 D , η ( x ) ‖L∞ ( [ −D , D ] ) + max [ −D , D ] |x|‖Ψk−1D , η ( x ) − x k−1‖L∞ ( [ −D , D ] ) ≤ η + dDeη k−3∑ s=0 dDes = η k−2∑ s=0 dDes , which completes the proof of the induction step . We are now ready to proceed to the construction of the network Φpm , D , approximating the polynomial pm ( x ) = ∑m i=0 aix i . To this end , we first note that the identity mapping x 7→ x and the linear combination x , y 7→ x + ai−1y are affine transformations and can thus be realized by a network of depth L = 2 . By Lemma 2.4 there hence exists a constant C2 such that for every m ≥ 2 , pm ( x ) =∑m ` =0 a ` x ` , i ∈ { 2 , 3 , . . . , m } , η ∈ ( 0 , 1/2 ) there is a network ϕipm , D , η ∈ NN∞,16,3,3 with L ( ϕipm , D , η ) ≤ C2 log ( dH i−1 D , ηe2η−1 ) , and B ( ϕipm , D , η ) ≤ max { 4 , 2dH i−1 D , ηe2 , maxi∈ { 0 , ... , m } |ai| } realizing the map ( x s y ) > → ( x s+ ai−1y ΦHi−1D , η , η ( x , y ) ) > . The statements in the following apply for all m ∈ N , A ∈ R+ , pm ( x ) = ∑m i=0 aix i with maxi=0 , ... , m |ai| ≤ A , D ∈ R+ , and ∈ ( 0 , 1/2 ) . The network Φpm , D , approximating the polynomial pm ( x ) = ∑m i=0 aix i is now constructed according to Φpm , D , ( x ) : = ( 0 1 am ) ϕ m pm , D , η ( ϕm−1pm , D , η ( . . . ϕ2pm , D , η ( ( 1 0 1 ) x+ ( 0 a0 0 ) ) ) ) , with η : = ( dAem2dDem ) −1 . This yields Φpm , D , ( x ) = m∑ i=0 aiΨ i D , η ( x ) , for allx ∈ R. Hence equation 5 implies∥∥∥Φpm , D , ( x ) − pm∥∥∥ L∞ ( [ −D , D ] ) ≤ m∑ i=0 |ai|‖ΨiD , η ( x ) − x i‖L∞ ( [ −D , D ] ) ≤ m∑ i=2 |ai| ( η i−2∑ s=0 dDes ) ≤ η max i∈ { 2 , ... , m } |ai| m∑ i=2 ( i− 1 ) dDei−2 ≤ Am2dDem−2η ≤ . Thanks to its compositional structure , the width of Φpm , D , equals the maximum width of the individual networks in the composition , i.e. , W ( Φpm , D , ) ≤ 16 . Since Hi−1D , η ≤ 2dDe m−1 , for i ≤ m , we further have L ( Φpm , D , ) ≤ m∑ i=2 L ( ϕipm , D , η ) ≤ m∑ i=2 C2 log ( dHi−1D , η e 2η−1 ) ≤ C2m ( log ( dAe ) + log ( −1 ) + ( 3m− 2 ) log ( dDe ) + 2 log ( m ) + 2 ) ≤ 4C2m ( log ( dAe ) + log ( −1 ) +m log ( dDe ) + log ( m ) ) . Finally , we note that B ( Φpm , D , ) = max { 1 , |a0| , |am| , max i∈ { 2,3 , ... , m } B ( ϕipm , D , η ) } ≤ max { A , 8dDe 2m−2 } . This finalizes the proof . We conclude this section with a result on ReLU networks approximating smooth functions with exponential accuracy . The proof of this statement , provided in the supplement , is based on the theory developed above . Definition 3.4 . For D ∈ R+ , let the set SD ⊆ C∞ ( [ −D , D ] , R ) be given by SD = { f ∈ C∞ ( [ −D , D ] , R ) : ‖f ( n ) ( x ) ‖L∞ ( [ −D , D ] ) ≤ n ! , for all n ∈ N0 } . ( 7 ) Lemma 3.5 . There exist constants C > 0 and a polynomial π such that for all D ∈ R+ , f ∈ SD , and ∈ ( 0 , 1/2 ) , there is a network Ψf , ∈ NN∞,23,1,1 satisfying L ( Ψf , ) ≤ CdDe ( log ( −1 ) ) 2 , B ( Ψf , ) ≤ max { 1/D , dDe } π ( −1 ) , and ‖Ψf , − f‖L∞ ( [ −D , D ] ) ≤ . ( 8 ) Note that for expositional simplicity Lemma 3.5 covers functions defined on symmetric intervals [ −D , D ] . Close inspection of the proof reveals , however , that for arbitrary intervals [ a , b ] and functions f ∈ C∞ ( [ a , b ] , R ) with ‖f ( n ) ‖L∞ ( [ a , b ] ) ≤ n ! , for all n ∈ N , the same statement ( with D replaced by b− a in the bounds on L ( Ψf , ) , B ( Ψf , ) ) holds .

Summary: The focus of this paper is to show that finite-width deep neural networks with fully connected layers and ReLU activations are rate-distortion optimal approximators of certain classes of functions, meaning the approximation error decays exponentially in the number of neurons in the network. The function classes explored in this paper are: 1-d polynomials (on bounded intervals), 1-d sinusoidal functions (on bounded intervals), and other 1-d functions built from compositions or linear combinations of these, such as the so-called class of “oscillatory textures” and a class of continuous but nowhere differentiable functions known as Weierstrass functions. Finally, the paper also shows that as the desired approximation accuracy goes to zero finite-width deep ReLU networks require asymptotically fewer neurons than finite-depth wide ReLU networks in approximating a broad class of smooth functions.