Minimum Width for Universal Approximation

1 INTRODUCTION . The study of the expressive power of neural networks investigates what class of functions neural networks can/ can not represent or approximate . Classical results in this field are mostly focused on shallow neural networks . An example of such results is the universal approximation theorem ( Cybenko , 1989 ; Hornik et al. , 1989 ; Pinkus , 1999 ) , which shows that a neural network with fixed depth and arbitrary width can approximate any continuous function on a compact set , up to arbitrary accuracy , if the activation function is continuous and nonpolynomial . Another line of research studies the memory capacity of neural networks ( Baum , 1988 ; Huang and Babri , 1998 ; Huang , 2003 ) , trying to characterize the maximum number of data points that a given neural network can memorize . After the advent of deep learning , researchers started to investigate the benefit of depth in the expressive power of neural networks , in an attempt to understand the success of deep neural networks . This has led to interesting results showing the existence of functions that require the network to be extremely wide for shallow networks to approximate , while being easily approximated by deep and narrow networks ( Telgarsky , 2016 ; Eldan and Shamir , 2016 ; Lin et al. , 2017 ; Poggio et al. , 2017 ) . A similar trade-off between depth and width in expressive power is also observed in the study of the memory capacity of neural networks ( Yun et al. , 2019 ; Vershynin , 2020 ) . In search of a deeper understanding of the depth in neural networks , a dual scenario of the classical universal approximation theorem has also been studied ( Lu et al. , 2017 ; Hanin and Sellke , 2017 ; Johnson , 2019 ; Kidger and Lyons , 2020 ) . Instead of bounded depth and arbitrary width studied in classical results , the dual problem studies whether universal approximation is possible with a network of bounded width and arbitrary depth . A very interesting characteristic of this setting is that there exists a critical threshold on the width that allows a neural network to be a universal approximator . For example , one of the first results ( Lu et al. , 2017 ) in the literature shows that universal approximation of L1 functions from Rdx to R is possible for a width- ( dx + 4 ) RELU network , but impossible for a width-dx RELU network . This implies that the minimum width required for universal approximation lies between dx + 1 and dx + 4 . Subsequent results have shown upper/lower bounds on the minimum width , but none of the results has succeeded in a tight characterization of the minimum width . 1.1 WHAT IS KNOWN SO FAR ? . Before summarizing existing results , we first define function classes studied in the literature . For a domain X ⊆ Rdx and a codomain Y ⊆ Rdy , we define C ( X , Y ) to be the class of continuous ∗emails : sejun.park @ kaist.ac.kr , chulheey @ mit.edu , jaeho-lee @ kaist.ac.kr , jinwoos @ kaist.ac.kr functions from X to Y , endowed with the uniform norm : ‖f‖∞ : = supx∈X ‖f ( x ) ‖∞ . For p ∈ [ 1 , ∞ ) , we also define Lp ( X , Y ) to be the class of Lp functions from X to Y , endowed with the Lp-norm : ‖f‖p : = ( ∫ X ‖f ( x ) ‖ p p dx ) 1/p . The summary of known upper and lower bounds in the literature , as well as our own results , is presented in Table 1 . We use wmin to denote the minimum width for universal approximation . First progress . As aforementioned , Lu et al . ( 2017 ) show that universal approximation of L1 ( Rdx , R ) is possible for a width- ( dx + 4 ) RELU network , but impossible for a width-dx RELU network . These results translate into bounds on the minimum width : dx+1 ≤ wmin ≤ dx+4 . Hanin and Sellke ( 2017 ) consider approximation of C ( K , Rdy ) , where K ⊂ Rdx is compact . They prove that RELU networks of width dx + dy are dense in C ( K , Rdy ) , while width-dx RELU networks are not . Although this result fully characterizes wmin in case of dy = 1 , it fails to do so for dy > 1 . General activations . Later , extensions to activation functions other than RELU have appeared in the literature . Johnson ( 2019 ) shows that if the activation function ρ is uniformly continuous and can be uniformly approximated by a sequence of one-to-one functions , a width-dx network can not universally approximate C ( K , R ) . Kidger and Lyons ( 2020 ) show that if ρ is continuous , nonpolynomial , and continuously differentiable at some z with ρ′ ( z ) 6= 0 , then networks of width dx + dy + 1 with activation ρ are dense in C ( K , Rdy ) . Furthermore , Kidger and Lyons ( 2020 ) prove that RELU networks of width dx + dy + 1 are dense in Lp ( Rdx , Rdy ) . Limitations of prior arts . Note that none of the existing works succeeds in closing the gap between the upper bound ( at least dx + dy ) and the lower bound ( at most dx + 1 ) . This gap is significant especially for applications with high-dimensional codomains ( i.e. , large dy ) , arising for many practical applications of neural networks , e.g. , image generation ( Kingma and Welling , 2013 ; Goodfellow et al. , 2014 ) , language modeling ( Devlin et al. , 2019 ; Liu et al. , 2019 ) , and molecule generation ( Gómez-Bombarelli et al. , 2018 ; Jin et al. , 2018 ) . In the prior arts , the main bottleneck for proving an upper bound below dx + dy is that they maintain all dx neurons to store the input and all dy neurons to construct the function output ; this means every layer already requires at least dx + dy neurons . In addition , the proof techniques for the lower bounds only consider the input dimension dx regardless of the output dimension dy . 1.2 SUMMARY OF RESULTS . We mainly focus on characterizing the minimum width of RELU networks for universal approximation . Nevertheless , our results are not restricted to RELU networks ; they can be generalized to networks with general activation functions . Our contributions can be summarized as follows . • Theorem 1 states that the minimum width for RELU networks to be dense in Lp ( Rdx , Rdy ) is exactly max { dx + 1 , dy } . This is the first result fully characterizing the minimum width of RELU networks for universal approximation . In particular , the upper bound on the minimum width is significantly smaller than the best known result dx + dy + 1 ( Kidger and Lyons , 2020 ) . • Given the full characterization of wmin of RELU networks for approximating Lp ( Rdx , Rdy ) , a natural question arises : Is wmin also the same for C ( K , Rdy ) ? We prove that it is not the case ; Theorem 2 shows that the minimum width for RELU networks to be dense in C ( [ 0 , 1 ] , R2 ) is 3 . Namely , RELU networks of width max { dx + 1 , dy } are not dense in C ( K , Rdy ) in general . • In light of Theorem 2 , is it impossible to approximate C ( K , Rdy ) in general while maintaining width max { dx + 1 , dy } ? Theorem 3 shows that an additional activation comes to rescue . We show that if networks use both RELU and threshold activation functions ( which we refer to as STEP ) 1 , they can universally approximate C ( K , Rdy ) with the minimum width max { dx+1 , dy } . • Our proof techniques for tight upper bounds are not restricted to RELU networks . In Theorem 4 , we extend our results to general activation functions covered in Kidger and Lyons ( 2020 ) . 1.3 ORGANIZATION . We first define necessary notation in Section 2 . In Section 3 , we formally state our main results and discuss their implications . In Section 4 , we present our “ coding scheme ” for proving upper bounds on the minimum width in Theorems 1 , 3 and 4 . In Section 5 , we prove the lower bound in Theorem 2 by explicitly constructing a counterexample . Finally , we conclude the paper in Section 6 . We note that all formal proofs of Theorems 1–4 are presented in Appendix . 2 PROBLEM SETUP AND NOTATION . Throughout this paper , we consider fully-connected neural networks that can be described as an alternating composition of affine transformations and activation functions . Formally , we consider the following setup : Given a set of activation functions Σ , an L-layer neural network f of input dimension dx , output dimension dy , and hidden layer dimensions d1 , . . . , dL−12 is represented as f : = tL ◦ σL−1 ◦ · · · ◦ t2 ◦ σ1 ◦ t1 , ( 1 ) where t ` : Rd ` −1 → Rd ` is an affine transformation and σ ` is a vector of activation functions : σ ` ( x1 , . . . , xd ` ) = ( ρ1 ( x1 ) , . . . , ρd ` ( xd ` ) ) , where ρi ∈ Σ . While we mostly consider the cases where Σ is a singleton ( e.g. , Σ = { RELU } ) , we also consider the case where Σ contains both RELU and STEP activation functions as in Theorem 3 . We denote a neural network with Σ = { ρ } by a “ ρ network ” and a neural network with Σ = { ρ1 , ρ2 } by a “ ρ1+ρ2 network. ” We define the width w of f as the maximum over d1 , . . . , dL−1 . We use “ ρ networks ( or ρ1 + ρ2 networks ) of width w ” for denoting the collection of all ρ networks ( or ρ1 + ρ2 networks ) of width w having a finite number of layers . For describing the universal approximation of neural networks , we say ρ networks ( or ρ1+ρ2 networks ) of width w are dense in C ( X , Y ) if for any f∗ ∈ C ( X , Y ) and ε > 0 , there exists a ρ network ( or a ρ1+ρ2 network ) f of width w such that ‖f∗ − f‖∞ ≤ ε . Likewise , we say ρ networks ( or ρ1+ρ2 networks ) are dense in Lp ( X , Y ) if for any f∗ ∈ Lp ( X , Y ) and ε > 0 , there exists a ρ network ( or a ρ1+ρ2 network ) f such that ‖f∗ − f‖p ≤ ε . 3 MINIMUM WIDTH FOR UNIVERSAL APPROXIMATION . Lp approximation with RELU . We present our main theorems in this section . First , for universal approximation of Lp ( Rdx , Rdy ) using RELU networks , we give the following theorem . Theorem 1 . For any p ∈ [ 1 , ∞ ) , RELU networks of width w are dense in Lp ( Rdx , Rdy ) if and only if w ≥ max { dx + 1 , dy } . 1The threshold activation function ( i.e. , STEP ) denotes x 7→ 1 [ x ≥ 0 ] . 2For simplicity of notation , we let d0 : = dx and dL : = dy . This theorem shows that the minimum width wmin for universal approximation is exactly equal to max { dx + 1 , dy } . In order to provide a tight characterization of wmin , we show three new upper and lower bounds : wmin ≤ max { dx + 1 , dy } through a construction utilizing a coding approach , wmin ≥ dy through a volumetric argument , and wmin ≥ dx + 1 through an extension of the same lower bound for L1 ( Rdx , Rdy ) ( Lu et al. , 2017 ) . Combining these bounds gives the tight minimum width wmin = max { dx + 1 , dy } . Notably , using our new proof technique , we overcome the limitation of existing upper bounds that require width at least dx + dy . Our construction first encodes the dx dimensional input vectors into one-dimensional codewords , and maps the codewords to target codewords using memorization , and decodes the target codewords to dy dimensional output vectors . Since we construct the map from input to target using scalar codewords , we bypass the need to use dx + dy hidden nodes . More details are found in Section 4 . Proofs of the lower bounds are deferred to Appendix B . Uniform approximation with RELU . In Theorem 1 , we have seen the tight characterization wmin = max { dx + 1 , dy } for Lp ( Rdx , Rdy ) functions . Does the same hold for C ( K , Rdy ) , for a compact K ⊂ Rdx ? Surprisingly , we show that the same conclusion does not hold in general . Indeed , we show the following result , proving that width max { dx + 1 , dy } is provably insufficient for dx = 1 , dy = 2 . Theorem 2 . RELU networks of width w are dense in C ( [ 0 , 1 ] , R2 ) if and only if w ≥ 3 . Theorem 2 translates to wmin = 3 , and the upper bound wmin ≤ 3 = dx + dy is given by Hanin and Sellke ( 2017 ) . The key is to prove a lower bound wmin ≥ 3 , i.e. , width 2 is not sufficient . Recall from Section 1.1 that all the known lower bounds are limited to showing that width dx is insufficient for universal approximation . A closer look at their proof techniques reveals that they heavily rely on the fact that the hidden layers have the same dimensions as the input space . As long as the width w > dx , their arguments break because such a network maps the input space into a higher-dimensional space . Although only for dx = 1 and dy = 2 , we overcome this limitation of the prior arts and show that width w = 2 > dx is insufficient for universal approximation , by providing a counterexample . We use a novel topological argument which comes from a careful observation on the image created by RELU operations . In particular , we utilize the property of RELU that it projects all negative inputs to zero , without modifying any positive inputs . We believe that our proof will be of interest to readers and inspire follow-up works . Please see Section 5 for more details . Theorem 1 and Theorem 2 together imply that for RELU networks , approximating C ( K , Rdy ) requires more width than approximating Lp ( Rdx , Rdy ) . Interestingly , this is in stark contrast with existing results , where the minimum depth of RELU networks for approximating C ( K , Rdy ) is two ( Leshno et al. , 1993 ) but it is greater than two for approximating Lp ( Rdx , Rdy ) ( Wang and Qu , 2019 ) . Uniform approximation with RELU+STEP . While width max { dx + 1 , dy } is insufficient for RELU networks to be dense in C ( K , Rdy ) , an additional STEP activation function helps achieve the minimum width max { dx + 1 , dy } , as stated in the theorem below . Theorem 3 . RELU+STEP networks of width w are dense in C ( K , Rdy ) if and only if w ≥ max { dx+ 1 , dy } . Theorem 2 and Theorem 3 indicate that the minimum width for universal approximation is indeed dependent on the choice of activation functions . This is also in contrast to the classical results where RELU networks of depth 2 are universal approximators ( Leshno et al. , 1993 ) , i.e. , the minimum depths for universal approximation are identical for both RELU networks and RELU+STEP networks . Theorem 3 comes from a similar proof technique as Theorem 1 . Due to its discontinuous nature , the STEP activation can be used in our encoder to quantize the input without introducing uniform norm errors . Lower bounds on wmin can be proved in a similar way as Theorem 1 . General activations . Our proof technique for upper bounds in Theorems 1 and 3 can be easily extended to networks using general activations . Indeed , we prove the following theorem , which shows that adding a width of 1 is enough to cover the networks with general activations . Theorem 4 . Let ρ : R → R be any continuous nonpolynomial function which is continuously differentiable at some z with ρ′ ( z ) 6= 0 . Then , ρ networks of width w are dense in Lp ( K , Rdy ) for all p ∈ [ 1 , ∞ ) if w ≥ max { dx + 2 , dy + 1 } . Please notice that unlike other theorems , Theorem 4 only proves an upper bound wmin ≤ max { dx + 2 , dy + 1 } . We note that Theorem 4 significantly improves over the previous upper bound of width dx + dy + 1 by ( Kidger and Lyons , 2020 , Remark 4.10 ) .

Summary: This paper wants to establish tight theoretical lower-bounds on the minimum width required by a ReLU neural network to approximate "almost all" functions up to epsilon where the distance of approximation is defined using the Lp-norm. The paper improves the previously known bounds which lied in the range of d+1 and d+4 to exactly d+1. One of the results of this paper is to establish that this bound is exactly d+1 (I am summarizing the result while ignoring some precision/nuances). This result holds only when the distance is measured using the p-norm where p is finite. In the infinity norm setting, they show that this minimum width actually is indeed not a universal approximator. However, to ensure that the claimed minimum width also holds in the infinity-norm setting, they modify the activation functions to also allow for the threshold functions which is 1 if larger than 0 and 0 otherwise.