Minimum Width for Universal Approximation

1 INTRODUCTION . The study of the expressive power of neural networks investigates what class of functions neural networks can/ can not represent or approximate . Classical results in this field are mostly focused on shallow neural networks . An example of such results is the universal approximation theorem ( Cybenko , 1989 ; Hornik et al. , 1989 ; Pinkus , 1999 ) , which shows that a neural network with fixed depth and arbitrary width can approximate any continuous function on a compact set , up to arbitrary accuracy , if the activation function is continuous and nonpolynomial . Another line of research studies the memory capacity of neural networks ( Baum , 1988 ; Huang and Babri , 1998 ; Huang , 2003 ) , trying to characterize the maximum number of data points that a given neural network can memorize . After the advent of deep learning , researchers started to investigate the benefit of depth in the expressive power of neural networks , in an attempt to understand the success of deep neural networks . This has led to interesting results showing the existence of functions that require the network to be extremely wide for shallow networks to approximate , while being easily approximated by deep and narrow networks ( Telgarsky , 2016 ; Eldan and Shamir , 2016 ; Lin et al. , 2017 ; Poggio et al. , 2017 ) . A similar trade-off between depth and width in expressive power is also observed in the study of the memory capacity of neural networks ( Yun et al. , 2019 ; Vershynin , 2020 ) . In search of a deeper understanding of the depth in neural networks , a dual scenario of the classical universal approximation theorem has also been studied ( Lu et al. , 2017 ; Hanin and Sellke , 2017 ; Johnson , 2019 ; Kidger and Lyons , 2020 ) . Instead of bounded depth and arbitrary width studied in classical results , the dual problem studies whether universal approximation is possible with a network of bounded width and arbitrary depth . A very interesting characteristic of this setting is that there exists a critical threshold on the width that allows a neural network to be a universal approximator . For example , one of the first results ( Lu et al. , 2017 ) in the literature shows that universal approximation of L1 functions from Rdx to R is possible for a width- ( dx + 4 ) RELU network , but impossible for a width-dx RELU network . This implies that the minimum width required for universal approximation lies between dx + 1 and dx + 4 . Subsequent results have shown upper/lower bounds on the minimum width , but none of the results has succeeded in a tight characterization of the minimum width . 1.1 WHAT IS KNOWN SO FAR ? . Before summarizing existing results , we first define function classes studied in the literature . For a domain X ⊆ Rdx and a codomain Y ⊆ Rdy , we define C ( X , Y ) to be the class of continuous ∗emails : sejun.park @ kaist.ac.kr , chulheey @ mit.edu , jaeho-lee @ kaist.ac.kr , jinwoos @ kaist.ac.kr functions from X to Y , endowed with the uniform norm : ‖f‖∞ : = supx∈X ‖f ( x ) ‖∞ . For p ∈ [ 1 , ∞ ) , we also define Lp ( X , Y ) to be the class of Lp functions from X to Y , endowed with the Lp-norm : ‖f‖p : = ( ∫ X ‖f ( x ) ‖ p p dx ) 1/p . The summary of known upper and lower bounds in the literature , as well as our own results , is presented in Table 1 . We use wmin to denote the minimum width for universal approximation . First progress . As aforementioned , Lu et al . ( 2017 ) show that universal approximation of L1 ( Rdx , R ) is possible for a width- ( dx + 4 ) RELU network , but impossible for a width-dx RELU network . These results translate into bounds on the minimum width : dx+1 ≤ wmin ≤ dx+4 . Hanin and Sellke ( 2017 ) consider approximation of C ( K , Rdy ) , where K ⊂ Rdx is compact . They prove that RELU networks of width dx + dy are dense in C ( K , Rdy ) , while width-dx RELU networks are not . Although this result fully characterizes wmin in case of dy = 1 , it fails to do so for dy > 1 . General activations . Later , extensions to activation functions other than RELU have appeared in the literature . Johnson ( 2019 ) shows that if the activation function ρ is uniformly continuous and can be uniformly approximated by a sequence of one-to-one functions , a width-dx network can not universally approximate C ( K , R ) . Kidger and Lyons ( 2020 ) show that if ρ is continuous , nonpolynomial , and continuously differentiable at some z with ρ′ ( z ) 6= 0 , then networks of width dx + dy + 1 with activation ρ are dense in C ( K , Rdy ) . Furthermore , Kidger and Lyons ( 2020 ) prove that RELU networks of width dx + dy + 1 are dense in Lp ( Rdx , Rdy ) . Limitations of prior arts . Note that none of the existing works succeeds in closing the gap between the upper bound ( at least dx + dy ) and the lower bound ( at most dx + 1 ) . This gap is significant especially for applications with high-dimensional codomains ( i.e. , large dy ) , arising for many practical applications of neural networks , e.g. , image generation ( Kingma and Welling , 2013 ; Goodfellow et al. , 2014 ) , language modeling ( Devlin et al. , 2019 ; Liu et al. , 2019 ) , and molecule generation ( Gómez-Bombarelli et al. , 2018 ; Jin et al. , 2018 ) . In the prior arts , the main bottleneck for proving an upper bound below dx + dy is that they maintain all dx neurons to store the input and all dy neurons to construct the function output ; this means every layer already requires at least dx + dy neurons . In addition , the proof techniques for the lower bounds only consider the input dimension dx regardless of the output dimension dy . 1.2 SUMMARY OF RESULTS . We mainly focus on characterizing the minimum width of RELU networks for universal approximation . Nevertheless , our results are not restricted to RELU networks ; they can be generalized to networks with general activation functions . Our contributions can be summarized as follows . • Theorem 1 states that the minimum width for RELU networks to be dense in Lp ( Rdx , Rdy ) is exactly max { dx + 1 , dy } . This is the first result fully characterizing the minimum width of RELU networks for universal approximation . In particular , the upper bound on the minimum width is significantly smaller than the best known result dx + dy + 1 ( Kidger and Lyons , 2020 ) . • Given the full characterization of wmin of RELU networks for approximating Lp ( Rdx , Rdy ) , a natural question arises : Is wmin also the same for C ( K , Rdy ) ? We prove that it is not the case ; Theorem 2 shows that the minimum width for RELU networks to be dense in C ( [ 0 , 1 ] , R2 ) is 3 . Namely , RELU networks of width max { dx + 1 , dy } are not dense in C ( K , Rdy ) in general . • In light of Theorem 2 , is it impossible to approximate C ( K , Rdy ) in general while maintaining width max { dx + 1 , dy } ? Theorem 3 shows that an additional activation comes to rescue . We show that if networks use both RELU and threshold activation functions ( which we refer to as STEP ) 1 , they can universally approximate C ( K , Rdy ) with the minimum width max { dx+1 , dy } . • Our proof techniques for tight upper bounds are not restricted to RELU networks . In Theorem 4 , we extend our results to general activation functions covered in Kidger and Lyons ( 2020 ) . 1.3 ORGANIZATION . We first define necessary notation in Section 2 . In Section 3 , we formally state our main results and discuss their implications . In Section 4 , we present our “ coding scheme ” for proving upper bounds on the minimum width in Theorems 1 , 3 and 4 . In Section 5 , we prove the lower bound in Theorem 2 by explicitly constructing a counterexample . Finally , we conclude the paper in Section 6 . We note that all formal proofs of Theorems 1–4 are presented in Appendix . 2 PROBLEM SETUP AND NOTATION . Throughout this paper , we consider fully-connected neural networks that can be described as an alternating composition of affine transformations and activation functions . Formally , we consider the following setup : Given a set of activation functions Σ , an L-layer neural network f of input dimension dx , output dimension dy , and hidden layer dimensions d1 , . . . , dL−12 is represented as f : = tL ◦ σL−1 ◦ · · · ◦ t2 ◦ σ1 ◦ t1 , ( 1 ) where t ` : Rd ` −1 → Rd ` is an affine transformation and σ ` is a vector of activation functions : σ ` ( x1 , . . . , xd ` ) = ( ρ1 ( x1 ) , . . . , ρd ` ( xd ` ) ) , where ρi ∈ Σ . While we mostly consider the cases where Σ is a singleton ( e.g. , Σ = { RELU } ) , we also consider the case where Σ contains both RELU and STEP activation functions as in Theorem 3 . We denote a neural network with Σ = { ρ } by a “ ρ network ” and a neural network with Σ = { ρ1 , ρ2 } by a “ ρ1+ρ2 network. ” We define the width w of f as the maximum over d1 , . . . , dL−1 . We use “ ρ networks ( or ρ1 + ρ2 networks ) of width w ” for denoting the collection of all ρ networks ( or ρ1 + ρ2 networks ) of width w having a finite number of layers . For describing the universal approximation of neural networks , we say ρ networks ( or ρ1+ρ2 networks ) of width w are dense in C ( X , Y ) if for any f∗ ∈ C ( X , Y ) and ε > 0 , there exists a ρ network ( or a ρ1+ρ2 network ) f of width w such that ‖f∗ − f‖∞ ≤ ε . Likewise , we say ρ networks ( or ρ1+ρ2 networks ) are dense in Lp ( X , Y ) if for any f∗ ∈ Lp ( X , Y ) and ε > 0 , there exists a ρ network ( or a ρ1+ρ2 network ) f such that ‖f∗ − f‖p ≤ ε . 3 MINIMUM WIDTH FOR UNIVERSAL APPROXIMATION . Lp approximation with RELU . We present our main theorems in this section . First , for universal approximation of Lp ( Rdx , Rdy ) using RELU networks , we give the following theorem . Theorem 1 . For any p ∈ [ 1 , ∞ ) , RELU networks of width w are dense in Lp ( Rdx , Rdy ) if and only if w ≥ max { dx + 1 , dy } . 1The threshold activation function ( i.e. , STEP ) denotes x 7→ 1 [ x ≥ 0 ] . 2For simplicity of notation , we let d0 : = dx and dL : = dy . This theorem shows that the minimum width wmin for universal approximation is exactly equal to max { dx + 1 , dy } . In order to provide a tight characterization of wmin , we show three new upper and lower bounds : wmin ≤ max { dx + 1 , dy } through a construction utilizing a coding approach , wmin ≥ dy through a volumetric argument , and wmin ≥ dx + 1 through an extension of the same lower bound for L1 ( Rdx , Rdy ) ( Lu et al. , 2017 ) . Combining these bounds gives the tight minimum width wmin = max { dx + 1 , dy } . Notably , using our new proof technique , we overcome the limitation of existing upper bounds that require width at least dx + dy . Our construction first encodes the dx dimensional input vectors into one-dimensional codewords , and maps the codewords to target codewords using memorization , and decodes the target codewords to dy dimensional output vectors . Since we construct the map from input to target using scalar codewords , we bypass the need to use dx + dy hidden nodes . More details are found in Section 4 . Proofs of the lower bounds are deferred to Appendix B . Uniform approximation with RELU . In Theorem 1 , we have seen the tight characterization wmin = max { dx + 1 , dy } for Lp ( Rdx , Rdy ) functions . Does the same hold for C ( K , Rdy ) , for a compact K ⊂ Rdx ? Surprisingly , we show that the same conclusion does not hold in general . Indeed , we show the following result , proving that width max { dx + 1 , dy } is provably insufficient for dx = 1 , dy = 2 . Theorem 2 . RELU networks of width w are dense in C ( [ 0 , 1 ] , R2 ) if and only if w ≥ 3 . Theorem 2 translates to wmin = 3 , and the upper bound wmin ≤ 3 = dx + dy is given by Hanin and Sellke ( 2017 ) . The key is to prove a lower bound wmin ≥ 3 , i.e. , width 2 is not sufficient . Recall from Section 1.1 that all the known lower bounds are limited to showing that width dx is insufficient for universal approximation . A closer look at their proof techniques reveals that they heavily rely on the fact that the hidden layers have the same dimensions as the input space . As long as the width w > dx , their arguments break because such a network maps the input space into a higher-dimensional space . Although only for dx = 1 and dy = 2 , we overcome this limitation of the prior arts and show that width w = 2 > dx is insufficient for universal approximation , by providing a counterexample . We use a novel topological argument which comes from a careful observation on the image created by RELU operations . In particular , we utilize the property of RELU that it projects all negative inputs to zero , without modifying any positive inputs . We believe that our proof will be of interest to readers and inspire follow-up works . Please see Section 5 for more details . Theorem 1 and Theorem 2 together imply that for RELU networks , approximating C ( K , Rdy ) requires more width than approximating Lp ( Rdx , Rdy ) . Interestingly , this is in stark contrast with existing results , where the minimum depth of RELU networks for approximating C ( K , Rdy ) is two ( Leshno et al. , 1993 ) but it is greater than two for approximating Lp ( Rdx , Rdy ) ( Wang and Qu , 2019 ) . Uniform approximation with RELU+STEP . While width max { dx + 1 , dy } is insufficient for RELU networks to be dense in C ( K , Rdy ) , an additional STEP activation function helps achieve the minimum width max { dx + 1 , dy } , as stated in the theorem below . Theorem 3 . RELU+STEP networks of width w are dense in C ( K , Rdy ) if and only if w ≥ max { dx+ 1 , dy } . Theorem 2 and Theorem 3 indicate that the minimum width for universal approximation is indeed dependent on the choice of activation functions . This is also in contrast to the classical results where RELU networks of depth 2 are universal approximators ( Leshno et al. , 1993 ) , i.e. , the minimum depths for universal approximation are identical for both RELU networks and RELU+STEP networks . Theorem 3 comes from a similar proof technique as Theorem 1 . Due to its discontinuous nature , the STEP activation can be used in our encoder to quantize the input without introducing uniform norm errors . Lower bounds on wmin can be proved in a similar way as Theorem 1 . General activations . Our proof technique for upper bounds in Theorems 1 and 3 can be easily extended to networks using general activations . Indeed , we prove the following theorem , which shows that adding a width of 1 is enough to cover the networks with general activations . Theorem 4 . Let ρ : R → R be any continuous nonpolynomial function which is continuously differentiable at some z with ρ′ ( z ) 6= 0 . Then , ρ networks of width w are dense in Lp ( K , Rdy ) for all p ∈ [ 1 , ∞ ) if w ≥ max { dx + 2 , dy + 1 } . Please notice that unlike other theorems , Theorem 4 only proves an upper bound wmin ≤ max { dx + 2 , dy + 1 } . We note that Theorem 4 significantly improves over the previous upper bound of width dx + dy + 1 by ( Kidger and Lyons , 2020 , Remark 4.10 ) .

Summary: The paper studies the minimal neural network width needed for universal approximation. While previous papers on the subject merely provided lower and upper bounds, this paper derives exact bounds on the minimal width. The paper considers both ReLU networks, as well as general activation functions, and examines approximation under both uniform norm and $L_p$ norm. In the case of functions with high-dimensional outputs, the derived bounds are better than previously thought ($d_x + d_y + 1$ to $\max(d_x + 1, d_y)$), with possible implications to practical  network design.