Expressiveness of Neural Networks Having Width Equal or Below the Input Dimension

1 INTRODUCTION . In the course of the increasing popularity of deep neural networks for applications in technical , ecological and many other fields , there has been huge progress in the research on understanding the mathematical properties of the mathematical mapping implemented by a deep neural network . The approximation properties , or expressiveness , of neural network functions have attracted intense interest . The central result in this field is the classic universal approximation theorem , which states that any continuous function can be approximated with arbitrary accuracy ( in terms of uniform approximation or Lp norms ) by neural network functions that have only one hidden layer for nearly every activation function ( Cybenko , 1989 ; Hornik , 1991 ) . In recent years , these kind of results have become more precise , for instance , in terms of estimates on the order of magnitude of parameters needed to achieve an approximation of a certain accuracy ( Petersen & Voigtlaender , 2018 ; Yarotsky , 2017 ; 2018 ) , or by investigating the number off piece-wise linear regions ( Montufar et al. , 2014 ; Serra et al. , 2018 ; Hanin & Rolnick , 2019 ) ; we refer to ( DeVore et al. , 2020 ) for an overview on recent developments . Since empirical observations showed that depth has a significant impact on the performance of neural networks , a lot of research has been dedicated to the effect of depth on the expressive power of neural networks ( Telgarsky , 2016 ; Mhaskar & Poggio , 2016 ; Montufar et al. , 2014 ; Raghu et al. , 2017 ; Rolnick & Tegmark , 2018 ; Lin et al. , 2017 ; Cohen et al. , 2016 ) . Besides this , the role of width in expressiveness of network functions has been investigated ( Lu et al. , 2017 ; Hanin & Sellke , 2017 ; Hanin , 2019 ; Johnson , 2018 ; Kidger & Lyons , 2020 ; Park et al. , 2021 ) . An important result states that a width larger than the input dimension is needed to allow universal approximation on arbitrary compact sets . It has also been shown that the capability to learn disconnected or bounded decision regions depends on whether the width is larger than the input dimension ( Nguyen et al. , 2018 ; Beise et al. , 2021 ) . In a further line of research , the capabilities of neural networks to memorize finite sample sets is investigated ( Yun et al. , 2019 ; Vershynin , 2020 ; Bubeck et al. , 2020 ; Park et al. , 2020 ) , which provides mathematical foundations to the surprising findings that neural networks can interpolate the training data ( Zhang et al. , 2017 ) in practise . In this work , we investigate what kind of subsets M ⊂ Rn admit universal approximation by neural network functions with a maximum layer width less than or equal to the input dimension , i.e . network functions that violate the necessary condition for universal approximation given in Hanin ( 2019 ) and Park et al . ( 2021 ) for arbitrary compact sets . Let us introduce the following notation : For a set D ⊂ Rn we denote by D◦ the set of interior points , by D its closure and by ∂D = D \D◦ the boundary of D. By ‖·‖ we mean the Euclidean norm . For a linear subspace U ⊂ Rn , we denote by dim ( U ) its dimension and by PU : Rn → U the orthogonal linear projection on U . We denote by ej the unit vector of the j-th coordinate axis in Rn . For some v ∈ Rn and j ∈ { 1 , ... , n } , we write v ( j ) for the j−th component of v. For some depth L ∈ N , we consider neural network functions of some , F : Rn0 → RnL where n0 is called the input dimension and nL the output dimension . Our network functions have the following form F : =WL ( AL−1 ◦ ... ◦A1 ) + bL ( 1 ) where Aj ( x ) = σ ( Wjx+ bj ) with Wj ∈ Rnj×nj−1 ( weights ) , bj ∈ Rnj ( bias ) , where j = 1 , ... , L , and σ : R → R the activation function . The application of σ and the inverse image σ−1 are understood to be applied element wise to vectors or subsets of Rn . The widely used activation function rectified linear unit , shortly ReLU , is defined by t 7→ max { t , 0 } . As a shorthand notation of the function implemented by the first k layers of a network function , we set Fk =Wk ( Ak−1 ◦ ... ◦A1 ) + bk , ( 2 ) for k ∈ { 1 , ... , L−1 } . We call nj the width of layer j = 1 , ... , L. The width of the network is defined as ω ( F ) = max { nj : j = 1 , ... , L } and L is called the depth of the network . For m , L ∈ N and an activation function σ , NN kσ ( m , L ) is the set of network functions F : Rn0 → Rk of the form ( 1 ) with activation σ and of maximum width m , i.e . ω ( F ) ≤ m , and depth L , i.e . L layers including the final linear layer . In case that Rk = R we omit the latter and write NN σ ( m , L ) instead , and in case the the depth is not specified , meaning that arbitrary depth is allowed , we write NN kσ ( m ) , NN σ ( m ) , respectively . We say that a compact subset M ⊂ Rn0 admits universal uniform approximation of functions in some function class of mappings from M to R by network function of NN σ ( m , L ) or NN σ ( m ) , if for every f : M → R in that class and every ε > 0 , there exists some F ∈ NN σ ( m , L ) , respectively F ∈ NN σ ( m ) , such that max x∈M |f ( x ) − F ( x ) | < ε . ( 3 ) 2 RELATED WORK . Approximation properties of width bounded neural networks have been studied in several papers . A common goal in these lines of research is to provide upper and lower bounds on the minimum width of the network , i.e . the minimum number of neurons per layers , here denoted by ωmin , needed for universal approximation in C ( M , RnL ) , i.e . the space of continuous functions from a compact set M to RnL endowed with the norm of uniform convergence , and Lp ( M , RnL ) with some set M ⊂ Rn0 , i.e . the space of functions f = ( f1 , ... , fnL ) from M to RnL such that every |fj |p is integrable over M , endowed with the usual Lp norm . Since in this work we are interested in universal approximation in C ( M , R ) , we refer to Lu et al . ( 2017 ) , Kidger & Lyons ( 2020 ) and Park et al . ( 2021 ) for the case of Lp approximation . For the case of C ( M , RnL ) , it was proven in Hanin & Sellke ( 2017 ) that n0 + 1 ≤ ωmin ≤ n0 + nL , for ReLU activation , ( 4 ) which has been tightened in Park et al . ( 2021 ) to ωmin ≥ max { n0 + 1 , nL } for ReLU and STEP activation , ( 5 ) where STEP refers to the threshold activation that maps to 1 on { x ≥ 0 } and 0 otherwise . It has been shown at the same time in Johnson ( 2018 ) and Beise et al . ( 2021 ) that ωmin ≥ n0 + 1 for ReLU + injective continuous activation . ( 6 ) The activation functions allowed in Johnson ( 2018 ) are slightly more general , as it is only required that they admit arbitrary accurate uniform approximation by injective continuous functions on arbitrary compact subset of R. In Kidger & Lyons ( 2020 ) the above upper bounds are extended to further classes of activation functions . For these results and a general overview on recent developments in this field , we refer to Park et al . ( 2021 ) . As we also formulate results for finite sets , our work partially exhibits relations to the research on finite sample memorization ( Yun et al. , 2019 ; Vershynin , 2020 ; Bubeck et al. , 2020 ; Park et al. , 2020 ) . In this work , we only consider width less than or equal to n0 and investigate uniform approximation on certain compacts subsets of the input space . According to the above result ( 6 ) , we can not expect universal approximation for all compact M ⊂ Rn0 . However , a common assumption in applications is that approximation is only needed on a certain subset . We derive some topological conditions on M that allow or exclude a kind of universal approximation under these circumstances . Although narrow neural networks , as they are considered in this work , are not common in practical applications , many networks exhibit a decaying layer width at the later layers . From this perspective , our results give theoretical insights on the kind feature extraction earlier layers need to implement in order to allow the later layers to solve a given machine learning task . Our main contributions are as follows : 1 . A maximum principle is given for F ∈ NN σ ( n0 ) with σ continuous and monotonic . This allows to conclude that the lower bound ωmin ≥ n0 + 1 in ( 6 ) is sharp for a wide range of subsets M , e.g . when M = ∂D ∪ { c } for some D with non-empty interior and c ∈ D◦ . 2 . We show that for the case of two disjoint compact sets , the existence of a cone-like sector that contains one of these sets and does not intersect with the other one , is sufficient to allow exact fit of functions that take constant values on each of these sets by network functions from NNReLU ( n0 , 4 ) . A weaker result is concluded for the case of multiple pairwise compact components . 3 MAXIMUM PRINCIPLE . Let us recall thatNN σ ( n0 ) designates the set of network functions with activation function σ , having maximum width n0 and arbitrary depth . In this section , we will prove a maximum principle for network functions ofNN σ ( n0 ) for a wide class of activation functions . This principle can be viewed as a root cause why universal approximation with functions from NN σ ( n0 ) on arbitrary compact sets is impossible as shown in Johnson ( 2018 ) and Beise et al . ( 2021 ) , see ( 6 ) . This also immediately leads to a topological condition on subsets of Rn0 that do not admit universal approximation by functions in NN σ ( n0 ) . Theorem 1 ( Maximum Principle ) . Let M be some compact subset of Rn0 and σ a continuous , monotonic activation function . Then every F ∈ NN σ ( n0 ) takes its maximum value at the boundary ∂M . Note that , considering −F instead of F , the latter result also implies that the minimum value is taken on ∂M . This implies that universal uniform approximation of continuous functions is impossible on compact sets with non-empty interior . Even more , it can be concluded that : Corollary 1 . Let M be some compact subset of Rn0 such that ∂D ⊂M and M ∩D◦ 6= ∅ for some non-empty open set D ⊂ Rn . Then M does not admit universal uniform approximation of continuous function by network functions from NN σ ( n0 ) . The latter results naturally lead to the following question : Is it possible to uniformly approximate arbitrary continuous functions on the unit sphere ( in Rn0 ) with arbitrary accuracy by network functions as they are considered in Theorem 1 ? Some positive approximation results are given in the next section , but a gap to an answer to the question remains . It is not clear to us whether the conditions of Theorem 1 can be weakened in a way that it also applies to network function having width larger than the input dimensions under certain circumstances . The following might be a natural question in this context : Let F : Rn0 → R be a neural network function of arbitrary width but with weight matrices having rank less than or equal to n0 . Does a maximum principle similar to Theorem 1 apply in this case ? The following example shows that the maximum principle in Theorem 1 does not hold in this case . Example 1 . Let M = [ −1 , 1 ] and let W1 ∈ R2×1 and b1 ∈ R2 implement the following linear affine mapping W1x+ b1 = ( x+ 1 x ) . Then M1 = ReLU ( W1M + b1 ) is given by M1 =M1,1 ∪M1,2 where M1,1 : = { ( x+ 1 , 0 ) T : x ∈M , and x < 0 } M1,2 : = { ( x+ 1 , x ) T : xT ∈M , and x ≥ 0 } . Now , let v = ( 1 , 1/2 ) T ∈ R2 , W2 ∈ R2×2 and b2 ∈ R2 such that x 7→ W2x + b2 implements the orthogonal projection onto the hypersurface V : = { x ∈ R2 : xT v = ( 1 , 0 ) v } . That is , the hypersurface orthogonal to v which intersects ( 1 , 0 ) T . Then both , M1,1 and M1,2 are mapped to the line M2 : = { y ∈ R2 : y = ( 1 , 0 ) +λ ( −1/2 , 1 ) , 0 ≤ λ ≤ 1/2 } and since those vectors are contained in the first quadrant we have M2 = ReLU ( W2 ( ReLU ( W1M + b1 ) + b1 ) . It can further easily be verified that , under the latter two layer network function , 0 , which is an inner point of M , is the only vector that is mapped to the extreme point ( 1 , 0 ) of M2 . Now it is easy to find a weight W3 ∈ R2 such that the linear map x 7→ W3x as a mapping from M2 to R takes its minimum or maximum value at ( 1 , 0 ) only . By concatenation of additional components to the input dimension such that M = [ −1 , 1 ] × [ 0 , 1 ] d , d ∈ N , and adaptation of the weights W1 , W2 and bias b1 , b2 in a way that these additional components are mapped identically under the action of the two layer network function x 7→ ReLU ( W2 ( ReLU ( W1x+ b1 ) + b1 ) , the above example can be extended to higher dimensions . The following proposition provides the main observation for the proof of Theorem 1 . Proposition 1 . Let M be a compact subset of Rn0 , σ a continuous , monotonic activation function and F ∈ NNn0σ ( n0 ) such that all weight matrices Wj , j = 1 , ... , L , are square and have full rank . If for some x ∈M◦ the image F ( x ) is a boundary point of F ( M ) , i.e . F ( x ) ∈ ∂F ( M ) , then there is an x̃ ∈ ∂M such that F ( x̃ ) = F ( x ) . Proof . In case that σ is injective , the whole network function F is injective since the weight matrices are assumed to be square and of full rank . Hence , the invariance of the domain theorem gives that no x ∈ M◦ can be mapped to ∂F ( M ) . The remaining case is that σ is partially constant . Then , given an x ∈ M◦ with F ( x ) ∈ ∂F ( M ) , let k̃ be the smallest k ∈ { 1 , ... , L − 1 } such that σ ( Fk̃ ( x ) ) ∈ ∂σ ( Fk̃ ( M ) ) . Notice that such a k̃ must exist since , according to the invariance of the domain theorem , the last injective , linear affine mapping x 7→ WLx can not map inner points to boundary points . Then , necessarily for at least one component of Fk̃ ( x ) , a partially constant part of σ is active . More explicitly , say ( Fk̃ ( x ) ) ( j ) ∈ [ aj , bj ] ⊂ R with σ ( t ) = cj , where cj is the constant value taken for all t ∈ [ aj , bj ] , for some indices j ∈ I ⊂ { 1 , ... , n0 } . In case that one of those intervals can be enlarged arbitrarily to either [ aj , ∞ ) or ( −∞ , bj ] , which would be the case for ReLU activation , we can find a t ∈ R such that xk̃ = Fk̃ ( x ) + tej ∈ ∂Fk̃ ( M ) and with σ ( xk̃ ) = σ ( Fk̃ ( x ) ) . In a next step we show that such an xk̃ can also be found in cases where the length of the intervals [ aj , bj ] are upper bounded . In those cases we may assume that the intervals [ aj , bj ] are maximum in the sense that at their left and right end σ is not constant any more , and I is maximum in the sense that for every other index , the activation function σ is injective in a neighbourhood . Indeed , if such components would not exist , the component wise application of σ would be injective in a small neighbourhood of Fk̃ ( x ) and the invariance of the domain theorem would exclude that σ ( Fk̃ ( x ) ) ∈ ∂σ ( Fk̃ ( M ) ) . By the selection of k̃ , Fk̃ ( x ) is still an inner point of Fk̃ ( M ) . The fact that the inner point Fk̃ ( x ) ∈ ( Fk̃ ( M ) ) ◦ is mapped to a boundary point by the element wise application of σ implies that there is some non-empty Ĩ ⊂ I with corresponding tj ∈ R such that tj ∈ [ − ( Fk̃ ( x ) ) ( j ) + aj , − ( Fk̃ ( x ) ) ( j ) + bj ] , j ∈ Ĩ ( 7 ) xk̃ : = Fk̃ ( x ) + ∑ j∈Ĩ tjej ∈ ∂Fk̃ ( M ) . ( 8 ) Indeed , otherwise , Fk̃ ( x ) + n0∑ j=1 tjej ∈ ( Fk̃ ( M ) ) ◦ where for some small ε > 0 and all j ∈ I , tj may be chosen arbitrarily in ( − ( Fk̃ ( x ) ) ( j ) + aj − ε , − ( Fk̃ ( x ) ) ( j ) + bj + ε ) and ε > |tj | > 0 for j /∈ I . By the definition of I and the corresponding intervals [ aj , bj ] , this would imply that σ ( Fk̃ ( x ) ) is an inner point of σ ( Fk̃ ( M ) ) . Now , ( 8 ) implies that there is a x̃ ∈M such that Fk̃ ( x̃ ) = Fk̃ ( x ) + n0∑ j=1 tjej ∈ ∂Fk̃ ( M ) , with tj as in ( 7 ) . Considering that Wk̃ is square and has full rank , the inverse image of Fk̃ ( x̃ ) under x 7→Wk̃x+ bk̃ is a boundary point of σ ( Fk̃−1 ( M ) ) in case that k̃ > 1 and a boundary point of M in case when k̃ = 1 . In the last case we are done . In the first case , the same reasoning can be applied to x̃ instead of x , and the iterative application finally leads to a boundary point of M that is mapped to σ ( Fk̃ ( x̃ ) ) = σ ( Fk̃ ( x ) ) and hence follows the same trajectory when passed through the last layers from k̃ to L. Proof . ( Theorem 1 ) We may assume that M◦ is non-empty and that every weight matrix Wj , j = 1 , ... , L− 1 , is square and of full rank . Otherwise the result is trivial , because F ( M ) = ∂F ( M ) . Hence the assumption of Proposition 1 holds for F̃ = AL−1 ◦ ... ◦A1 . Since the linear mapping x 7→ WLx from RnL−1 to R takes its maximum and minimum values on ∂F̃ ( M ) , the latter proposition also implies that F takes its maximum value on ∂M .

Summary: This paper presents three results concerning the universal approximation of functions on compact sets under a width regime for which universal approximation in general is not possible. The authors make a convincing case that this represents an important frontier both for applications and for a better understanding of what neural networks can represent. While I appreciate the first result, I am confused with the second and I do not think the third is relevant.  These results are the following:  1) The maximum principle  This is an interesting observation about the maximum (and minimum) value achieved by neural network with continuous monotonic activation and width bounded by the input layer on a compact domain is always at the boundary of such domain. Trying to break their result made me understand and agree with their claim that this could be understood as a "root cause why universal approximation with functions from NNσ(n0) on arbitrary compact sets is impossible".  2) The result on disjoint domains  In Section 4, the authors claim through Theorem 2 a result applicable to a "finite collection of disjoint n0-dimensional compact sets". Notably, most of their work there focuses on exact representation, which reflects the fact that they are considering functions that are constant on each disjoint set. We can reasonably associate that to classification tasks, but then we cannot assume any probabilistic meaning with something like a softmax layer.  3) The result on harmonic activations  The authors shows that universal approximation can be achieved on a finite and discrete domain with the composition of cosine functions. This result lies entirely on the domain being discrete and finite, which actually breaks any dependence to the width of the input layer. For that reason, I do not think that it is that relevant because it has nothing to do with the main theme of the paper.