Deep, Skinny Neural Networks are not Universal Approximators

1 INTRODUCTION . Neural networks have become the model of choice in a variety of machine learning applications , due to their flexibility and generality . However , selecting network architectures and other hyperparameters is typically a matter of trial and error . To make the choice of neural network architecture more straightforward , we need to understand the limits of each architecture , both in terms of what kinds of functions any given network architecture can approximate and how those limitations impact its ability to learn functions within those limits . A number of papers ( 3 ; 6 ; 11 ; 13 ) have shown that neural networks with a single hidden layer are a universal approximator , i.e . that they can approximate any continuous function on a compact domain to arbitrary accuracy if the hidden layer is allowed to have an arbitrarily high dimension . In practice , however , the neural networks that have proved most effective tend to have a large number of relatively low-dimensional hidden layers . This raises the question of whether neural networks with an arbitrary number of hidden layers of bounded dimension are also a universal approximator . In this paper we demonstrate a fairly general limitation on functions that can be approximated with the L∞ norm on compact subsets of a Euclidean input space by layered , fully-connected feedforward neural networks of arbitrary depth and activation functions from a broad family including sigmoids and ReLus , but with layer widths bounded by the dimension of the input space . By a layered network , we mean that hidden nodes are grouped into successive layers and each node is only connected to nodes in the previous layer and the next layer . The constraints on the functions are defined in terms of topological properties of the level sets in the input space . This analysis is not meant to suggest that deep networks are worse than shallow networks , but rather to better understand how and why they will perform differently on different data sets . In fact , these limitations may be part of the reason deep nets have proven more effective on datasets whose structures are compatible with these limitations . By a level set , we mean the set of all points in the input space that the model maps to a given value in the output space . For classification models , a level set is just a decision boundary for a particular cutoff . For regression problems , level sets don ’ t have a common interpretation . The main result of the paper , Theorem 1 , states that the deep , skinny neural network architectures described above can not approximate any function with a level set that is bounded in the input space . This can be rephrased as saying that for every function that can be approximated , every level set must be unbounded , extending off to infinity . While a number of recent papers have made impressive progress in understanding the limitations of different neural network architectures , this result is notable because it is independent of the number of layers in the network , and because the limitations are defined in terms of a very simple topological property . Topological tools have recently been employed to study the properties of data sets within the field known as Topological Data Analysis ( 9 ) , but this paper exploits topological ideas to examine the topology of the models themselves . By demonstrating topological constraints on a widely used family of models , we suggest that there is further potential to apply topological ideas to understand the strengths and weaknesses of algorithms and methodologies across machine learning . After discussing the context and related work in Section 2 , we introduce the basic definitions and notation in Section 3 , then state the main Theorem and outline the proof in Section 4 . The detailed proof is presented in Sections 5 and 6 . We present experimental results that demonstrate the constraints in Section 7 , then in Section 8 we present conclusions from this work . 2 RELATED WORK . A number of papers have demonstrated limitations on the functions that can be approximated by neural networks with particular architectures ( 2 ; 12 ; 14 ; 15 ; 18 ; 19 ; 21 ; 22 ; 23 ; 24 ; 25 ; 27 ; 29 ; 32 ) . These are typically presented as asymptotic bounds on the size of network needed to approximate any function in a given family to a given ε. Lu et al ( 17 ) gave the first non-approximation result that is independent of complexity , showing that there are functions that no ReLu-based deep network of width equal to the dimension of the input space can approximate , no matter how deep . However , they consider convergence in terms of the L1 norm on the entire space Rn rather than L∞ on a compact subset . This is a much stricter definition than the one used in this paper so even for ReLu networks , Theorem 1 is a stronger result . The closest existing result to Theorem 1 is a recent paper by Nguyen , Mukkamala and Hein ( 26 ) which shows that for multi-label classification problems defined by an argmax condition on a higherdimensional output function , if all the hidden layers of a neural network have dimension less than or equal to the input dimension then the region defining each class must be connected . The result applies to one-to-one activation functions , but could probably be extended to the family of activation functions in this paper by a similar limiting argument . Universality results have been proved for a number of variants of the networks described in Theorem 1 . Rojas ( 28 ) showed that any two discrete classes of points can be separated by a decision boundary of a function defined by a deep , skinny network in which each layer has a single perceptron that is connected both to the previous layer and to the input layer . Because of the connections back to the input space , such a network is not layered as defined above , so Theorem 1 doesn ’ t contradict this result . In fact , to carry out Rojas ’ construction with a layered feed-forward network , you would need to put all the perceptrons in a single hidden layer . Sutskever and Hinton ( 30 ) showed that deep belief networks whose hidden layers have the same dimension as the input space can approximate any function over binary vectors . This binary input space can be interpreted as a discrete subset of Euclidean space . So while Theorem 1 does not apply to belief networks , it ’ s worth noting that any function on a discrete set can be extended to the full space in such a way that the resulting function satisfies the constraints in Theorem 1 . This unexpected constraint on skinny deep nets raises the question of whether such networks are so practically effective despite being more restrictive than wide networks , or because of it . Lin , Tegmark and Rolnick ( 16 ) showed that for data sets with information-theoretic properties that are common in physics and elsewhere , deep networks are more efficient than shallow networks . This may be because such networks are restricted to a smaller search space concentrated around functions that model shapes of data that are more likely to appear in practice . Such a conclusion would be consistent with a number of papers showing that there are functions defined by deep networks that can only by approximated by shallow networks with asymptotically much larger number of nodes ( 4 ; 7 ; 10 ; 20 ; 31 ) . A slightly different phenomenon has been observed for recurrent neural networks , which are universal approximators of dynamic systems ( 8 ) . In this setting , Collins , Sohl-Dickstein and Sussillo ( 5 ) showed that many differences that have been reported on the performance of RNNs are due to their training effectiveness , rather than the expressiveness of the networks . In other words , the effectiveness of a given family of models appears to have less to do with whether it includes an accurate model , and more to do with whether a model search algorithm like gradient descent is likely to find an accurate model within the search space of possible models . 3 TERMINOLOGY AND NOTATION . A model family is a subset M of the space C ( Rn , Rm ) of continuous functions from input space Rn to output space Rm . For parametric models , this subset is typically defined as the image of a map Rk → C ( Rn , Rm ) , where Rk is the parameter space . A non-parametric model family is typically the union of a countably infinite collection of parametric model families . We will not distinguish between parametric and non-parametric families in this section . Given a function g : Rn → Rm , a compact subset A ⊂ Rn and a value > 0 , we will say that a second function f ( , A ) -approximates g if for every x ∈ A , we have |f ( x ) − g ( x ) | < . Similarly , we will say that a model family M ( , A ) -approximates g if there is a function f in M that ( , A ) -approximates g. More generally , we will say that M approximates f if for every compact A ⊂ Rn and value > 0 there is a function f in M that ( , A ) -approximates g. This is equivalent to the statement that there is a sequence of functions fi ∈M that converges pointwise ( though not necessarily uniformly ) to g on all of Rn . However , we will use the ( , A ) definition throughout this paper . We ’ ll describe families of layered neural networks with the following notation : Given an activation function ϕ : R → R and a finite sequence of positive integers n0 , n1 , . . . , nκ , let Nϕ , n0 , n1 , ... , nκ be the family of functions defined by a layered feed-forward neural network with n0 inputs , nκ outputs and fully connected hidden layers of width n1 , . . . , nκ−1 . With this terminology , Hornik et al ’ s results can be restated as saying that the ( non-parametric ) model family defined as the union of all families Nϕ , n0 , n1,1 approximates any continuous function . ( Here , κ = 2 and n2 = 1 . ) We ’ re interested in deep networks with bounded dimensional layers , so we ’ ll let N ∗ϕ , n be the union of all the model families Nϕ , n0 , n1 , ... , nκ−1,1 such that ni ≤ n for all i < κ . For the main result , we will restrict our attention to a fairly large family of activation functions . We will say that an activation function ϕ is uniformly approximated by one-to-one functions if there is a sequence of continuous , one-to-one functions that converge to ϕ uniformly ( not just pointwise ) . Note that if the activation function is itself one-to-one ( such as a sigmoid ) then we can let every function in the sequence be ϕ and it will converge uniformly . For the ReLu function , we need to replace the the large horizontal portion with a function such as 1n arctan ( x ) . Since this function is one-to-one and negative for x < 0 , each function in this sequence will be one-to-one . Since it ’ s bounded between − 1n and 0 , the sequence will converge uniformly to the ReLu function .

Summary: This paper shows that deep "narrow" neural networks (i.e. all hidden layers have maximum width at most the input dimension) with a variety of activation functions, including ReLU and sigmoid, can only learn functions with unbounded level set components, and thus cannot be a universal approximator. This complements previous work, such as Nguyen et. al 2018 which study connectivity of decision regions and Lu et. al 2017 on ReLU networks in different ways.