Deep, Skinny Neural Networks are not Universal Approximators

1 INTRODUCTION . Neural networks have become the model of choice in a variety of machine learning applications , due to their flexibility and generality . However , selecting network architectures and other hyperparameters is typically a matter of trial and error . To make the choice of neural network architecture more straightforward , we need to understand the limits of each architecture , both in terms of what kinds of functions any given network architecture can approximate and how those limitations impact its ability to learn functions within those limits . A number of papers ( 3 ; 6 ; 11 ; 13 ) have shown that neural networks with a single hidden layer are a universal approximator , i.e . that they can approximate any continuous function on a compact domain to arbitrary accuracy if the hidden layer is allowed to have an arbitrarily high dimension . In practice , however , the neural networks that have proved most effective tend to have a large number of relatively low-dimensional hidden layers . This raises the question of whether neural networks with an arbitrary number of hidden layers of bounded dimension are also a universal approximator . In this paper we demonstrate a fairly general limitation on functions that can be approximated with the L∞ norm on compact subsets of a Euclidean input space by layered , fully-connected feedforward neural networks of arbitrary depth and activation functions from a broad family including sigmoids and ReLus , but with layer widths bounded by the dimension of the input space . By a layered network , we mean that hidden nodes are grouped into successive layers and each node is only connected to nodes in the previous layer and the next layer . The constraints on the functions are defined in terms of topological properties of the level sets in the input space . This analysis is not meant to suggest that deep networks are worse than shallow networks , but rather to better understand how and why they will perform differently on different data sets . In fact , these limitations may be part of the reason deep nets have proven more effective on datasets whose structures are compatible with these limitations . By a level set , we mean the set of all points in the input space that the model maps to a given value in the output space . For classification models , a level set is just a decision boundary for a particular cutoff . For regression problems , level sets don ’ t have a common interpretation . The main result of the paper , Theorem 1 , states that the deep , skinny neural network architectures described above can not approximate any function with a level set that is bounded in the input space . This can be rephrased as saying that for every function that can be approximated , every level set must be unbounded , extending off to infinity . While a number of recent papers have made impressive progress in understanding the limitations of different neural network architectures , this result is notable because it is independent of the number of layers in the network , and because the limitations are defined in terms of a very simple topological property . Topological tools have recently been employed to study the properties of data sets within the field known as Topological Data Analysis ( 9 ) , but this paper exploits topological ideas to examine the topology of the models themselves . By demonstrating topological constraints on a widely used family of models , we suggest that there is further potential to apply topological ideas to understand the strengths and weaknesses of algorithms and methodologies across machine learning . After discussing the context and related work in Section 2 , we introduce the basic definitions and notation in Section 3 , then state the main Theorem and outline the proof in Section 4 . The detailed proof is presented in Sections 5 and 6 . We present experimental results that demonstrate the constraints in Section 7 , then in Section 8 we present conclusions from this work . 2 RELATED WORK . A number of papers have demonstrated limitations on the functions that can be approximated by neural networks with particular architectures ( 2 ; 12 ; 14 ; 15 ; 18 ; 19 ; 21 ; 22 ; 23 ; 24 ; 25 ; 27 ; 29 ; 32 ) . These are typically presented as asymptotic bounds on the size of network needed to approximate any function in a given family to a given ε. Lu et al ( 17 ) gave the first non-approximation result that is independent of complexity , showing that there are functions that no ReLu-based deep network of width equal to the dimension of the input space can approximate , no matter how deep . However , they consider convergence in terms of the L1 norm on the entire space Rn rather than L∞ on a compact subset . This is a much stricter definition than the one used in this paper so even for ReLu networks , Theorem 1 is a stronger result . The closest existing result to Theorem 1 is a recent paper by Nguyen , Mukkamala and Hein ( 26 ) which shows that for multi-label classification problems defined by an argmax condition on a higherdimensional output function , if all the hidden layers of a neural network have dimension less than or equal to the input dimension then the region defining each class must be connected . The result applies to one-to-one activation functions , but could probably be extended to the family of activation functions in this paper by a similar limiting argument . Universality results have been proved for a number of variants of the networks described in Theorem 1 . Rojas ( 28 ) showed that any two discrete classes of points can be separated by a decision boundary of a function defined by a deep , skinny network in which each layer has a single perceptron that is connected both to the previous layer and to the input layer . Because of the connections back to the input space , such a network is not layered as defined above , so Theorem 1 doesn ’ t contradict this result . In fact , to carry out Rojas ’ construction with a layered feed-forward network , you would need to put all the perceptrons in a single hidden layer . Sutskever and Hinton ( 30 ) showed that deep belief networks whose hidden layers have the same dimension as the input space can approximate any function over binary vectors . This binary input space can be interpreted as a discrete subset of Euclidean space . So while Theorem 1 does not apply to belief networks , it ’ s worth noting that any function on a discrete set can be extended to the full space in such a way that the resulting function satisfies the constraints in Theorem 1 . This unexpected constraint on skinny deep nets raises the question of whether such networks are so practically effective despite being more restrictive than wide networks , or because of it . Lin , Tegmark and Rolnick ( 16 ) showed that for data sets with information-theoretic properties that are common in physics and elsewhere , deep networks are more efficient than shallow networks . This may be because such networks are restricted to a smaller search space concentrated around functions that model shapes of data that are more likely to appear in practice . Such a conclusion would be consistent with a number of papers showing that there are functions defined by deep networks that can only by approximated by shallow networks with asymptotically much larger number of nodes ( 4 ; 7 ; 10 ; 20 ; 31 ) . A slightly different phenomenon has been observed for recurrent neural networks , which are universal approximators of dynamic systems ( 8 ) . In this setting , Collins , Sohl-Dickstein and Sussillo ( 5 ) showed that many differences that have been reported on the performance of RNNs are due to their training effectiveness , rather than the expressiveness of the networks . In other words , the effectiveness of a given family of models appears to have less to do with whether it includes an accurate model , and more to do with whether a model search algorithm like gradient descent is likely to find an accurate model within the search space of possible models . 3 TERMINOLOGY AND NOTATION . A model family is a subset M of the space C ( Rn , Rm ) of continuous functions from input space Rn to output space Rm . For parametric models , this subset is typically defined as the image of a map Rk → C ( Rn , Rm ) , where Rk is the parameter space . A non-parametric model family is typically the union of a countably infinite collection of parametric model families . We will not distinguish between parametric and non-parametric families in this section . Given a function g : Rn → Rm , a compact subset A ⊂ Rn and a value > 0 , we will say that a second function f ( , A ) -approximates g if for every x ∈ A , we have |f ( x ) − g ( x ) | < . Similarly , we will say that a model family M ( , A ) -approximates g if there is a function f in M that ( , A ) -approximates g. More generally , we will say that M approximates f if for every compact A ⊂ Rn and value > 0 there is a function f in M that ( , A ) -approximates g. This is equivalent to the statement that there is a sequence of functions fi ∈M that converges pointwise ( though not necessarily uniformly ) to g on all of Rn . However , we will use the ( , A ) definition throughout this paper . We ’ ll describe families of layered neural networks with the following notation : Given an activation function ϕ : R → R and a finite sequence of positive integers n0 , n1 , . . . , nκ , let Nϕ , n0 , n1 , ... , nκ be the family of functions defined by a layered feed-forward neural network with n0 inputs , nκ outputs and fully connected hidden layers of width n1 , . . . , nκ−1 . With this terminology , Hornik et al ’ s results can be restated as saying that the ( non-parametric ) model family defined as the union of all families Nϕ , n0 , n1,1 approximates any continuous function . ( Here , κ = 2 and n2 = 1 . ) We ’ re interested in deep networks with bounded dimensional layers , so we ’ ll let N ∗ϕ , n be the union of all the model families Nϕ , n0 , n1 , ... , nκ−1,1 such that ni ≤ n for all i < κ . For the main result , we will restrict our attention to a fairly large family of activation functions . We will say that an activation function ϕ is uniformly approximated by one-to-one functions if there is a sequence of continuous , one-to-one functions that converge to ϕ uniformly ( not just pointwise ) . Note that if the activation function is itself one-to-one ( such as a sigmoid ) then we can let every function in the sequence be ϕ and it will converge uniformly . For the ReLu function , we need to replace the the large horizontal portion with a function such as 1n arctan ( x ) . Since this function is one-to-one and negative for x < 0 , each function in this sequence will be one-to-one . Since it ’ s bounded between − 1n and 0 , the sequence will converge uniformly to the ReLu function .

Summary: This paper proves a theoretical limitation of narrow-and-deep neural networks. It shows that, for any function that can be approximated by such networks, its level set (or decision boundary for binary classification) must be unbounded. The conclusion means that if some problem's decision boundary is a closed set, then it cannot be represented by such narrow networks.