# ReLU Network with Bounded Width Is a Universal Approximator in View of an Approximate Identity
## 1 Introduction  
Over the past several years, deep neural networks have achieved state-of-the-art performance in a wide range of tasks such as image recognition/segmentation and machine translation (see the review article [1] and recent book [2] for more background). In spite of their outstanding successes, many aspects of why they work so well are still not well understood. Some works to date have focused on the problem of explaining and quantifying the expressivity of a deep neural network [3, 4, 5].  
This line of research about the expressive power of neural networks, its ability to approximate functions, dates back to at least the work of [PERSON] [6], in which a fully connected neural network with a sigmoid activation function and one single hidden layer can approximate any continuous univariate function on a bounded domain with an arbitrarily small error. [PERSON] [7], [PERSON] et al. [8], and [PERSON] [9] generalized the sigmoid function to a large class of activation functions to get universal approximation. However, they do not consider the number of nodes of a hidden layer; actually, the number of nodes of a hidden layer can be exponentially increased in the input dimension.  
Because the outstanding performance of deep neural networks has recently been exhibited, there is a lot of literature concerning their expressive power theoretically. In 2011, [PERSON] and [PERSON] [10] gave a family of functions which can be represented much more efficiently with a deep network than with a neural network with one hidden layer. [PERSON] and [PERSON] [4] provided conditions under which deep convolutional neural networks perform much better in function approximation than a neural network with one hidden layer. [PERSON] and [PERSON] [11] constructed a three-layer network which cannot be realized by any 2-layer if the number of nodes of an output layer is no more than an exponential bound.  
Actually, the Rectified Linear Units (ReLU) activation function is the most popular choice in practical use of the neural network [12]. In this reason, most of the recent results on the universal approximation theory is about the ReLU network [5, 13, 14, 15, 16, 17, 18, 19, 20]. [PERSON] et al. [13] provided the deep convolutional neural network with the ReLU activation function thatcannot be realized by a shallow network if the number of nodes of its hidden layer is no more than an exponential bound.  
[PERSON] et al. [14] presented a universal approximation theorem for deep neural networks with ReLU activation functions and hidden layers with a bounded width in 2017, since the expressive power of depth in ReLU networks with a bounded width has received a lot of attention. [PERSON] proved universal approximation theorem using convex function theory in [15; 16], and [PERSON] and [PERSON] showed this theorem for residual networks with one-neuron hidden layers in [19].  
[PERSON] and [PERSON] [18] showed that the ReLU activation function can be represented by a linear combination of piecewise linear activation function and, using this fact, extended many results in [5; 17; 20] to any continuous piecewise linear activation function. However, their results are not about the bounded width.  
Here, we study a universal approximation theorem for ReLU networks in view of approximate identity by constructing a network having \(n_{inf}k^{n_{in}}+1\) hidden layers with width of at most \(n_{in}+5\). As a measure of approximation error, we adopt the supremum norm and \(L^{p}\) distance.  
The remainder of the paper is organized as follows: in the next subsection, we introduce some notations. Sections 2 and 3 are devoted to our theorem statement and its proof, respectively. Finally, Section 4 offers a conclusion.