# The Construction and Approximation of ReLU Neural Network Operators

Authors: Hengjie Chen; Dansheng Yu; Zhong Li
Journal: Journal of Function Spaces
 (2022)

---

## Abstract

In the present paper, we construct a new type of two-hidden-layer feedforward neural network operators with ReLU activation function. We estimate the rate of approximation by the new operators by using the modulus of continuity of the target function. Furthermore, we analyze features such as parameter sharing and local connectivity in this kind of network structure.


---

## Body

## 1. Introduction

Artificial neural network is a fundamental method in machine learning, which has been applied in many fields such as pattern recognition, automatic control, signal processing, auxiliary decision-making, and artificial intelligence. In particular, the successful applications of deep (multihidden layer) neural networks in image recognition, natural language processing, computer vision, etc. developed in recent years have made neural networks attract great attention. In fact, ever the function of XOR gate was implemented by adding one layer from the simplest perceptron, which leaded to the single-hidden-layer feedforward neural network.A single-hidden-layer feedforward neural network has the expression form(1)Nx=∑i=1n1ciϕωi·x+θi,whereci,θii=1,2,⋯,n are called as output weights and thresholds, the dimension of input weights ωii=1,2,⋯,n corresponds to that of the input x, ϕ, is called the activation function of this network, and n1 is the number of neurons in the hidden layer. If x∈ℝd,A1=ω1ω2⋯ωn1T (T denotes the transpose) is the input weight matrix of size n1×d,Θ1=θ1,θ2,⋯,θnT and C1=c1,c2,⋯,cnT are vectors of thresholds and output weights, respectively, then 1 can be written as
(2)Nx=C1TϕA1x+Θ1,whereϕA1x+Θ1 means that ϕ acts on each component of A1x+Θ1. Now, the architecture of the neural network with two hidden layers is really not difficult to understand. If the second hidden layer contains n2 neurons, the input weight matrix A2 is the size of n2×n1, the vector of thresholds is Θ2, and the output weight vector is O; then, the two-hidden-layer feedforward neural network can be mathematically expressed as
(3)N¯x=OTϕA2ϕA1x+Θ1+Θ2.We callw=maxn1,n2 as the width of the network N¯x, and its depth is naturally 2.The theoretical research and applications of the single-hidden-layer neural network model had been greatly developed in the 80’s and 90’s of last century; particularly, there were also some research results on the neural networks with multihidden layers at that time. So, in [1], Pinkus pointed out that “Nonetheless there seems to be reasonable to conjecture that the two-hidden-layer model may be significantly more promising than the single layer model, at least from a purely approximation-theoretical point of view. This problem certainly warrants further study.” However, whether it is a single-hidden-layer or multihidden-layer neural network, three fundamental issues are always involved: density, complexity, and algorithms.The so-called density or universal approximation of a neural network structure means that for any given error accuracy and the target function in a function space with some metrics, there is a specific neural network model (except for the inputx, other parameters are determined) such that the error between the output and target is less than the preaccuracy. In the 1980s and 1990s, the research on the density of feedforward neural network has achieved many satisfactory results [2–9]. Since the single-hidden-layer neural network is an extreme case of the multilayer neural networks, the current focus of neural network research is still on complexity and algorithms. So-called the complexity of a neural network means that to guarantee a prescribed degree of approximation, a neural network model requires the numbers of structural parameters, including the number of layers (or depth), the number of neuron in each layer (sometimes use width), and the number of link weights and the number of thresholds. In particular, it is of interest to have more equal weights and thresholds, which is called as the parameter sharing, as this reduces computational complexity. The representation ability that has attracted much attention in deep neural networks is actually the study of complexity problem, which needs to be investigated extensively and urgently.The constructive method is an important approach to the study of complexity, which is applicable to single- and multiple-hidden-layer neural network. In fact, there are two cases here: one is that the depth, width, and approximation degree are given, while the weights and thresholds are uncertain; the other is that all these are given; that is, the neural network model is completely determined. In order to determine the weights and thresholds in the first kind of neural network, we simply use samples to learn or train. Theoretically, the second kind of neural network can be applied directly, but the parameters are often fine-tuned with a small number of samples before use. There have been many results about the constructions of network operators [10–26]. It can be seen that these research results have an important guiding role in the construction and design of neural networks. Therefore, the purpose of this paper is to construct a kind of two-hidden-layer feedforward neural network operators with the ReLU activation function, and the upper bound estimate of approximation (or regression) ability of this neural network for two variable continuous function defined on −1,12 is given.The rest of the paper is organized as follows: in Section2, we introduce new two-hidden-layer neural network operators with ReLU activation function and give the approximation rate of approximation by the new operators. In Section 3, we give the proof of the main result. Finally, in Section 4, we give some numerical experiments and discussions.


## 2. Construction of ReLU Neural Network Operators and Its Approximation Properties

Letr:ℝ⟶ℝ denote the rectified linear unit (ReLU), i.e., rx=max0,x. For any x1,x2∈ℝ×ℝ≕ℝ2, we define
(4)σx1,x2≔34r1−12x1+x2−12x1−x2.Obviously,σ is a continuous function of two variables supported on −1,12. By using the fact that x=rx+r−x, σ can be rewritten as follows:
(5)σx1,x2=34r1−12rx1+x2+rx1−x2+r−x1−x2+r−x1+x2.From the above representation, we see thatσx1,x2 can be explained as the output of a two-hidden-layer feedforward neural network. It is obvious that σ possesses the following some important properties:(A1)σ−x1,x2=σx1,x2,σx1,−x2=σx1,x2(A2) For any givenx1∈ℝ,σx1,x2, it is nondecreasing for x2≤0 and nonincreasing for x2≥0. Simultaneously, for any given x2∈ℝ,σx1,x2,it is nondecreasing for x1≤0 and nonincreasing for x1≥0(A3)0≤σx1,x2≤3/4(A4)∫−11∫−11σx1,x2dx1dx2=1For any continuous functionfx1,x2 on −1,12, we define the following neural network operator:
(6)Nnf;x1,x2=1n∑k1=−n−nn+n∑k2=−n−nn+nfk1n+n,k2n+n·σnx1−k1n,nx2−k2n,where x is the largest integer not greater than x, and x denotes the smallest integer not less than x.We prove that the rate of approximation byNnf can be estimated by using the modulus of smoothness of the target function. In fact, we haveTheorem 1.
Letfx1,x2 is a continuous function defined on −1,12. Then,
(7)Nnf;x1,x2−fx1,x2≤322+1n2ωf;1n+2Mfn11+32n,x1,x2∈−1,12,
whereMf=maxx1,x2∈−1,12fx1,x2 and ωf;1/n are the modulus of continuity of f defined by
(8)ωf;δ=supx1−x1′≤δ,x2−x2′≤δfx1,x2−fx1′,x2′.Remark 2.
For0<α<1, we define the following neural network operators:
(9)N~nf;x1,x2=1n2α∑−n−nαn+nα∑−n−nαn+nαfk1n+nα,k2n+nα·σnx1−k1nα,nx2−k2nα,x1,x2∈−1,12.
Using a similar process of the proof in Theorem1, we can get
(10)N~nf;x1,x2−fx1,x2≤322+1nα2ωf;1n1−α+2Mfnα11+32nα,x1,x2∈−1,12.Remark 3.
Letβ (0<β≤1) be a fixed number, if there is a constant L>0 such that
(11)fx1,x2−fx1′,x2′≤Lx1−x1′2+x2−x2′2β,for any x1,x2,x1′,x2′∈−1,12, we say that f is a Lipschitz function of order β and write f∈LipLβ. Obviously, when f∈LipLβ, we have ωf;δ≤2Lδβ. Consequently, it follows from 7 that
(12)N~nf;x1,x2−fx1,x2≤322+1nα22Ln1−αβ+2Mfnα11+32nα,x1,x2∈−1,12.Remark 4.
Now, we describe the structure ofNnf by using the form (3).
The input matrix of the first hidden layer is(13)A1=n−n−nn⋯n−n−nn−n−nnn⋯−n−nnnT,
and its size is4n+n−−n−n+12×2. The bias vector of the first hidden layer is
(14)Θ1=−2n+nn,2n+nn,0,0,⋯,−k1+k2n,k1+k2n,k1−k2n,k2−k1n,⋯T,
and the dimension is4n+n−−n−n+12. The input matrix of the second hidden layer is
(15)A2=−12−12−12−120000⋯00000000−12−12−12−12⋯0000⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮00000000⋯−12−12−12−12,
and its size isn+n−−n−n+12×4n+n−−n−n+12. Θ2 is a constant 1 vector with dimension n+n−−n−n+12. The output weight vector is
(16)O=fn+nn+n,n+nn+n,⋯,f−n−nn+n,−n−nn+nT.Its general term and dimension are fk1/n+n,k2/n+n and n+n−−n−n+12, respectively.We can see that there are two different numbers in weight matricesA1 and A2, respectively. That is, neural network operators Nnf have a strong weight sharing feature. There are some results about the constructions of this kind of neural networks [14, 27–29]. Moreover, A2 shows that this neural network is locally connected. Finally, the simplicity of bias vector Θ2 also greatly reduces the complexity of the neural network.


## 3. Proof of the Main Result

To prove Theorem1, we need the following auxiliary lemma.Lemma 5.
For functionσx1,x2, we have
(17)∫−n−n−1nx1−1∫−n−n−1nx2−1σnx1−t1n,nx2−t2ndt1dt2≤∑k1=−n−nnx1−1∑k2=−n−nnx2−1σnx1−k1n,nx2−k2n≤∫−n−nnx1∫−n−nnx2σnx1−t1n,nx2−t2ndt1dt2,∫−n−n−1nx1−1∫nx2+1n+n+1σnx1−t1n,nx2−t2ndt1dt2≤∑k1=−n−nnx1−1∑k2=nx2+1n+nσnx1−k1n,nx2−k2n≤∫−n−nnx1∫nx2n+nσnx1−t1n,nx2−t2ndt1dt2,∫nx1+1n+n+1∫−n−n−1nx2−1σnx1−t1n,nx2−t2ndt1dt2≤∑k1=nx1+1n+n∑k2=−n−nnx2−1σnx1−k1n,nx2−k2n≤∫nx1n+n∫−n−nnx2σnx1−t1n,nx2−t2ndt1dt2,∫nx1+1n+n+1∫nx2+1n+n+1σnx1−t1n,nx2−t2ndt1dt2≤∑k1=nx1+1n+n∑k2=nx2+1n+nσnx1−k1n,nx2−k2n≤∫nx1n+n∫nx2n+nσnx1−t1n,nx2−t2ndt1dt2.Proof.
We only prove (1), and (2), (3), and (4) can be proved similarly.(1)
Whenki−1<ki<ki+1≤nxi−1i=1,2, we have(18)nxi−ki−1>nxi−ki>nxi−ki+1≥1.
Considering the monotonicity ofσx1,x2, we have
(19)σnx1−k1n,nx2−k2n≤∫k1k1+1σnx1−t1n,nx2−k2ndt1,(20)σnx1−t1n,nx2−k2n≤∫k2k2+1σnx1−t1n,nx2−t2ndt2.
Combining (19) and (20) leads to
(21)σnx1−k1n,nx2−k2n≤∫k1k1+1∫k2k2+1σnx1−t1n,nx2−t2ndt1dt2.
Similarly, we have(22)σnx1−k1n,nx2−k2n≥∫k1−1k1∫k2−1k2σnx1−t1n,nx2−t2ndt1dt2.
By (21), (22), and summation from −n−n to nxi−1i=1,2, we obtain (1) of Lemma 5.
(2)
Whenk2+1>k2>k2−1≥nx2+1, we have(23)nx2−k2+1<nx2−k2<nx2−k2−1≤−1.
From (18), (23), and in a similar way to the proof in proving (1), we get
(24)∫k1−1k1∫k2k2+1σnx1−t1n,nx2−t2ndt1dt2≤σnx1−k1n,nx2−k2n,σnx1−k1n,nx2−k2n≤∫k1k1+1∫k2−1k2σnx1−t1n,nx2−t2ndt1dt2.
By summation for−n−n≤k1≤nx1−1 and nx2+1≤k2≤n+n, we obtain (2) of Lemma 5.Proof of Theorem 6.
Let(25)Mf=maxx1,x2∈−1,12fx1,x2.
Then,(26)Nnf;x1,x2−fx1,x2=1n∑−n−nn+n∑−n−nn+nfk1n+n,k2n+nσnx1−k1n,nx2−k2n−fx1,x2≤1n∑−n−nn+n∑−n−nn+nfk1n+n,k2n+n−fx1,x2σnx1−k1n,nx2−k2n+Mf1n∑−n−nn+n∑−n−nn+nσnx1−k1n,nx2−k2n−1≕I1+MfI2.
We further estimateNnf;x1,x2−fx1,x2 by estimating I1 and I2, respectively.
Set(27)∑−n−nn+n∑−n−nn+n=∑x1−k1/n<1/n∑x2−k2/n<1/n+∑x1−k1/n≥1/n∑x2−k2/n<1/n+∑x1−k1/n<1/n∑x2−k2/n≥1/n+∑x1−k1/n≥1/n∑x2−k2/n≥1/n≕∑1+∑2+∑3+∑4.
Since in∑2+∑3+∑4, at least one of inequalities x1−k1/n≥1/n and x2−k2/n≥1/n holds; so, either
(28)x1−k1n+x2−k2n+x1−k1n+x2−k2n≥2x1−k1nor
(29)x1−k1n+x2−k2n+x1−k1n−x2−k2n≥2x2−k2nis valid. Therefore,
(30)1−12nx1−k1n+nx2−k2n+nx1−k1n−nx2−k2n≤0,which implies that
(31)∑2+∑3+∑4fk1n+n,k2n+n−fx1,x2σnx1−k1n,nx2−k2n=0.
For∑1, by the facts that x1−k1/n<1/n,x2−k2/n<1/n, for x1,x2∈−1,12, we obtain that
(32)−n<ki−nxi<n,i=1,2;kin+n−xi≤nxi−ki+nn+n≤2n,i=1,2.
Hence,(33)1n∑1fk1n+n,k2n+n−fx1,x2σnx1−k1n,nx2−k2n≤ωf;2n1n∑1σnx1−k1n,nx2−k2n≤322+1n2ωf;1n,
where we have used the inequality0≤σ≤3/4, and the fact that the number of the terms in ∑1 is no more than 2n+12. From (27)-(33), it follows that
(34)I1≤322+1n2ωf;1n.
Set(35)D≔∑−n−nnx1−1∑−n−nnx2−1+∑−n−nnx1−1∑nx2+1n+n+∑nx1+1n+n∑−n−nnx2−1+∑nx1+1n+n∑nx2+1n+n.
Then,(36)I2≤D1nσnx1−k1n,nx2−k2n−1+1n∑−n−nn+nσnx1−nx1n,nx2−k2n+1n∑−n−nn+nσnx1−nx1n,nx2−k2n+1n∑−n−nn+nσnx1−k1n,nx2−nx2n+1n∑−n−nn+nσnx1−k1n,nx2−nx2n≕I21+I22+I23+I24+I25.
Firstly, we have(37)I22=1n∑−n−nn+nσnx1−nx1n,nx2−k2n=1n∑x2−k2/n≥1/nσnx1−nx1n,nx2−k2n+1n∑x2−k2/n<1/nσnx1−nx1n,nx2−k2n.≕Δ1+Δ2.
Noting thatx2−k2/n≥1/n, we get Δ1=0 by the similar arguments for estimating ∑2+∑3+∑4 in (27). Therefore,
(38)Δ2=1n∑x2−k2/n<1/nσnx1−nx1n,nx2−k2n≤34n·2n+1≤32n1+12n.
Consequently,(39)I22≤32n1+12n.
Similarly, we have(40)I2i≤32n1+12n,i=3,4,5.
Thus, we already have(41)I22+I23+I24+I25≤6n1+12n.
Now, let us estimateI21. By
(42)1n∫−n−n−1nx1−1∫−n−n−1nx2−1σnx1−t1n,nx2−t2ndt1dt2=∫nx1−nx1+1/nnx1−−n−n+1/n∫nx2−nx2+1/nnx2−−n−n+1/nσt1,t2dt1dt2,1n≤nxi−nxi+1n≤2n;nxi−−n−n+1n≥1,i=1,2,we deduce that
(43)∫nx1−nx1+1/nnx1−−n−n+1/n∫nx2−nx2+1/nnx2−−n−n+1/nσt1,t2dt1dt2−∫01∫01σt1,t2dt1dt2≤4n,where we used the fact that the support of σt1,t2 is −1,12.
Similarly, by(44)1n∫−n−nnx1∫−n−nnx2σnx1−t1n,nx2−t2ndt1dt2=∫nx1−nx1+1/nnx1−−n−n+1/n∫nx2−nx2+1/nnx2−−n−n+1/nσt1,t2dt1dt2,0≤nxi−nxin≤1n;nxi−−n−nn>1−1n,i=1,2,we have
(45)∫nx1−nx1+1/nnx1−−n−n+1/n∫nx2−nx2+1/nnx2−−n−n+1/nσt1,t2dt1dt2−∫01∫01σt1,t2dt1dt2≤4n.
By (1) of Lemma5, (43), and (45), we have
(46)1n∑−n−nnx1−1∑−n−nnx2−1σnx1−k1n,nx2−k2n−∫01∫01σt1,t2dt1dt2≤4n.
By (2)-(4) of Lemma5, and the arguments similar to (43) and (45), we obtain that
(47)1n∑−n−nnx1−1∑nx2+1n+nσnx1−k1n,nx2−k2n−∫01∫−10σt1,t2dt1dt2≤4n,(48)1n∑nx1+1n+n∑−n−nnx2−1σnx1−k1n,nx2−k2n−∫−10∫01σt1,t2dt1dt2≤4n,(49)1n∑nx1+1n+n∑nx2+1n+nσnx1−k1n,nx2−k2n−∫−10∫−10σt1,t2dt1dt2≤4n.
By (46)-(49) and the identity ∫−11∫−11σt1,t2dt1dt2=1, we have
(50)I21=D1nσnx1−k1n,nx2−k2n−1≤16n.
It follows from (26), (34)-(41), and (50) that
(51)Nnf;x1,x2−fx1,x2≤322+1n2ωf;1n+2Mfn11+32n,x1,x2∈−1,12,
which completes the proof of Theorem6.


## 4. Numerical Experiments and Some Discussions

In this section, we give some numerical experiments to illustrate the theoretical results. We takefx1,x2=x12+x22,x1,x2∈−1,12 as the target function.Set(52)enx1,x2≔Nnf;x1,x2−fx1,x2,x1,x2∈−1,12.Figures1–3 show the results of e100x1,x2,e1000x1,x2 and e10000x1,x2, respectively. When n equals to 106, the amount of calculation of Nnf;x1,x2 is large. Therefore, we choose 6 specific points and the corresponding values of enx1,x2, which are shown in Table 1.Figure 1
Errors of approximation of network operators (6) with n=100.Figure 2
Errors of approximation of network operators (6) with n=1000.Figure 3
Errors of approximation of network operators (6) with n=10000.Table 1
The error values ofenx1,x2 for 6 specific points with n=1000000.
x1,x20,−1−1,10.5,0.50,00.5,−0.60.25,0.8enx1,x20.00200.0040-9.982e-043.992e-070.0012-0.0014From the results of experiments we see that as the parametern of neural network operators increases, the approximation effect increases; we only need to notice Mf=2,ωf;1/n=4/n, and after the simple calculation, we can demonstrate the validity of the obtained result.If we investigate network operators (6) carefully, we cannot help but ask why we use fk1/n+n,k2/n+n instead of fk1/n,k2/n in (6). Because k1/n,k2/n are the conventional grid points on −1,12, this will reduce the amount of calculation. Now, we might as well introduce the following network operators:
(53)N¯nf;x1,x2=1n∑−nn∑−nnfk1n,k2nσnx1−k1n,nx2−k2n.Then, from the proof of Theorem6, we have
(54)N¯nf;x1,x2−fx1,x2=1n∑−nn∑−nnfk1n,k2nσnx1−k1n,nx2−k2n−fx1,x2≤1n∑−nn∑−nnfk1n,k2n−fx1,x2σnx1−k1n,nx2−k2n+Mf1n∑−nn∑−nnσnx1−k1n,nx2−k2n−1≕I1′+MfI2′.It is not difficult to obtain the same estimate ofI′ as I1, but it is not inconvenient to estimate I2′. In fact, if we set
(55)e~nx1,x2≔N¯nf;x1,x2−fx1,x2,x1,x2∈−1,12.Figure4 shows the e~nx1,x2 with n=1000. We can see that next to the border of −1,12 the effect of N¯nf;x1,x2 approximating f is not satisfactory. Particularly, we can see this phenomenon from Table 2 below. So, we modified fk1/n,k2/n to construct operators (6). Then, we obtain the error estimation of approximation of operators (6) and give the numerical experiments.Figure 4
Errors of approximation of network operators (53) with n=1000.Table 2
The error values ofe~nx1,x2 and enx1,x2 for 6 specific points with n=10000.
x1,x20,−1−1,10.5,0.50,00.5,−0.60.25,0.8e~nx1,x2-0.5000-1.48622.749e-053.999e-052.47e-052.24e-05enx1,x2-0.0197-0.0394-0.00983.921e-05-0.0120-0.0138



---

*Source: 1713912-2022-09-28.xml*