# The Estimate for Approximation Error of Neural Network with Two Weights

Authors: Fanzi Zeng; Yuting Tang
Journal: The Scientific World Journal
 (2013)

---

## Abstract

The neural network with two weights is constructed and its approximation ability to any continuous functions is proved. For this neural network, the activation function is not confined to the odd functions. We prove that it can limitlessly approach any continuous function from limited close subset ofRm to Rn and any continuous function, which has limit at infinite place, from limitless close subset of Rm to Rn. This extends the nonlinear approximation ability of traditional BP neural network and RBF neural network.


---

## Body

## 1. Introduction

So far, many neural networks models have been presented and applied in pattern recognition, automatic control, signal processing, and aided decision. Among these models, BP (feedforward) neural networks and RBF (radial basis function) neural networks are widely used because of their nonlinear ability of approximation to any continuous function. Up to now, these two classes of neural networks have successfully been applied to approach any nonlinear continuous function defined on bounded close subsets [1–7]. However, the approximation result to any continuous function defined on unbounded set has been very rare [8]. This inspires us to look for a new network which is of more wide approximation ability and can approach any continuous function both on a bounded set and on an unbounded set.Wang et al. [9] presented a new neural network called neural networks with two weights on the base of combining the advantage of BP neural network with that of RBF neural network. This model can not only simulate BP neural network and RBF neural network, but also simulate neural networks with higher order. This neural network not only contains direction weight value with respect to BP network but also contains core weight value with respect to RBF network. The function of its neurons is of the following form:
(1)y=ϕ[∑j=1m(ωj(xj-zj)|ωj(xj-zj)|)s|ωj(xj-zj)|p+θ],
where y is the output of neurons, ϕ is the activation function, θ is the threshold value, ωj is the direction weight value, zj is the core weight value, xj is the input, and p and s are two parameters.In (1), when zj=0, p=0, and s=1, then (1) reduces to mathematical model BP network neurons; when ωj takes a fixed value, p=2 and s=0, then (1) becomes mathematical model RBF network neurons.In [10], the approximation operators with logarithmic sigmoidal function of a class of neural networks with two weights and a class of quasi-interpolation operators were investigated. Using these operators as approximation tools, the upper bounds of estimate errors were estimated.In [2], the authors considered single hidden layer neural networks:
(2)Nn(x)=c0+∑i=1nciϕ(ωix+θi),Nn:R⟶R,
where ci represents output weights, ϕ is the activaion function, θi is threshold value, and ωj is direction weight value.The main result in [2] is as follows.Theorem 1.
Supposeϕ is a bounded, strictly monotonously increasing, and odd function defined on R,f∈C[a,b], and n∈N. Then there exists a feedforward neural network (BP) with one hidden layer Nn→R defined by
(3)Nn(x)=c0+∑i=1nciϕ(ωix+θi),Nn:R⟶R,
such that
(4)supx∈[a,b]|Nn(x)-f(x)|≤52ω(f,b-an),
where for i=1,2,…,n(5)xi=a+ib-an,x0=a,m=supx∈Rϕ(x),ci=12m(f(xi)-f(xi-1)),dn=ϕ-1(m-m2n),c0=f(a)-∑i=1nciϕ(ωix+θi),θi=-dnnb-a[2a+(2i-1)b-an],ωi=2dnnb-a.In this paper, one is concerned with the neural networks with two weights and a single hidden layer:(6)Nn(x)=∑i=1nciϕ(lis(x-z)s|(x-z)|s|x-z|p-θi).One objective is to construct a new BP neural network which is different from the BP neural network in [2] and prove this network has the approximation ability to any nonlinear continuous function. Another objective is to prove that, by adjusting the values of two parameters s and p, the neural network with two weights can not only approximate any continuous functions defined on a bounded close subset, namely, it has the same approximation ability as BP neural network in [2], but also approximate any continuous function defined on an unbounded set. Namely, the neural network with two weights has a better approximation ability than BP neural network and RBF neural network.


## 2. Theoretical Result

We use the following notations: the symbolsN,N0, and R represent the set of positive integers, nonnegative integers, and real number, respectively. Using C[a,b] to denote the space of continuous real-valued functions defined on [a,b], it is equalled by the supremum norm ∥·∥. Let f be a real-valued function defined on [a,b]. η(f,δ), the modulus of continuity of f, is defined as η(f,δ)=sup0<h≤δ∥f(·+h)-f(·)∥. Then we have limδ→0η(f,δ)=0, and for any real number λ≥0,η(f,λδ)≤(1+λ)η(f,δ).The modulus of continuity is usually considered as the measure of the smoothness of function and the approximation error in approximation theory. The functionf is called Lipschitz α(0<α≤1) continuous and is written as f∈LipC(f)α, if there exists a constant C(f) such that η(f,δ)≤C(f)δα. C(h) denotes the positive constant dependents only on h and its value may be different at different occurrence.Our main result is as follows.Theorem 2.
Supposeϕ is a bounded, strictly monotonously increasing, and odd function defined on R,f∈C[a,b], and n∈N. Then there exists a feedforward neural network (BP) with one hidden layer Nn→R defined by
(7)Nn(x)=∑i=1nciϕ(ωix+θi),
such that
(8)supx∈[a,b]|Nn(x)-f(x)|≤(1+n+14n)ω(f,b-an),
where, for i=1,2,…,n,
(9)xi=a+ib-an,dn=ϕ-1(m2n-m),x0=a,m=supx∈R|ϕ(x)|,ci=12m(f(xi)-f(xi-1)),ωi=(dn-dn*)nb-a,θi=dn-(dn-dn*)nb-a[a+(i-1)b-an],dn*=ϕ-1(m-m2n).Proof.
Divide[a,b] into n equal intervals; each has length of (b-a)/n and let a=x0<x1<⋯<xn=b. For i=1,2,…,n, we set ci=(1/2m)(f(xi)-f(xi-1)), dn=ϕ-1((m/2n)-m), m=supx∈R|ϕ(x)|, θi=dn-((dn-dn*)n/(b-a))[a+(i-1)((b-a)/n)], dn*=ϕ-1(m-(m/2n)),ωi=(dn-dn*)n/(b-a).
For anyx∈[a,b], given n, we assume that x∈[xi-1,xi] and note that
(10)Nn(x)=∑i=1n12m(f(xi)-f(xi-1))ϕ(ωix+θi).
Fix x*∈[xi-1,xi] and choose f, so that Nn(xi*)=f(xi*). Let Gi(xi*)=ϕ(ωix+θi); it follows that
(11)Nn(x)-f(xi*)=Nn(x)-∑i=1n12m(f(xi)-f(xi-1))Gi(xi*)=∑i=0n12m(f(xi)-f(xi-1))[Gi(x)-Gi(xi*)].
There are two possible cases to consider: (i) x>xi*; (ii) x≤xi*.
Case (i). When x>xi*, we have, for i=1,2,…,n,
(12)0<Gi(x)-Gi(xi*)<Gi(xi)-Gi(xi*)<ϕ(ωi(a+ib-an)+θi)+m=ϕ(dn)+m=m2n.
Case (ii). When x≤xi*, we have, for i=1,2,…,n,
(13)0>Gi(x)-Gi(xi*)>Gi(xi-1)-m=ϕ(ωi(a+(i-1)b-an)+θi)-m=ϕ(dn*)-m=-m2n.
From (12) and (13), it follows that
(14)|Gi(x)-Gi(xi*)|≤m2n,i=1,2,…,n.
Thus from (14), we have
(15)Nn(x)-f(xi*)≤∑i=0n12m|f(xi)-f(xi-1)||Gi(x)-Gi(xi*)|≤n+12mη(f,b-an)×m2n=n+14nη(f,b-an).
Therefore, we have
(16)|Nn(x)-f(x)|≤|Nn(x)-f(xi*)|+|f(x)-f(xi*)|=(1+n+14n)η(f,b-an).
This ends the proof of Theorem 2.Remark 3.
The activation functions are assumed to be odd functions in [2], while in this paper, this assumption is removed. On the other hand, in the construction of neural networks, threshold values and direction weight values are different from those in [2].Theorem 4.
Supposeϕ is a bounded, strictly monotonously increasing, and odd function defined on R,f∈C[a,b], and n∈N, z∉[a,b]. Then there exists a neural network with two weights and one hidden layer Nn:[a,b]→R defined by
(17)Nn(x)=∑i=1nciϕ(lis(x-z)s|(x-z)|s|x-z|p-θi)
with p<s,p<0 such that
(18)supx∈[a,b]|Nn(x)-f(x)|≤(1+n+14n)η(f,b-an),
where ci is defined in Theorem 2 and  lis is equal to ωi in Theorem 2.Proof.
We make fractional transformationH:x→u, where u=(x-z)s/|x-z|p-s, x∈[a,b]. Since z∉[a,b], thus H is a continuous transformation. Hence f([a,b])=U is also a bounded close set; moreover, 0∉U.
There exists an inverse transformation for fractional transformationH-:u→x, where x=z+u1/s|u|(s-p)/sp,u∈U. So g(u)=f(x)=f(z+u1/s|u|(s-p)/sp) is a continuous real-valued function defined on bounded close set U. For g(u), from Theorem 2, there exists a neural network with two weights and one hidden layer Nn:[a,b]→R defined by
(19)Nn(u)=∑i=1nciϕ(lisu-θi)
such that
(20)g^(u)=∑i=1nciϕ(lisu-θi)=∑i=1nciϕ(lis(x-z)s|(x-z)|s|x-z|p-θi)=f^(x)
which satisfies
(21)sup|g(u)-g^(u)|≤(1+n+14n)η(f,b-an).
This ends the proof of Theorem 4.Remark 5.
Whenp<s,p<0, the network with two weights is not BP and RBF neural network; thus, Theorem 4 validates that the neural network with two weights has the same approximation ability as BP and RBF neural network on any bounded close set. When n→∞, from Theorem 4, this neural network can approximate any continuous function defined on a bounded close set.Theorem 6.
Supposeϕ is a bounded, strictly monotonously increasing, and odd function defined on R; D is an unbounded subset of R (all accumulation points are in D beside ∞) andf∈C(D), z∉D, limx→∞f(x)=b(constant) and then there exists a neural network with two weights and one hidden layer:
(22)Nn(x)=∑i=1nciϕ(lis(x-z)s|(x-z)|s|x-z|p-θi)
with p<s,p<0 such that
(23)f^(x)=∑i=1nciϕ(lis(x-z)s|(x-z)|s|x-z|p-θi)
which satisfies
(24)supx∈[a,b]|f^(x)-f(x)|<(1+n+14n)η(f,b-an).Proof.
We make fractional transformationH:x→u, where u=(x-z)s/|x-z|p-s, x∈D. Since z∉D, thus H is a contimuous transformation on D and as x→∞,u→0. Let H(∞)=0; H(x) is a continuous function on D∪{∞}. We prove that H(D∪{∞})=H(D)∪{0} is a close set. If v is an arbitrary accumulation point of H(D)∪{0}, then v=0 or v is an accumulation point of H(D). When v=0, z∈H(D)∪{0}. When v is an accumulation point of H(D), then there exists a sequence uk∈H(D) and another sequence xk∈D, such that xk→uk,uk→v≠0(k→∞). From fractional transformation, there exists u-,z=u-,xk→x- such that uk→u-. From uk→v≠0(k→∞) and inverse transformation H-1:x=z+u1/s|u|(s-p)/sp,u∈U, it follows that x→∞ does not hold. Hence x- is a finite point of D. Therefore x-∈D, v=u-∈H(D). So, H(D∪{∞})=H(D)∪{0} is a close set. Let
(25)g(u)=f(x)=f(z+u1/s|u|(s-p)/sp).
Then g(u) is a continuous function on H(D). Let g(0)=b=limx→∞f(x); it is easy to prove that g(u) is a continuous function on H(D)∪{0}. For a continuous function g(u) defined on H(D)∪{0}, by means of Theorem 4, there exists a neural network with two weights ∑i=1nciϕ(lis((x-z)s/|(x-z)|s)|x-z|p-θi) such that
(26)g^(u)=∑i=1nciϕ(ωi(x-z)s|(x-z)|s|x-z|p-θi)=f^(x)
which satisfies
(27)sup|g(u)-g^(u)|≤(1+n+14n)η(f,b-an).
This ends the proof of Theorem 6.Remark 7.
Whenp<s,p<0, the network with two weights is not BP and RBF neural network; thus, Theorem 6 validates that the neural network with two weights has the approximation ability that BP and RBF neural network do not have on any unbounded set. Namely, the network with two weights has a better approximation ability than BP and RBF neural network on any unbounded set. When n→∞, from Theorem 6, this neural network can approximate any continuous function defined on unbounded set.


## 3. Conclusions

In this paper, we construct a new BP neural network and prove that the network has the approximation ability to any nonlinear continuous function. In our result, the threshold values and direction weight values in [2] are different from those in our paper. Second, we prove that neural network with two weights has the same approximation ability as BP neural network and RBF neural network to any continuous function defined on any bounded close set; furthermore, we prove that neural network with two weights has a better approximation ability than BP neural network and RBF neural network to any continuous function defined on any unbounded set.



---

*Source: 935312-2013-12-28.xml*