# Estimation of Approximating Rate for Neural Network inLwp Spaces

Authors: Jian-Jun Wang; Chan-Yun Yang; Jia Jing
Journal: Journal of Applied Mathematics
 (2012)

---

## Abstract

A class of Soblove type multivariate function is approximated by feedforward network with one hidden layer of sigmoidal units and a linear output. By adopting a set of orthogonal polynomial basis and under certain assumptions for the governing activation functions of the neural network, the upper bound on the degree of approximation can be obtained for the class of Soblove functions. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.


---

## Body

## 1. Introduction

Artificial neural networks have been extensively applied in various fields of science and engineering. Why is so mainly because the feedforward neural networks (FNNs) have the universal approximation capability [1–13]. A typical example of such universal approximation assertions states that, for any given continuous function defined on a compact set K of ℛd, there exists a three-layer of FNN so that it can approximate the function arbitrarily well. A three-layer of FNN with one hidden layer, d inputs and one output can be mathematically expressed as(1.1)N(x)=∑i=1mciσ(∑j=1dwijxj+θi),x∈Rd,d≥1,
where 1≤i≤m,  θi∈ℛ are the thresholds, wi=(wi1,wi2,…,wid)T∈ℛd are connection weights of neuron i in the hidden layer with the input neurons, ci∈ℛ are the connection strength of neuron i with the output neuron, and σ is the activation function used in the network. The activation function is normally taken as sigmoid type; that is, it satisfies σ(t)→1 as t→+∞ and σ(t)→0 as t→-∞. Equation (1.1) can be further expressed in vector form as(1.2)N(x)=∑i=1mciσ(wi⋅x+θi),x∈Rd.Universal approximation capabilities for a broad range of neural network topologies have been established by researchers like Cybenko [1], Ito [5], and T. P. Chen and H. Chen [6]. Their work concentrated on the question of denseness. But from the point of application, we are concerned about the degree of approximation by neural networks.For any approximation problem, the establishment of performance bounds is an inevitable but very difficult issues. As we know, feedforward neural networks (FNNS) have been shown to be capable of approximating general class of functions, including continuous and integrable ones. Recently, several researchers have been derived approximation error bounds for various functional classes (see, e.g., [7–13]) approximated by neural networks. While many open issues remain concerning approximation degree, we stress in this paper on the issue of approximation of functions defined over [-1,1]d by FNNS. In [10], the researcher took some basics tools from the theory of weighted polynomial of functions (The weight function is ω(x)=exp(-Q(x))), under certain assumptions on the smoothness of functions being approximated and on the activation functions in the neural network, the authors present upper bounds on the degree of approximation achieved over the domain Rd.In this paper, using the Chebyshev Orthogonal series from the approximation theory and moduli of continuity, we obtain upper bounds on the degree of approximation in[-1,1]d. We take advantage of the properties of the Chebyshev polynomial and the methods of paper [10], we yield the desired results, which can be easily extended to the space Rd.


## 2. Multivariate Chebyshev Polynomial Approximation

Before introducing the main results, we firstly introduce some basic results on Chebyshev polynomials from the approximation theory. For convenience, we introduce a weighted norm of a functionf [14] given by(2.1)‖f‖p,ω=(∫[-1,1]dω(x)|f(x)|pdx)1/p,
where 1≤p<∞, ω(x)=∏i=1dω(xi) is multivariate weighted function, ω(xi)=(1-xi2)-1/2, x=(x1,x2,…,xd)∈Rd, m=(m1,m2,…,md)∈Zd, dx=dx1dx2…dxd. We denote the class of functions for which ∥f∥p,ω is finite by Lp,ω.For functionf:Rd→R, the class of functions we wish to approximate in this work is defined as follows:(2.2)Ψp,ωr,d={f:‖f(λ)‖p,ω≤M,|λ|≤r},
where λ=(λ1,λ2,…,λd), |λ|=λ1+λ2+⋯+λd, f(λ)=∂|λ|f/∂x1λ1…∂xdλd, r is a natural number, and M<∞.


### 2.1. A Chebyshev Polynomial Approximation of Multivariate Functions

As we know, Chebyshev polynomial of a single real variable is a very important polynomial in approximation theory. Using the above notation, we introduce multivariate Chebyshev polynomials:T0(x)=1/π, Tn(x)=∏i=1dTni(xi), Tni(xi)=2/πcos(niarccosxi). Evidently, for any m, l∈Zd, we have(2.3)∫[-1,1]dTm(x)Tl(x)ω(x)dx={1m=l,0m≠l.Forf∈Lp,ω, m∈Zk, let f̂(m)=∫[-1,1]df(x)Tm(x)ω(x)dx, then we have the orthogonal expansion f(x)~∑m=0∞f̂(m)Tm(x),x∈[-1,1]d.For one-dimension degree of approximation of a functiong by polynomials of degree m, one has the following:(2.4)Em(g,Pm,Lp,ω)=inf⁡P∈Pm‖g-P‖p,ω,
where Pm stands for the class of degree-m algebraic polynomials. From [15], we have a simple relationship which we will be used in the following. Let g be differentiable, then we have(2.5)E(g,Pm,Lp,ω)≤M1m-1E(g′,Pm,Lp,ω),E(g,Pm,Lp,ω)≤‖g‖p,ω.LetSn(f,t)=∑k=1n-1f̂(k)Tk(x), and the de la Valle Poussin Operators is defined, that is,(2.6)Vn(f,t)=1n+1∑m=n+3/2n+1Sn(f,t).Furthermore, we can simplifyVn(f,t) as follows:(2.7)Vn(f,t)=∑k=1nξkf̂(k)Tk(t),
where(2.8)ξk={m-12(m+1),if0≤k≤m+32,m-km+1,ifm+32≤k≤m.A basic result concerning Valle Poussin OperatorsVm(f,t) is(2.9)E2m(f,P2m,Lp,ω)≤‖f-Vmf‖p,ω≤Em(f,Pm,Lp,ω).Now we consider a class of multivariate polynomials defined as follows:(2.10)Pm={P:P(x)=∑0≤|i|≤|m|bi1,i2,…,idx1i1⋯xdid,bi1,i2,…,id∈R,∀i1,…,id}.Hence, we have the following theorem.Theorem 2.1.
For1≤p≤∞, let f∈Ψp,ωr,d. Then for any m=(m1,m2,…,md), mi≤m, we have
(2.11)inf⁡P∈Pm‖f-P‖p,ω≤Cm-r.Proof.
We consider the Chebyshev orthogonal polynomialsTm(x), and obtain the following equality from (2.7):
(2.12)Vi,mi,(f)=∑s=1miξsf̂s,iTs(xi),
where f̂s,i=∫[-1,1]df(x)Ts(xi)ω(xi)dxi. Hence, we define the following operators:
(2.13)V(f)=V1,m1V2,m2⋯Vd,mdf=∑s1=1m1⋯∑sd=1mdξs1⋯ξsdfs1,…,sdTs1(x1)⋯Tsd(xd),
where fs1,…,sd=∫[-1,1]d(∏i=1dω(xi)Tsi(xi))f(x)dx. Then we have
(2.14)‖f-V(f)‖p,ω=‖f-V1,m1(f)+V1,m1(f)-V1,m1V2,m2(f)+V1,m1V2,m2(f)-⋯-V(f)‖p,ω≤∑i=1d‖V0⋯Vi-1,mi-1f-V0⋯Vi,mif‖p,ω,
where V0 is the identity operator. Let g=V0⋯Vi-1,mi-1f, then Vi,mig=V0⋯Vi,mif, gri(x)=V0⋯Vi-1,mi-1Drif(x). We view Vi,mig as a one-dimensional function xi. Using (2.4), (2.5), and (2.6), we have
(2.15)‖g-Vi,mig‖p,ω≤C1Emi(g,Pmi,Lp,ω)≤C1M1ri(1mi)⋯(1mi-ri+1)Emi-ri(gri,Pmi-ri,Lp,ω)≤C1M1ri(1mi)⋯(1mi-ri+1)‖gri‖p,ω=Cri(1mi)⋯(1mi-ri+1)‖V0⋯Vi-1,mi-1Drif‖p,ω.
Letting ri=r,  mi=m,  i=1,…,d, if m>r(r-1), we get from (2.15), (2.13), (2.14) and the inequality ∏i=1n(1+ai)≥1+∑i=1nai,(ai≥-1),
(2.16)‖f-V(f)‖p,ω≤Cr∑i=1d(1m)⋯(1m-r+1)‖Drf‖p,ω≤CrdM(1m)⋯(1m-r+1)=CrdMm-r(1-1m)-1(1-2m)-1⋯(1-r-1m)≤CrdMm-r(1-r(r-1)2m)-1≤2dCrMm-r.
In order to obtain a bound valid for all m, for m≤r(r-1), we always have the trivial bound ∥f-V(f)∥p,ω≤M2 since ∥f∥p,ω≤M2. Letting C=max{2dCrdM,2-1M2(r(r-1))r}, we conclude an inequality of the desired type for every m.This theorem reveals two things: (i) for any multivariate functionsf∈Ψp,ωr,d, there is a polynomial P∈Pm that approximates f arbitrarily well in Lωp, (ii) quantitatively, the approximation accuracy of a polynomial P∈Pm can attain the order of ◯(m-r), where m is the dimension of multivariate polynomial, and r is the smoothness of the function to be approximated.


## 2.1. A Chebyshev Polynomial Approximation of Multivariate Functions

As we know, Chebyshev polynomial of a single real variable is a very important polynomial in approximation theory. Using the above notation, we introduce multivariate Chebyshev polynomials:T0(x)=1/π, Tn(x)=∏i=1dTni(xi), Tni(xi)=2/πcos(niarccosxi). Evidently, for any m, l∈Zd, we have(2.3)∫[-1,1]dTm(x)Tl(x)ω(x)dx={1m=l,0m≠l.Forf∈Lp,ω, m∈Zk, let f̂(m)=∫[-1,1]df(x)Tm(x)ω(x)dx, then we have the orthogonal expansion f(x)~∑m=0∞f̂(m)Tm(x),x∈[-1,1]d.For one-dimension degree of approximation of a functiong by polynomials of degree m, one has the following:(2.4)Em(g,Pm,Lp,ω)=inf⁡P∈Pm‖g-P‖p,ω,
where Pm stands for the class of degree-m algebraic polynomials. From [15], we have a simple relationship which we will be used in the following. Let g be differentiable, then we have(2.5)E(g,Pm,Lp,ω)≤M1m-1E(g′,Pm,Lp,ω),E(g,Pm,Lp,ω)≤‖g‖p,ω.LetSn(f,t)=∑k=1n-1f̂(k)Tk(x), and the de la Valle Poussin Operators is defined, that is,(2.6)Vn(f,t)=1n+1∑m=n+3/2n+1Sn(f,t).Furthermore, we can simplifyVn(f,t) as follows:(2.7)Vn(f,t)=∑k=1nξkf̂(k)Tk(t),
where(2.8)ξk={m-12(m+1),if0≤k≤m+32,m-km+1,ifm+32≤k≤m.A basic result concerning Valle Poussin OperatorsVm(f,t) is(2.9)E2m(f,P2m,Lp,ω)≤‖f-Vmf‖p,ω≤Em(f,Pm,Lp,ω).Now we consider a class of multivariate polynomials defined as follows:(2.10)Pm={P:P(x)=∑0≤|i|≤|m|bi1,i2,…,idx1i1⋯xdid,bi1,i2,…,id∈R,∀i1,…,id}.Hence, we have the following theorem.Theorem 2.1.
For1≤p≤∞, let f∈Ψp,ωr,d. Then for any m=(m1,m2,…,md), mi≤m, we have
(2.11)inf⁡P∈Pm‖f-P‖p,ω≤Cm-r.Proof.
We consider the Chebyshev orthogonal polynomialsTm(x), and obtain the following equality from (2.7):
(2.12)Vi,mi,(f)=∑s=1miξsf̂s,iTs(xi),
where f̂s,i=∫[-1,1]df(x)Ts(xi)ω(xi)dxi. Hence, we define the following operators:
(2.13)V(f)=V1,m1V2,m2⋯Vd,mdf=∑s1=1m1⋯∑sd=1mdξs1⋯ξsdfs1,…,sdTs1(x1)⋯Tsd(xd),
where fs1,…,sd=∫[-1,1]d(∏i=1dω(xi)Tsi(xi))f(x)dx. Then we have
(2.14)‖f-V(f)‖p,ω=‖f-V1,m1(f)+V1,m1(f)-V1,m1V2,m2(f)+V1,m1V2,m2(f)-⋯-V(f)‖p,ω≤∑i=1d‖V0⋯Vi-1,mi-1f-V0⋯Vi,mif‖p,ω,
where V0 is the identity operator. Let g=V0⋯Vi-1,mi-1f, then Vi,mig=V0⋯Vi,mif, gri(x)=V0⋯Vi-1,mi-1Drif(x). We view Vi,mig as a one-dimensional function xi. Using (2.4), (2.5), and (2.6), we have
(2.15)‖g-Vi,mig‖p,ω≤C1Emi(g,Pmi,Lp,ω)≤C1M1ri(1mi)⋯(1mi-ri+1)Emi-ri(gri,Pmi-ri,Lp,ω)≤C1M1ri(1mi)⋯(1mi-ri+1)‖gri‖p,ω=Cri(1mi)⋯(1mi-ri+1)‖V0⋯Vi-1,mi-1Drif‖p,ω.
Letting ri=r,  mi=m,  i=1,…,d, if m>r(r-1), we get from (2.15), (2.13), (2.14) and the inequality ∏i=1n(1+ai)≥1+∑i=1nai,(ai≥-1),
(2.16)‖f-V(f)‖p,ω≤Cr∑i=1d(1m)⋯(1m-r+1)‖Drf‖p,ω≤CrdM(1m)⋯(1m-r+1)=CrdMm-r(1-1m)-1(1-2m)-1⋯(1-r-1m)≤CrdMm-r(1-r(r-1)2m)-1≤2dCrMm-r.
In order to obtain a bound valid for all m, for m≤r(r-1), we always have the trivial bound ∥f-V(f)∥p,ω≤M2 since ∥f∥p,ω≤M2. Letting C=max{2dCrdM,2-1M2(r(r-1))r}, we conclude an inequality of the desired type for every m.This theorem reveals two things: (i) for any multivariate functionsf∈Ψp,ωr,d, there is a polynomial P∈Pm that approximates f arbitrarily well in Lωp, (ii) quantitatively, the approximation accuracy of a polynomial P∈Pm can attain the order of ◯(m-r), where m is the dimension of multivariate polynomial, and r is the smoothness of the function to be approximated.


## 3. Approximation by Feedforward Neural Networks

We consider the approximation of functions by feedforward neural networks with a ridge functions. We define the approximating function class composed of a single hidden layer feedforward neural network withn hidden units. The class of function is(3.1)Fn={f:f(x)=∑k=1ndkϕ(ak⋅x+bk);ak∈Rd,bk,dk∈R,k=1,2,…,n},
where ϕ(x) satisfy the following assumptions.(1)
There is a constantCϕ such that |ϕ(k)(x)|≥Cϕ>0,  k=0,1,…(2)
For each finitek, there is a finite constant lk such that  |ϕ(k)(x)|≤lk.We define the distance fromF to G as(3.2)dist⁡(F,G,Lp,ω)=sup⁡f∈Finf⁡g∈G‖f-g‖p,ω,
where F, G are two sets in Lωp. We have the following results.Theorem 3.1.
Let condition (1) and (2) hold for the activation functionϕ(x). Then for every 0<L<∞, m=(m1,m2,…,md)∈Z+d, mi≤m,ϵ>0 and n>(m+1)d, we have
(3.3)dist⁡(BPm(L),Fn,Lp,ω)≤ϵ,
where
(3.4)BPm(L)={P:p(x)=∑0≤s≤masxs;max⁡0≤s≤m⁡|as|≤L}.Proof.
Firstly, we consider the partial derivative(3.5)ϕ(s)(w⋅x+b)=∂(|s|)∂w1s1⋯∂wdsd(ϕ(w⋅x+b))=xsϕ|s|(w⋅x+b),
where |s|=s1+⋯+sd, and xs=∏i=1dxisi. Thus ϕ(s)(b)=xsϕ|s|(b).
For any fixedb and |x|<∞ (here |x|=∑i=1dxi), we consider a finite difference of orders
(3.6)∆h,xsϕ(b)=∑0≤l≤s(-1)|l|Cslϕ(hl⋅x+b)=xs∫0h⋯∫0hϕ(|s|)[((a1+⋯+as1)x1+⋯+(a|s|-sd+1+⋯+a|s|)xd)+b]da1⋯da|s|≐xsAh|s|ϕ(x),
where Csl=∏i=1dCsili,  ∆h,xsϕ(b)∈Fn with n=∏i=1d(1+si), So
(3.7)|ϕs(b)-h-|s|∆h,xsϕ(b)|=|xs(ϕ|s|(b)-h-|s|Ahsϕ|s|(x))|=|xs(ϕ|s|(b)-ϕ|s|(b+η))|≤Csω(ϕ|s|,h),
where we derive (3.7) by using (3.6), the mean value theorem of integral, (i.e., there is a η∈[0,h|s·x|], such that Ahsϕ|s|(x)=h|s|ϕ|s|(b+η)) and the moduli of continuity ω(g,h)=sup|t|≤h|f(x+t)-f(x)|.
From the definition ofdist(F,G,Lp,ω) and (3.7), we have
(3.8)dist⁡(BPm(L),Fn,Lp,ω)p≤‖∑0≤s≤masxs-∑0≤s≤mas∆h,xsϕ(b)hsϕ|s|(b)‖p,ωp≤(m+1)dmax⁡0≤s≤m⁡{|as|‖xs-∆h,xsϕ(b)hsϕ|s|(b)‖p,ωp}≤(m+1)dLmax⁡0≤s≤m⁡{ϕ|s|(b)}-pω(ϕ|s|,h)≤(m+1)dLCϕpω(ϕ|s|,h)<ϵ.
The last step ω(ϕ|s|,h) can be made arbitrarily small by letting h→0.
Using the Theorems2.1 and 3.1, we can easily establish our final result.Theorem 3.2.
For1≤p≤∞, we have
(3.9)dist⁡(Ψp,ωr,d,Fn,Lp,ω)≤Cn-r/d.This theorem reveals two things: (i) for any multivariate functionsf∈Ψp,ωr,d, there is a single hidden layer feedforward neural network N∈Fn with n hidden units that approximates f arbitrarily well in Lωp. That is, the feedforward neural networks can be used as the universal approximator of functions in Ψp,ωr,d; (ii) quantitatively, the approximation accuracy of a mixture network of the form (3.1) can attain the order of ◯(n-r/d), where d is the dimension of input space, and r is the smoothness of the function to be approximated.


## 4. Conclusion

In this work, the approximation order of feedforward neural networks with the form (3.1) has been studied. In terms of smoothness of a function, an upper bound estimation on approximation precision and speed of the neural networks is developed. Our research reveals that the approximation precision and speed of the neural networks depend not only on the number of hidden neurons used, but also on the smoothness of the functions to be approximated. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.



---

*Source: 636078-2012-05-28.xml*