# A Smoothing Interval Neural Network

Authors: Dakun Yang; Wei Wu
Journal: Discrete Dynamics in Nature and Society
 (2012)

---

## Abstract

In many applications, it is natural to use interval data to describe various kinds of uncertainties. This paper is concerned with an interval neural network with a hidden layer. For the original interval neural network, it might cause oscillation in the learning procedure as indicated in our numerical experiments. In this paper, a smoothing interval neural network is proposed to prevent the weights oscillation during the learning procedure. Here, by smoothing we mean that, in a neighborhood of the origin, we replace the absolute values of the weights by a smooth function of the weights in the hidden layer and output layer. The convergence of a gradient algorithm for training the smoothing interval neural network is proved. Supporting numerical experiments are provided.


---

## Body

## 1. Introduction

In the last two decades artificial neural networks have been successfully applied to various domains, including pattern recognition [1], forecasting [2, 3], and data mining [4, 5]. One of the most widely used neural networks is the feedforward neural network with the well-known error backpropagation learning algorithm. But in most neural network architectures, input variables and the predicted results are represented in the form of single point value, not in the form of intervals. However, in real-life situations, available information is often uncertain, imprecise, and incomplete, which can be represented by fuzzy data, a generalization of interval data. So in many applications it is more natural to treat the input variables and the predicted results in the form of intervals than a set of single-point value.Since multilayer feedforward neural networks have high capability as a universal approximator of nonlinear mappings [6–8], some methods via neural networks for handling interval data have been proposed. For instance, in [9], the BP algorithm [10, 11] was extended to the case of interval input vectors. In [12], the author proposed a new extension of backpropagation by using interval arithmetic which called Interval Arithmetic Back-propagation (IABP). This new algorithm permits the use of training samples and targets which can be indistinctly points and intervals. In [13], the author proposed a new model of multilayer perceptron based on interval arithmetic that facilitates handling input and output interval data, where weights and biases are single valued and not interval valued.However, weights oscillation phenomena during the learning procedure were observed in our numerical experiments for these interval neural networks models. In order to prevent the weights oscillation, a smoothing interval neuron is proposed in this paper. Here, by smoothing we mean that, in the activation function and in a neighborhood of the origin, we replace the absolute values of the weights by a smooth function of the weights. Gradient algorithms [14–17] are applied to train the smoothing interval neural network. The weak and strong convergence theorems of the algorithms are proved. Supporting numerical results are provided.The remainder of this paper is organized as follows. Some basic notations of interval analysis are described in Section2. The traditional interval neural network is introduced in Section 3. Section 4 is devoted to our smoothing interval neural network and the gradient algorithm. The convergence results of the gradient learning algorithm are shown in Section 5. Supporting numerical experiments are provided in Section 6. The appendix is devoted to the proof of the theorem.


## 2. Interval Arithmetic

Interval arithmetic as a tool appeared in numerical computing in late 1950s. Then the interval mathematic is a theory introduced by Moore [18] and Sunaga [19] in order to give control of errors in numeric computations. Fundamentals used in this paper are described below.Let us denote the intervals by uppercase letters such asA and the real numbers by lowercase letters such as a. An interval can be represented by its lower bounds L and upper bounds U as A=[aL,aU], or equivalently by its midpoint C and radius R as A=〈aC,aR〉, where
(2.1)aC=aL+aU2,aR=aU-aL2.
For intervals A=[aL,aU] and B=[bL,bU], the basic interval operations are defined by
(2.2)A+B=[aL+bL,aU+bU],A-B=[aL-bU,aU-bL],k·A={[k·aL,k·aU],k>0,[k·aU,k·aL],k<0,
where k is a constant.Iff is an increasing function, then the interval output is given by
(2.3)f(A)=[f(aL),f(aU)].
In this paper, we use the following weighted Euclidean distance for a pair of intervals A and B(2.4)d(A,B)=β(aC-bC)2+(1-β)(aR-bR)2,β∈(0,1).
The parameter β∈[0,1] facilitates giving more importance to the prediction of the output centres or to the prediction of the radii. For β=1 learning concentrates on the prediction of the output interval centre and no importance is given to the prediction of its radius. For β=0.5 both predictions (centres and radii) have the same weights in the objective function. For our purpose, we assume β∈(0,1).


## 3. Interval Neural Network

In this paper, we consider an interval neural network with three layers, where the input and output are interval value, the weights are real value. The numbers of neurons for the input, hidden and output layers areN,M,1, respectively. Let Wm=(wm1,wm2,…,wmN)T∈ℝN, m=1,2,…,M be the weight matrix connecting the input and the hidden layers. The weight vector connecting the hidden and the output layers is denoted by W0=(w0,1,w0,2,…,w0,M)T∈ℝM. To simplify the presentation, we write W=(W0T,W1T,…,WMT)T∈ℝNM+M. In the interval neural network, a nonlinear activation function f(x) is used in the hidden layer, and a linear activation function in the output layer.For an arbitrary interval-valued inputX=(X1,X2,…,XN), where Xi=〈xiC,xiR〉, i=1,2,…,N, as the weights of the proposed structure are real value, this linear combination results in a interval given by
(3.1)Sm=∑i=1N‍wmiXi=〈smC,smR〉=〈∑i=1N‍wmixiC,∑i=1N‍|wmi|xiR〉.
Then the output of the interval neuron in the hidden layer is given by
(3.2)Hm=f(Sm)=[f(smC-smR),f(smC+smR)]=〈hmC,hmR〉=〈f(sC-sR)+f(sC+sR)2,f(sC+sR)-f(sC-sR)2〉.
Finally, the output of the interval neuron in the output layer is given by
(3.3)Y=〈yC,yR〉,(3.4)yC=∑m=1M‍w0mhmC,(3.5)yR=∑m=1M‍|w0m|hmR.


## 4. Smoothing Interval Neural Network

### 4.1. Smoothing Interval Neural Network Structure

As revealed in the numerical experiment below in this paper, there appear weights oscillation phenomena during the learning procedure for the original interval neural network presented in the last section. In order to prevent the weights oscillation, we propose a smoothing interval neural network by replacing|wmi| and |w0m| with a smooth function φ(wmi) and φ(w0m) in (3.1) and (3.5). Then, the output of the smoothing interval neuron in the hidden layer is defined as
(4.1)Sm=∑i=1N‍wmiXi=〈∑i=1N‍wmixiC,∑i=1N‍φ(wmi)xiR〉,(4.2)Hm=f(Sm)=[f(smC-smR),f(smC+smR)]=〈hmC,hmR〉=〈f(smC-smR)+f(smC+smR)2,f(smC+smR)-f(smC-smR)2〉.
The output of the smoothing interval neuron in the output layer is given by
(4.3)yC=∑m=1M‍w0mhmC,yR=∑m=1M‍φ(w0m)hmR.
For our purpose, φ(x) can be chosen as any smooth function that approximates |x| near the origin. For definiteness and simplicity, we choose φ(x) as a polynomial function:
(4.4)φ(x)={-x,x≤-μ,φ^(x),-μ<x<μ,x,x≥μ,
where μ>0 is a small constant and
(4.5)φ^(x)=-18μ3x4+34μx2+38μ.
We observe that the above defined φ(x) is a convex function in C2(R), and it is identical to the absolute value function |x| outside the zero neighborhood (-μ,μ).


### 4.2. Gradient Algorithm of the Smoothing Interval Neural Network

Suppose that we are supplied with a training sample set{Xj,Oj}j=1J, where Xj’s and Oj’s are input and ideal output samples, respectively, as follows: Xj=(X1j,X2j,…,XNj)T, Xij=[xijL,xijU]=〈xijC,xijR〉, i=1,2,…,N, Oj=[ojL,ojU]=〈ojC,ojR〉. Our task is to find the weights W=(W0T,W1T,…,WMT)T such that
(4.6)Oj=Y(Xj),j=1,2,…,J.
But usually, the weight W=(W0T,W1T,…,WMT)T satisfying (4.6) does not exit and, instead, the aim of the network learning is to choose the weight W to minimize an error function of the smoothing interval neural network. By (2.4), a simple and typical error function is the quadratic error function:
(4.7)E(W)=12∑j=1J‍(β(ojC-yjC)2+(1-β)(ojR-yjR)2).
Let us denote fjC(t)=(1/2)(ojC-tjC)2, fjR(t)=(1/2)(ojR-tjR)2, j=1,2,…,J, t∈ℝ, then the error function (4.7) is rewritten as
(4.8)E(W)=∑j=1J‍(βfjC(y)+(1-β)fjR(y)).
Now, we introduce the gradient algorithm [15, 16] for the smoothing interval neural network. The gradient of the error function E(W) with respect to W0  is given by
(4.9)∂E(W)∂W0=∑j=1J‍(β(yjC-ojC)∂yjC∂W0+(1-β)(yjR-ojR)∂yjR∂W0)=∑j=1J‍(βfj'C(y)∂yjC∂W0+(1-β)fj'R(y)∂yjR∂W0),
where
(4.10)∂yjC∂W0=hjC,∂yjR∂W0=φ′(W0)hjR.
The gradient of the error function E(W) with respect to Wm,m=1,2,…,M is given by
(4.11)∂E(W)∂Wm=∑j=1J‍(β(yjC-ojC)∂yjC∂hjmC∂hjmC∂Wm+(1-β)(yjR-ojR)∂yjR∂hjmR∂hjmR∂Wm)=∑j=1J‍(βfj'C(y)∂yjC∂hjmC∂hjmC∂Wm+(1-β)fj'R(y)∂yjR∂hjmR∂hjmR∂Wm),
where
(4.12)∂yjC∂hjmC=w0m,∂yjR∂hjmR=φ(w0m),∂hjmC∂Wm=f′(sjmC-sjmR)(xjC-φ′(Wm)xjR)2+f′(sjmC+sjmR)(xjC+φ′(Wm)xjR)2,∂hjmR∂Wm=f′(sjmC+sjmR)(xjC+φ′(Wm)xjR)2-f′(sjmC-sjmR)(xjC-φ′(Wm)xjR)2.
In the learning procedure, the weights W are iteratively refined as follows:
(4.13)Wk+1=Wk+ΔWk,
where
(4.14)ΔWk=-η∂E(Wk)∂W,
where η>0 a constant learning rate and k=1,2,.…


## 4.1. Smoothing Interval Neural Network Structure

As revealed in the numerical experiment below in this paper, there appear weights oscillation phenomena during the learning procedure for the original interval neural network presented in the last section. In order to prevent the weights oscillation, we propose a smoothing interval neural network by replacing|wmi| and |w0m| with a smooth function φ(wmi) and φ(w0m) in (3.1) and (3.5). Then, the output of the smoothing interval neuron in the hidden layer is defined as
(4.1)Sm=∑i=1N‍wmiXi=〈∑i=1N‍wmixiC,∑i=1N‍φ(wmi)xiR〉,(4.2)Hm=f(Sm)=[f(smC-smR),f(smC+smR)]=〈hmC,hmR〉=〈f(smC-smR)+f(smC+smR)2,f(smC+smR)-f(smC-smR)2〉.
The output of the smoothing interval neuron in the output layer is given by
(4.3)yC=∑m=1M‍w0mhmC,yR=∑m=1M‍φ(w0m)hmR.
For our purpose, φ(x) can be chosen as any smooth function that approximates |x| near the origin. For definiteness and simplicity, we choose φ(x) as a polynomial function:
(4.4)φ(x)={-x,x≤-μ,φ^(x),-μ<x<μ,x,x≥μ,
where μ>0 is a small constant and
(4.5)φ^(x)=-18μ3x4+34μx2+38μ.
We observe that the above defined φ(x) is a convex function in C2(R), and it is identical to the absolute value function |x| outside the zero neighborhood (-μ,μ).


## 4.2. Gradient Algorithm of the Smoothing Interval Neural Network

Suppose that we are supplied with a training sample set{Xj,Oj}j=1J, where Xj’s and Oj’s are input and ideal output samples, respectively, as follows: Xj=(X1j,X2j,…,XNj)T, Xij=[xijL,xijU]=〈xijC,xijR〉, i=1,2,…,N, Oj=[ojL,ojU]=〈ojC,ojR〉. Our task is to find the weights W=(W0T,W1T,…,WMT)T such that
(4.6)Oj=Y(Xj),j=1,2,…,J.
But usually, the weight W=(W0T,W1T,…,WMT)T satisfying (4.6) does not exit and, instead, the aim of the network learning is to choose the weight W to minimize an error function of the smoothing interval neural network. By (2.4), a simple and typical error function is the quadratic error function:
(4.7)E(W)=12∑j=1J‍(β(ojC-yjC)2+(1-β)(ojR-yjR)2).
Let us denote fjC(t)=(1/2)(ojC-tjC)2, fjR(t)=(1/2)(ojR-tjR)2, j=1,2,…,J, t∈ℝ, then the error function (4.7) is rewritten as
(4.8)E(W)=∑j=1J‍(βfjC(y)+(1-β)fjR(y)).
Now, we introduce the gradient algorithm [15, 16] for the smoothing interval neural network. The gradient of the error function E(W) with respect to W0  is given by
(4.9)∂E(W)∂W0=∑j=1J‍(β(yjC-ojC)∂yjC∂W0+(1-β)(yjR-ojR)∂yjR∂W0)=∑j=1J‍(βfj'C(y)∂yjC∂W0+(1-β)fj'R(y)∂yjR∂W0),
where
(4.10)∂yjC∂W0=hjC,∂yjR∂W0=φ′(W0)hjR.
The gradient of the error function E(W) with respect to Wm,m=1,2,…,M is given by
(4.11)∂E(W)∂Wm=∑j=1J‍(β(yjC-ojC)∂yjC∂hjmC∂hjmC∂Wm+(1-β)(yjR-ojR)∂yjR∂hjmR∂hjmR∂Wm)=∑j=1J‍(βfj'C(y)∂yjC∂hjmC∂hjmC∂Wm+(1-β)fj'R(y)∂yjR∂hjmR∂hjmR∂Wm),
where
(4.12)∂yjC∂hjmC=w0m,∂yjR∂hjmR=φ(w0m),∂hjmC∂Wm=f′(sjmC-sjmR)(xjC-φ′(Wm)xjR)2+f′(sjmC+sjmR)(xjC+φ′(Wm)xjR)2,∂hjmR∂Wm=f′(sjmC+sjmR)(xjC+φ′(Wm)xjR)2-f′(sjmC-sjmR)(xjC-φ′(Wm)xjR)2.
In the learning procedure, the weights W are iteratively refined as follows:
(4.13)Wk+1=Wk+ΔWk,
where
(4.14)ΔWk=-η∂E(Wk)∂W,
where η>0 a constant learning rate and k=1,2,.…


## 5. Convergence Theorem for SINN

For anyx∈ℝn, its Euclidean norm is ∥x∥=∑i=1n‍xi2. Let Ω0={W∈Ω:EW(W)=0} be the stationary point set of the error function E(W), where Ω⊂ℝNM+M is a bounded region satisfying (A2) below. Let Ω0,s⊂ℝ be the projection of Ω0 onto the sth coordinate axis, that is,
(5.1)Ω0,s={ws∈ℝ:W=(w1,…,ws,…,wNM+M)T∈Ω0},
for s=1,2,…,NM+M. To analyze the convergence of the algorithm, we need the following assumptions. (A1)
|f(t)|, |f′(t)|, |f′′(t)| are uniformly bounded for t∈R.(A2)
There exists a bounded regionΩ⊂ℝn such that {Wk}⊂Ω(k∈ℕ).(A3)
The learning rateη is small enough such that (A.10) below is valid.(A4)
Ω0,s does not contain any interior point for every s=1,2,…,NM+M.Now we are ready to present one convergence theorem of the learning algorithms. Its proof is given in the appendix later on.Theorem 5.1.
Let the error functionE(W) be defined by (4.7), and the weight sequence {Wk} be generated by the learning procedure (4.13) and (4.14) for smoothing interval neuron with W0 being an arbitrary initial guess. If Assumptions (A1), (A2), and (A3) are valid, then we have
(5.2)E(Wk+1)≤E(Wk),(5.3)limk→∞∥EW(Wk)∥=0.
Furthermore, if Assumption (A4) also holds, there exists a point W*∈Ω0 such that
(5.4)limk→∞Wk=W*.


## 6. Numerical Experiment

We compare the performances of the interval neural network and the smoothing interval neural network by approximating a simple interval function(6.1)Y=0.01×(X+11)2.
In this example, the training set contains five training samples. Their midpoints are all 0 and their radii are (0.85522.62488.01010.29229.2885), respectively. The corresponding outputs of the samples are Y=〈yC,yR〉=0.01×(X+11)2.For the above two interval neural networks, the error functionE(W) is defined as in (4.7). But in order to see the error more clearly in the figures, we will also use the error D defined by
(6.2)D=lnE=ln(12∑j=1J‍(β(ojC-yjC)2+(1-β)(ojR-yjR)2)).The number of training iterations is 2000, the initial midpoint of weight vector is selected randomly from[-0.01,0.01], and two neurons are selected in the hidden layer. The fix learning rate is η=0.2, β=0.5, and μ=0.5.In the learning procedure for the interval neural network, we clearly see from Figure1(a) that the gradient norm is not convergent. Figure 2(a) shows that the error function D is oscillating and not convergent. On the contrary, we see from Figure 1(b) that the gradient norm of the smoothing interval neural network is convergent. Figure 2(b) shows that the error function D, as well as E, is monotone decreasing and convergent.Norm of gradient of the interval neural network and the smoothing interval neuron in the training.
(a)
Interval neural network
(b)
Smoothing interval neural networkValues of the error functionD for the interval neural network and the smoothing neural network.
(a)
Value ofD for interval neural network
(b)
Value ofD for smoothing interval neural networkFrom this numerical experiment, we can see that the proposed smoothing neural network can efficiently avoid the oscillation during the training process.



---

*Source: 456919-2012-12-29.xml*