# Lyapunov Stability Analysis of Gradient Descent-Learning Algorithm in Network Training

Authors: Ahmad Banakar
Journal: ISRN Applied Mathematics
 (2011)

---

## Abstract

The Lyapunov stability theorem is applied to guarantee the convergence and stability of the learning algorithm for several networks. Gradient descent learning algorithm and its developed algorithms are one of the most useful learning algorithms in developing the networks. To guarantee the stability and convergence of the learning process, the upper bound of the learning rates should be investigated. Here, the Lyapunov stability theorem was developed and applied to several networks in order to guaranty the stability of the learning algorithm.


---

## Body

## 1. Introduction

Science has evolved from an attempt to understand and predict the behavior of the universe and the systems within it. Much of this owes to the development of suitable models, which agree with the observations. These models are either in a symbolic form which the humans use or in mathematical form that are found from physical laws. Most systems are causal, which can be categorized as either static, where the output depends on the current inputs, or dynamic, where the output depends on not only the current inputs but also past inputs and outputs. Many systems also possess unobservable inputs, which cannot be measured, but affect the system’s output, that is, time series systems. These inputs are known as disturbances and aggravate the modeling process.To cope with the complexity of dynamic systems, there have been significant developments in the field of artificial neural network during last three decades which have been applied for identification and modeling [1–5]. One major issue that instigates for proposing these different types of networks is to predict the dynamic behavior of many complex systems existing in nature. ANN is a powerful method in approximating a nonlinear system and mapping between input and output data [1]. Recently, wavelet neural networks (WNNs) have been introduced [6–10]. Such types of networks employ wavelets as the activation function in a hidden layer. Because of the ability of the localized analysis of wavelets collectively in their frequency and time domains and the learning ability of ANN, the WNN prompts a superior system model for complex and seismic applications. The majority of the applications of wavelet function are limited to a small dimension [11] although WNN can handle large-dimension problems as well [6]. Due to the dynamic behavior of recurrent network, they are suitable in dealing with the modeling of dynamic systems as compared with the static behavior of feed-forward network [12–19]. It has already been shown that recurrent networks are less sensitive to noise with relatively smaller network size and simpler structure. Their long-term prediction property makes them more powerful in dealing with dynamic systems. Recurrent networks are less sensitive to noise because the recurrent network could recognize and generate periodic waves in spite of the existence of a large amount of noise. This means that the network is able to regenerate the original periodic waves in the process of learning the teachers' signals with noises [2]. For unknown dynamic systems, the recurrent network results in a smaller-sized network as compared with the feed-forward network [12, 20]. For the time-series modeling, it generates a simpler structure [15–23] and gives long-term predictions [22, 24]. The recurrent network for system modeling learns and memorizes information in terms of embedded weights [21].Different methods have been introduced for learning the parameters onnetwork based of the gradient descent. All learning methods like backpropagation-through-time [16, 17] or real-time recurrent learning algorithm [18] can be applied in order to adjust parameters of the feed-forward or recurrent networks. In [19], the quasi-Newton method was applied to improve the rate of convergence. In  [9, 23], using the Lyapunov stability theorem, a mathematical way was introduced for calculating the upper bound of the learning rate for recurrent and feed-forward wavelet neural network based on the network parameters. Here, the Lyapunov stability theorem is developed and applied to several networks, and the learning procedure of the proposed networks is considered.


## 2. Methodology

### 2.1. Gradient-Descent Algorithm

The Gradient-descent (GD) learning can be achieved by minimizing the performance indexJ as follows:(2.1)J=12⋅P⋅yr2⋅∑p=1P(Y(p)-Ŷ(p))2,
where yr=(maxp=1PY(p)-minp=1PY(p)), Ŷ is the output of the known network, Y is the actual data, and P is the number of dataset. The reason for using a normalized mean square error is that it provides a universal platform for modeling evaluation irrespective of the application and target value specification while selecting an input to the model.In the batch-learning scheme employing theP-data set, achange in any parameter is covered by the following equation:(2.2)Δυ(q)=∑p=1PΔpυ(q),
and the parametric update equation is(2.3)υ(q+1)=υ(q)+∂J∂υ.


### 2.2. Lyapunov Method in Analysis of Stability

Consider a dynamic system, which satisfies(2.4)ẋ=f(x,t),x(t0)=x0,x∈R.The equilibrium pointx*=0 is stable (in the sense of Lyapunov) at t=t0 if for any ε>0 there exists a δ(t0,ε)>0 such that (2.5)‖x(t0)‖<δ⟹‖x(t)‖<ε,∀t≥t0.Lyapunov Stability Theorem
LetV(x,t) be a nonnegative function with the derivative V̇ along the trajectories of the system. Then(i)
The origin of the system is locally stable (in the sense of Lyapunov) ifV(x,t) is locally positive definite and -V̇(x,t)≤0 is locally in x and for all t;(ii)
The origin of the system is globally uniformly asymptotically stable ifV(x,t) is positive definite and excrescent and -V̇(x,t) is positive definite.
To approve stability analysis of the networks based on GD learning algorithm, we can define discreet function as(2.6)V(k)=E(k)=12⋅[e(k)]2.
Change of Lyapunov function is
(2.7)ΔV(k)=V(k+1)-V(k)=12⋅[e2(k+1)-e2(k)].from(2.8)e(k+1)=e(k)+Δe(k)⟹e2(k+1)=e2(k)+Δ2e(k)+2⋅e(k)⋅Δe(k).
Then
(2.9)ΔV(k)=Δe(k)⋅[e(k)+12⋅Δe(k)].
Difference of error is
(2.10)Δe(k)=e(k+1)-e(k)≈[∂e(k)∂υ]T⋅Δυ,
where υ is the learning parameter and e(k)=ŷ(k)-y(k) is error between output of plant and present output of network
(2.11)Δυ=-η⋅∂J∂υ.
By using (2.10) and (2.1) and putting them in (2.3),
(2.12)ΔV(k)=[∂e(k)∂υ]T⋅Δυ⋅{e(k)+12⋅[∂e(k)∂υ]T⋅Δυ},ΔV(k)=[∂e(k)∂υ]T⋅(-η⋅∂E(k)∂υ)⋅{e(k)+12⋅[∂e(k)∂υ]T⋅(-η⋅∂E(k)∂υ)},ΔV(k)=[∂e(k)∂υ]T⋅(-η)⋅1P⋅yr2⋅e(k)⋅∂ŷ(k)∂υ⋅{e(k)+12⋅[∂e(k)∂υ]T⋅(-η)⋅1P⋅yr2⋅e(k)⋅∂ŷ(k)∂υ},ΔV(k)=e2(k)⋅{-[∂ŷ(k)∂υ]T⋅η⋅1P⋅yr2⋅∂ŷ(k)∂υ+12⋅[∂ŷ(k)∂υ]T⋅[∂ŷ(k)∂υ]T⋅η2⋅1(P⋅yr2)2⋅(∂ŷ(k)∂υ)2}ΔV(k)=-e2(k)⋅12⋅ηP⋅yr2⋅(∂ŷ(k)∂υ)2⋅{2-ηP⋅yr2⋅(∂ŷ(k)∂υ)2},
where yr=(maxp=1Py(p)-minp=1Py(p)).
Therefore(2.13)ΔV(k)=-λ⋅e2(k),
where  λ=(1/2)·(η/(P·yr2))·(∂ŷ(k)/∂υ)2·{2-(η/(P·yr2))·(∂ŷ(k)/∂υ)2}.
From the Lyapunov stability theorem, the stability is guaranteed ifV(k) is positive and V(k) is negative. From (2.6), V(k) is already positive. The condition of stability depends on V(k) being negative. Therefore, λ>0 is considered for all models.
Because(1/2)·(η/(P·yr2))·(∂ŷ(k)/∂υ)2>0, then the convergence condition is limited to
(2.14)2-ηP⋅yr2⋅(∂ŷ(k)∂υ)2>0⟹ηP⋅yr2⋅(∂ŷ(k)∂υ)2<2⟹η<(2⋅P⋅yr2)(∂ŷ(k)/∂υ)2.
The maximum learning rate η changes in a fixed range. Since 2·P·yr2 does not depend on the model, the value of ηMax guarantees that the convergence can be found by minimizing the term of |∂ŷ(k)/∂υI|. Therefore,
(2.15)0<η<ηMax⁡,
where  ηMax=(2·P·yr2)/Max(∂ŷ(k)/∂υ)2.


## 2.1. Gradient-Descent Algorithm

The Gradient-descent (GD) learning can be achieved by minimizing the performance indexJ as follows:(2.1)J=12⋅P⋅yr2⋅∑p=1P(Y(p)-Ŷ(p))2,
where yr=(maxp=1PY(p)-minp=1PY(p)), Ŷ is the output of the known network, Y is the actual data, and P is the number of dataset. The reason for using a normalized mean square error is that it provides a universal platform for modeling evaluation irrespective of the application and target value specification while selecting an input to the model.In the batch-learning scheme employing theP-data set, achange in any parameter is covered by the following equation:(2.2)Δυ(q)=∑p=1PΔpυ(q),
and the parametric update equation is(2.3)υ(q+1)=υ(q)+∂J∂υ.


## 2.2. Lyapunov Method in Analysis of Stability

Consider a dynamic system, which satisfies(2.4)ẋ=f(x,t),x(t0)=x0,x∈R.The equilibrium pointx*=0 is stable (in the sense of Lyapunov) at t=t0 if for any ε>0 there exists a δ(t0,ε)>0 such that (2.5)‖x(t0)‖<δ⟹‖x(t)‖<ε,∀t≥t0.Lyapunov Stability Theorem
LetV(x,t) be a nonnegative function with the derivative V̇ along the trajectories of the system. Then(i)
The origin of the system is locally stable (in the sense of Lyapunov) ifV(x,t) is locally positive definite and -V̇(x,t)≤0 is locally in x and for all t;(ii)
The origin of the system is globally uniformly asymptotically stable ifV(x,t) is positive definite and excrescent and -V̇(x,t) is positive definite.
To approve stability analysis of the networks based on GD learning algorithm, we can define discreet function as(2.6)V(k)=E(k)=12⋅[e(k)]2.
Change of Lyapunov function is
(2.7)ΔV(k)=V(k+1)-V(k)=12⋅[e2(k+1)-e2(k)].from(2.8)e(k+1)=e(k)+Δe(k)⟹e2(k+1)=e2(k)+Δ2e(k)+2⋅e(k)⋅Δe(k).
Then
(2.9)ΔV(k)=Δe(k)⋅[e(k)+12⋅Δe(k)].
Difference of error is
(2.10)Δe(k)=e(k+1)-e(k)≈[∂e(k)∂υ]T⋅Δυ,
where υ is the learning parameter and e(k)=ŷ(k)-y(k) is error between output of plant and present output of network
(2.11)Δυ=-η⋅∂J∂υ.
By using (2.10) and (2.1) and putting them in (2.3),
(2.12)ΔV(k)=[∂e(k)∂υ]T⋅Δυ⋅{e(k)+12⋅[∂e(k)∂υ]T⋅Δυ},ΔV(k)=[∂e(k)∂υ]T⋅(-η⋅∂E(k)∂υ)⋅{e(k)+12⋅[∂e(k)∂υ]T⋅(-η⋅∂E(k)∂υ)},ΔV(k)=[∂e(k)∂υ]T⋅(-η)⋅1P⋅yr2⋅e(k)⋅∂ŷ(k)∂υ⋅{e(k)+12⋅[∂e(k)∂υ]T⋅(-η)⋅1P⋅yr2⋅e(k)⋅∂ŷ(k)∂υ},ΔV(k)=e2(k)⋅{-[∂ŷ(k)∂υ]T⋅η⋅1P⋅yr2⋅∂ŷ(k)∂υ+12⋅[∂ŷ(k)∂υ]T⋅[∂ŷ(k)∂υ]T⋅η2⋅1(P⋅yr2)2⋅(∂ŷ(k)∂υ)2}ΔV(k)=-e2(k)⋅12⋅ηP⋅yr2⋅(∂ŷ(k)∂υ)2⋅{2-ηP⋅yr2⋅(∂ŷ(k)∂υ)2},
where yr=(maxp=1Py(p)-minp=1Py(p)).
Therefore(2.13)ΔV(k)=-λ⋅e2(k),
where  λ=(1/2)·(η/(P·yr2))·(∂ŷ(k)/∂υ)2·{2-(η/(P·yr2))·(∂ŷ(k)/∂υ)2}.
From the Lyapunov stability theorem, the stability is guaranteed ifV(k) is positive and V(k) is negative. From (2.6), V(k) is already positive. The condition of stability depends on V(k) being negative. Therefore, λ>0 is considered for all models.
Because(1/2)·(η/(P·yr2))·(∂ŷ(k)/∂υ)2>0, then the convergence condition is limited to
(2.14)2-ηP⋅yr2⋅(∂ŷ(k)∂υ)2>0⟹ηP⋅yr2⋅(∂ŷ(k)∂υ)2<2⟹η<(2⋅P⋅yr2)(∂ŷ(k)/∂υ)2.
The maximum learning rate η changes in a fixed range. Since 2·P·yr2 does not depend on the model, the value of ηMax guarantees that the convergence can be found by minimizing the term of |∂ŷ(k)/∂υI|. Therefore,
(2.15)0<η<ηMax⁡,
where  ηMax=(2·P·yr2)/Max(∂ŷ(k)/∂υ)2.


## 3. Experimental Results

In this section, the proposed stability analysis is applied for some networks. The selected networks are neurofuzzy (ANFIA) [25, 26], Wavelet neurofuzzy, and recurrent wavelet network.


### 3.1. Example 1: Convergence Theorems of the TSK Neurofuzzy Model

TSKmodel has a linear or nonlinear relationship of inputswm(X) in the output space. The rules of TSK model are in the following way:(3.1)Rm:ifxisAmthenyiswm(X).
A linear form of wm(X) in  (3.1) is as follows:(3.2)wm(X)=w0m+w1mx1+⋯+wnmxn.
By taking the Gaussian membership function and an equal number of fuzzy sets to the rules with respect to the inputs, the firing strength of rules (3.1) can be written as(3.3)μAm(x)=∏i=1nexp⁡(-(xi-x̅miσmi)2),
where x̅mi and σmi are the center and standard deviation of the Gaussian membership functions, respectively. By applying the T-norm (product operator) of the membership functions of the premise parts of the rule and the weighted average gravity method for de-fuzzification, the output of the TSK model can be defined as(3.4)Ŷ=∑m=1MμAm(x)⋅wm(x)∑m=1MμAm(x).Theorem 3.1.
The asymptotic learning convergence of TSK neurofuzzy is guaranteed if the learning rate for different learning parameters follows the upper bound as will be mentioned below:(3.5)0<ηw<2⋅P⋅yr2,0<ησ<2⋅P⋅yr2max⁡m|w(X)|2⋅(2/σmin⁡3)2,0<ηx̅<2⋅P⋅yr2max⁡m|w(X)|2⋅(2/σmin⁡2)2.Proof.
In equation (2.15) for neurofuzzy models can be written as
(3.6)0<ηυ<2⋅P⋅yr2|∂ŶNF/∂υ|max⁡2.
Because βm=μAm(X)/∑m=1MμAm(X)≤1 for all m and since local models have same variables, that is, X, therefore, from (3.7), (3.5)  easily can be derived
(3.7)∂ŶNF∂wm0=βm,∂ŶNF∂wmi=xi⋅βm,∂ŶNF∂x̅mi=wm(X′)⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)σmi2,∂ŶNF∂σmi=wm(X′)⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)2σmi3.


### 3.2. Example 2: Convergence Theorems of Recurrent Wavelet Neuron Models

Each neuron model in the proposed recurrent neuron models is summation or multiplication of Sigmoid Activation Function (SAF) and Wavelet Activation Function (WAF) as shown in Figure1. Morlet wavelet function is considered in the recurrent models. In the series of developing different recurrent networks and neuron models, the proposed neurons’ model is used in a one-hidden-layer feed-forward neural network as shown in Figure 2.Figure 1
Summation/product recurrent sigmoid-wavelet neuron model.Figure 2
Feed-forward neural network.The output of feed-forward network is given in   the following equation:(3.8)ŶWNN=∑l=1LWl⋅yl,
where yl is the output of S-W neurons, Wl is the weights between hidden neuron and output neurons, and L is the number of hidden neuron,(3.9)yj(k)=yjθ(k)+yjψ(k).The functionsyjθ and yjψ are output of SAF and WAF for jth S-W neuron, in the hidden layer, respectively. The functions yjθ and yjψ are expressed as follow.(3.10)yjθ(k)=θ(∑i=1nCSij⋅xi(k)),yjψ(k)=ψ(∑i=1nCWij⋅xi(k)).xi is ith  input. CS and CW are weights to input signal for SAF and WAF, in each hidden neuron, respectively.To prove convergence of the recurrent networks, these facts are needed:Fact 1: letg(y)=ye(-y2). Then |g(y)|<1,forally∈ℜ.Fact 2: letf(y)=y2e(-y2). Then |f(y)|<1,forally∈ℜ.Fact 3: letθ(y)=1/(1+e-y)be a sigmoid function. Then |θ(y)|<1,forally∈ℜFact 4: letψa,b(y)=e-((y-b)/a)2cos(5((y-b)/a)) be a Morlet wavelet function. Then |ψa,b(y)|<1,forally,a,b∈ℜ.(a) Summation Sigmoid-Recurrent Wavelet
SupposeZ=∑i=1nCSij·xi(k) and S=∑i=1nCWij·xi(k)+QWj·yψj(k-1).
From the facts 3 and 4: For parameterW in all models
(3.11)∂ŷ∂Wj=yj<|yψj+yθj|<1+1=2.
Therefore 0<ηW<(2·P·yr2)/22=(P·yr2)/2.
Differential of output of the model for another learning parameter is(3.12)∂ŷ(k)∂CWij=xi(k)⋅Wj⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηCW<(2·P·yr2)/72=(2·P·yr2)/49(3.13)∂ŷ(k)∂CSij=xi(k)⋅Wi⋅θ′(∑i=1nCSij⋅xi(k))<1⋅1⋅θ(z)⋅(1-θ(z))<1⋅1=1.
Therefore 0<ηCS<(2·P·yr2)/12=2·P·yr2(3.14)∂ŷ(k)∂QWj=Wj⋅yψj(k-1)⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηQW<(2·P·yr2)/72=(2·P·yr2)/49.(b) Multiplication Sigmoid-Recurrent Wavelet
From facts 3 and 4 supposeZ=∑i=1nCSij·xi(k) and S=∑i=1nCWij·xi(k)+QWj·yψj(k-1).
For parameterW in all networks:
(3.15)∂ŷ∂Wj=yj=yψj⋅yθj<1⋅1<1.
Therefore, 0<ηW<(2·P·yr2)/1<2·P·yr2(3.16)∂ŷ(k)∂CWij=xi(k)⋅Wj⋅θ(∑i=1nCSij⋅xi(k))⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηCW<(2·P·yr2)/(7)2=(2·P·yr2)/49(3.17)∂ŷ(k)∂CSij=xi(k)⋅Wj⋅θ′(∑i=1nCSij⋅xi(k))⋅ψ(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅θ(Z)⋅(1-θ(Z))⋅1<1⋅1<1.
Therefore, 0<ηCS<(2·P·yr2)/(1)2=2·P·yr2(3.18)∂ŷ(k)∂QWj=Wj⋅yψj(k-1)⋅θ(∑i=1nCSij⋅xi(k))⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηQW<(2·P·yr2)/(7)2=(2·P·yr2)/49


### 3.3. Example 3: Convergence Theorems of the Wavelet Nuro-Fuzzy (WNF) Model

The consequent part of each fuzzy rule corresponds to a sub-WNN consisting of wavelet with the specified dilation value, where, in the TSK fuzzy model, a linear function of inputs is used whilewm(X)=ŶWNNm. Figure 1 shows the proposed WNN model which uses a combination of sigmoid and wavelet activation functions as a hidden neuron (Figure 2 without recurrent part) in the consequent part of each fuzzy rule.Theorem 3.2.
The asymptotic learning convergence is guaranteed if the learning rate for different learning parameters follows the upper bound as will be mentioned below:(3.19)0<ησ<2⋅P⋅yr2|ŶWNN|max⁡2⋅(2/σmin⁡3)2,0<ηx̅<2⋅P⋅yr2|ŶWNN|max⁡2⋅(2/σmin⁡2)2,0<ηw<2⋅P⋅yr2|∂ŶWNN/∂w|max⁡2,0<ηCS<2⋅P⋅yr2|∂ŶWNN/∂CS|max⁡2,0<ηCW<2⋅P⋅yr2|∂ŶWNN/∂CW|max⁡2,whereηw, ηCN, or ηCW and ησ or ηx̅ are the parameters’ learning rates of the consequent and the premise parts of the fuzzy rules. CS and CW are weights to inputs, signal for sigmoid and wavelet activation functions of local WNNs, in each hidden neuron, respectively. x̅m and σm are the center and standard deviation of the Gaussian membership functions of rule number m in WNF model, respectively.Proof.
In equation (2.15) for WNF models can be written as
(3.20)0<ηυ<2⋅P⋅yr2|∂ŶWNF/∂υ|max⁡2,∂ŶWNF∂w=βm⋅∂ŶWNNm∂w,∂ŶWNF∂CN=βm⋅∂ŶWNNm∂CN,∂ŶWNF∂CW=βm⋅∂ŶWNNm∂CW.
Because βm=μAm(X)/∑m=1MμAm(X)≤1 for all m, therefore (3.13) to (3.15) are easily derived.
From (2.15) and (3.4) for parameters σ or x̅,there is
(3.21)∂ŶWNF∂σ=ŶWNNm⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)2σmi3=ŶWNNm⋅(1-βm)∑m=1MμAm⋅2⋅(xi-x̅mi)2σmi3,∂ŶWNF∂x̅=ŶWNNm⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)σmi2=ŶWNNm⋅(1-βm)∑m=1MμAm⋅2⋅(xi-x̅mi)σmi2
and therefore (3.19) arederived.


## 3.1. Example 1: Convergence Theorems of the TSK Neurofuzzy Model

TSKmodel has a linear or nonlinear relationship of inputswm(X) in the output space. The rules of TSK model are in the following way:(3.1)Rm:ifxisAmthenyiswm(X).
A linear form of wm(X) in  (3.1) is as follows:(3.2)wm(X)=w0m+w1mx1+⋯+wnmxn.
By taking the Gaussian membership function and an equal number of fuzzy sets to the rules with respect to the inputs, the firing strength of rules (3.1) can be written as(3.3)μAm(x)=∏i=1nexp⁡(-(xi-x̅miσmi)2),
where x̅mi and σmi are the center and standard deviation of the Gaussian membership functions, respectively. By applying the T-norm (product operator) of the membership functions of the premise parts of the rule and the weighted average gravity method for de-fuzzification, the output of the TSK model can be defined as(3.4)Ŷ=∑m=1MμAm(x)⋅wm(x)∑m=1MμAm(x).Theorem 3.1.
The asymptotic learning convergence of TSK neurofuzzy is guaranteed if the learning rate for different learning parameters follows the upper bound as will be mentioned below:(3.5)0<ηw<2⋅P⋅yr2,0<ησ<2⋅P⋅yr2max⁡m|w(X)|2⋅(2/σmin⁡3)2,0<ηx̅<2⋅P⋅yr2max⁡m|w(X)|2⋅(2/σmin⁡2)2.Proof.
In equation (2.15) for neurofuzzy models can be written as
(3.6)0<ηυ<2⋅P⋅yr2|∂ŶNF/∂υ|max⁡2.
Because βm=μAm(X)/∑m=1MμAm(X)≤1 for all m and since local models have same variables, that is, X, therefore, from (3.7), (3.5)  easily can be derived
(3.7)∂ŶNF∂wm0=βm,∂ŶNF∂wmi=xi⋅βm,∂ŶNF∂x̅mi=wm(X′)⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)σmi2,∂ŶNF∂σmi=wm(X′)⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)2σmi3.


## 3.2. Example 2: Convergence Theorems of Recurrent Wavelet Neuron Models

Each neuron model in the proposed recurrent neuron models is summation or multiplication of Sigmoid Activation Function (SAF) and Wavelet Activation Function (WAF) as shown in Figure1. Morlet wavelet function is considered in the recurrent models. In the series of developing different recurrent networks and neuron models, the proposed neurons’ model is used in a one-hidden-layer feed-forward neural network as shown in Figure 2.Figure 1
Summation/product recurrent sigmoid-wavelet neuron model.Figure 2
Feed-forward neural network.The output of feed-forward network is given in   the following equation:(3.8)ŶWNN=∑l=1LWl⋅yl,
where yl is the output of S-W neurons, Wl is the weights between hidden neuron and output neurons, and L is the number of hidden neuron,(3.9)yj(k)=yjθ(k)+yjψ(k).The functionsyjθ and yjψ are output of SAF and WAF for jth S-W neuron, in the hidden layer, respectively. The functions yjθ and yjψ are expressed as follow.(3.10)yjθ(k)=θ(∑i=1nCSij⋅xi(k)),yjψ(k)=ψ(∑i=1nCWij⋅xi(k)).xi is ith  input. CS and CW are weights to input signal for SAF and WAF, in each hidden neuron, respectively.To prove convergence of the recurrent networks, these facts are needed:Fact 1: letg(y)=ye(-y2). Then |g(y)|<1,forally∈ℜ.Fact 2: letf(y)=y2e(-y2). Then |f(y)|<1,forally∈ℜ.Fact 3: letθ(y)=1/(1+e-y)be a sigmoid function. Then |θ(y)|<1,forally∈ℜFact 4: letψa,b(y)=e-((y-b)/a)2cos(5((y-b)/a)) be a Morlet wavelet function. Then |ψa,b(y)|<1,forally,a,b∈ℜ.(a) Summation Sigmoid-Recurrent Wavelet
SupposeZ=∑i=1nCSij·xi(k) and S=∑i=1nCWij·xi(k)+QWj·yψj(k-1).
From the facts 3 and 4: For parameterW in all models
(3.11)∂ŷ∂Wj=yj<|yψj+yθj|<1+1=2.
Therefore 0<ηW<(2·P·yr2)/22=(P·yr2)/2.
Differential of output of the model for another learning parameter is(3.12)∂ŷ(k)∂CWij=xi(k)⋅Wj⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηCW<(2·P·yr2)/72=(2·P·yr2)/49(3.13)∂ŷ(k)∂CSij=xi(k)⋅Wi⋅θ′(∑i=1nCSij⋅xi(k))<1⋅1⋅θ(z)⋅(1-θ(z))<1⋅1=1.
Therefore 0<ηCS<(2·P·yr2)/12=2·P·yr2(3.14)∂ŷ(k)∂QWj=Wj⋅yψj(k-1)⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηQW<(2·P·yr2)/72=(2·P·yr2)/49.(b) Multiplication Sigmoid-Recurrent Wavelet
From facts 3 and 4 supposeZ=∑i=1nCSij·xi(k) and S=∑i=1nCWij·xi(k)+QWj·yψj(k-1).
For parameterW in all networks:
(3.15)∂ŷ∂Wj=yj=yψj⋅yθj<1⋅1<1.
Therefore, 0<ηW<(2·P·yr2)/1<2·P·yr2(3.16)∂ŷ(k)∂CWij=xi(k)⋅Wj⋅θ(∑i=1nCSij⋅xi(k))⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηCW<(2·P·yr2)/(7)2=(2·P·yr2)/49(3.17)∂ŷ(k)∂CSij=xi(k)⋅Wj⋅θ′(∑i=1nCSij⋅xi(k))⋅ψ(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅θ(Z)⋅(1-θ(Z))⋅1<1⋅1<1.
Therefore, 0<ηCS<(2·P·yr2)/(1)2=2·P·yr2(3.18)∂ŷ(k)∂QWj=Wj⋅yψj(k-1)⋅θ(∑i=1nCSij⋅xi(k))⋅ψ′(∑i=1nCWij⋅xi(k)+QWj⋅yψj(k-1))<1⋅1⋅1⋅|-2a⋅S-ba⋅e-((S-b)/a)2⋅cos⁡(5S-ba)-e-((S-b)/a)2⋅5a⋅sin(5S-ba)|<{2amin⁡⋅1⋅1+5amin⁡⋅1}<7.
Therefore, 0<ηQW<(2·P·yr2)/(7)2=(2·P·yr2)/49


## 3.3. Example 3: Convergence Theorems of the Wavelet Nuro-Fuzzy (WNF) Model

The consequent part of each fuzzy rule corresponds to a sub-WNN consisting of wavelet with the specified dilation value, where, in the TSK fuzzy model, a linear function of inputs is used whilewm(X)=ŶWNNm. Figure 1 shows the proposed WNN model which uses a combination of sigmoid and wavelet activation functions as a hidden neuron (Figure 2 without recurrent part) in the consequent part of each fuzzy rule.Theorem 3.2.
The asymptotic learning convergence is guaranteed if the learning rate for different learning parameters follows the upper bound as will be mentioned below:(3.19)0<ησ<2⋅P⋅yr2|ŶWNN|max⁡2⋅(2/σmin⁡3)2,0<ηx̅<2⋅P⋅yr2|ŶWNN|max⁡2⋅(2/σmin⁡2)2,0<ηw<2⋅P⋅yr2|∂ŶWNN/∂w|max⁡2,0<ηCS<2⋅P⋅yr2|∂ŶWNN/∂CS|max⁡2,0<ηCW<2⋅P⋅yr2|∂ŶWNN/∂CW|max⁡2,whereηw, ηCN, or ηCW and ησ or ηx̅ are the parameters’ learning rates of the consequent and the premise parts of the fuzzy rules. CS and CW are weights to inputs, signal for sigmoid and wavelet activation functions of local WNNs, in each hidden neuron, respectively. x̅m and σm are the center and standard deviation of the Gaussian membership functions of rule number m in WNF model, respectively.Proof.
In equation (2.15) for WNF models can be written as
(3.20)0<ηυ<2⋅P⋅yr2|∂ŶWNF/∂υ|max⁡2,∂ŶWNF∂w=βm⋅∂ŶWNNm∂w,∂ŶWNF∂CN=βm⋅∂ŶWNNm∂CN,∂ŶWNF∂CW=βm⋅∂ŶWNNm∂CW.
Because βm=μAm(X)/∑m=1MμAm(X)≤1 for all m, therefore (3.13) to (3.15) are easily derived.
From (2.15) and (3.4) for parameters σ or x̅,there is
(3.21)∂ŶWNF∂σ=ŶWNNm⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)2σmi3=ŶWNNm⋅(1-βm)∑m=1MμAm⋅2⋅(xi-x̅mi)2σmi3,∂ŶWNF∂x̅=ŶWNNm⋅βmμAm⋅(1-βm)⋅2⋅(xi-x̅mi)σmi2=ŶWNNm⋅(1-βm)∑m=1MμAm⋅2⋅(xi-x̅mi)σmi2
and therefore (3.19) arederived.


## 4. Conclusion

In this paper, a developed Lyapunov stability theorem was applied to guarantee the convergence of the gradient-descent learning algorithm in network training. The experimental examples showed that the upper bound of the learning parameter could be easily considered using this theorem. So, an adaptive learning algorithm can guaranty the fast and stable learning procedure.



---

*Source: 145801-2011-07-05.xml*