# Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks

Authors: Huisheng Zhang; Chao Zhang; Wei Wu
Journal: Discrete Dynamics in Nature and Society
 (2009)

---

## Abstract

The batch split-complex backpropagation (BSCBP) algorithm for training complex-valued neural networks is considered. For constant learning rate, it is proved that the error function of BSCBP algorithm is monotone during the training iteration process, and the gradient of the error function tends to zero. By adding a moderate condition, the weights sequence itself is also proved to be convergent. A numerical example is given to support the theoretical analysis.


---

## Body

## 1. Introduction

Neural networks
are widely used in the fields of control, signal processing, and time series
analysis [1].
Traditional neural networks' parameters are usually real numbers for dealing
with real-valued signals. However, complex-valued signals also appear in
practical applications. As a result, complex-valued neural network (CVNN),
whose weights, threshold values, input and output signals are all complex
numbers, is proposed [2, 3]. CVNN has been extensively
used in processing complex-valued signals [4]. By encoding real-valued signals into complex numbers,
CVNN also has shown more powerful capability than real-valued neural networks
in processing real-valued signals. For example, two-layered CVNN [5] can successfully solve the
XOR problem which cannot be solved by two-layered real-valued neural networks. CVNN
can be trained by two types of complex backpropagation (BP) algorithms: fully
complex BP algorithm and split-complex BP algorithm. Different from the fully
complex BP algorithm [6], the operation of activation function in the
split-complex BP algorithm is split into real part and imaginary part [2–4, 7], and this makes the split-complex BP algorithm avoid the occurrence of singular points in the adaptive training process. Complex BP algorithms can be done using either a batch method or an online method. In online training, weights are updated after the presentation of each training
example, while in batch training, weights are not updated until all of the
examples are inputted into the networks. Compared with batch learning, online
learning is hard to parallelize.The convergence of neural networks learning algorithms
is crucial for practical applications. The dynamical behaviors of many neural
networks have been extensively analyzed [8, 9]. However, the existing convergence results of complex
BP algorithm are mainly focusing on fully complex BP algorithm for two-layered
CVNN (see, e.g., [10, 11]) and the convergence of split-complex BP algorithm is
seldom investigated. Nitta [12] used CVNN as a complex adaptive pattern classifier
and presented some heuristic convergence results. The purpose of this paper is
to give some rigorous convergence results of batch split-complex BP (BSCBP)
algorithm for three-layered CVNN. The monotonicity of the error function during
the training iteration process is also guaranteed.The remainder of this paper is organized as follows. The three-layered CVNN model and the BSCBP algorithm are described in the next section. Section3 presents the main convergence theorem. A numerical
example is given in Section 4 to verify our theoretical results. The details of the convergence proof are provided in Section 5.
Some conclusions are drawn in Section 6.


## 2. Network Structure and Learning Method

Figure1 shows the structure of the network we considered in this paper. It is a three-layered CVNN consisting of L input neurons, M hidden neurons, and 1 output neuron. For any positive integer d,
the set of all d-dimensional complex vectors is denoted by ℂd and the set of all d-dimensional real vectors is denoted by ℝd.
Let us write wm=wmR+iwmI=(wm1,wm2,…,wmL)T∈ℂL as the weight vector between the input neurons
and mth hidden neuron, where wml=wmlR+iwmlI, wmlR and wmlI∈ℝ1, i=−1, m=1,…,M,
and l=1,…,L.
Similarly, write v=vR+ivI=(v1,v2,…,vM)T∈ℂM as the weight vector between the hidden
neurons and the output neuron, where vm=vmR+ivmI, vmR and vmI∈ℝ1, m=1,…,M.
For simplicity, all the weight vectors are incorporated into a total weight
vector(2.1)W=((w1)T,(w2)T,…,(wM)T,vT)T∈ℂM(L+1).For input signals z=(z1,z2,…,zL)T=x+iy∈ℂL,
where x=(x1,x2,…,xL)T∈ℝL,
and y=(y1,y2,…,yL)T∈ℝL,
the input of the mth hidden neuron is(2.2)Um=UmR+iUmI=∑l=1L(wmlRxl−wmlIyl)+i∑l=1L(wmlIxl+wmlRyl)=(wmR−wmI)⋅(xy)+i(wmIwmR)⋅(xy).Here “⋅” denotes the inner product of two vectors.Figure 1
CVVN withL−M−1 structure.For the sake of using BSCBP algorithm to train the
network, we consider the following popular real-imaginary-type activation
function [5]: (2.3)fC(U)=fR(UR)+ifR(UI)
for any U=UR+iUI∈ℂ1,
where fR is a real function (e.g., sigmoid function).
If we simply denote fR as f,
the output Hm for the hidden neuron m is given by (2.4)Hm=HmR+iHmI=f(UmR)+if(UmI). Similarly, the input of the
output neuron is (2.5)S=SR+iSI=∑m=1M(vmRHmR−vmIHmI)+i∑m=1M(vmIHmR+vmRHmI)=(vR−vI)⋅(HRHI)+i(vIvR)⋅(HRHI) and the output of the network is
given by (2.6)O=OR+iOI=g(SR)+ig(SI), where HR=(H1R,H2R,…,HMR)T,HI=(H1I,H2I,…,HMI)T,
and g is a real function.We remark that, in practice, there should be
thresholds involved in the above formulas for the output and hidden neurons.
Here we have omitted the bias so as to simplify the presentation and deduction.Let the network be supplied with a given set of
training examples{zq,dq}q=1Q⊂ℂL×ℂ1.
For each input zq=xq+iyq(1≤q≤Q) from the training set, we write Umq=Umq,R+iUmq,I(1≤m≤M) as the input for the hidden neuron m, Hmq=Hmq,R+iHmq,I(1≤m≤M) as the output for the hidden neuron m, Sq=Sq,R+iSq,I as the input to the output neuron, and Oq=Oq,R+iOq,I as the actual output. The square error
function of CVNN trained by BSCBP algorithm can be represented as
follows:(2.7)E(W)=12∑q=1Q(Oq−dq)(Oq−dq)*=12∑q=1Q[(Oq,R−dq,R)2+(Oq,I−dq,I)2]=∑q=1Q[μqR(Sq,R)+μqI(Sq,I)],where “*” signifies
complex conjugate, and(2.8)μqR(t)=12(g(t)−dq,R)2,μqI(t)=12(g(t)−dq,I)2,t∈ℝ1,1≤q≤Q.The purpose of the network
training is to find W⋆ which can minimize E(W).
The gradient method is often used to solve the minimization problem.
Writing(2.9)Hq=Hq,R+iHq,I=(H1q,R,H2q,R,…,HMq,R)T+i(H1q,I,H2q,I,…,HMq,I)T,and differentiating E(W) with respect to the real parts and imaginary
parts of the weight vectors, respectively, give (2.10)∂E(W)∂vR=∑q=1Q[μqR′(Sq,R)Hq,R+μqI′(Sq,I)Hq,I],(2.11)∂E(W)∂vI=∑q=1Q[−μqR′(Sq,R)Hq,I+μqI′(Sq,I)Hq,R],(2.12)∂E(W)∂wmR=∑q=1Q[μqR′(Sq,R)(vmRf′(Umq,R)xq−vmIf′(Umq,I)yq)+μqI′(Sq,I)(vmIf′(Umq,R)xq+vmRf′(Umq,I)yq)],1≤m≤M,(2.13)∂E(W)∂wmI=∑q=1Q[μqR′(Sq,R)(−vmRf′(Umq,R)yq−vmIf′(Umq,I)xq)+μqI′(Sq,I)(−vmIf′(Umq,R)yq+vmRf′(Umq,I)xq)],1≤m≤M. Starting from an arbitrary initial value W0 at time 0,
BSCBP algorithm updates the weight vector W iteratively by(2.14)Wn+1=Wn+ΔWn,n=0,1,…,where ΔWn=((Δw1n)T,…,(ΔwMn)T,Δvn)T)T,
with(2.15)Δwmn=−η(∂E(Wn)∂wmR+i∂E(Wn)∂wmI),m=1,…,M,Δvn=−η(∂E(Wn)∂vR+i∂E(Wn)∂vI).Here η>0 stands for the learning rate. Obviously, we
can rewrite (2.14)
and (2.15) by dealing with the real parts and the imaginary parts of the weights separately(2.16)Δwmn,R=wmn+1,R−wmn,R=−η∂E(Wn)∂wmR,Δwmn,I=wmn+1,I−wmn,I=−η∂E(Wn)∂wmI,Δvn,R=vn+1,R−vn,R=−η∂E(Wn)∂vR,Δvn,I=vn+1,I−vn,I=−η∂E(Wn)∂vI,where m=1,…,M.


## 3. Main Results

Throughout the
paper∥⋅∥ denotes the usual Euclidean norm. We need the
following assumptions:(A1)
there exists a constantc1>0 such that(3.1)maxt∈ℝ1{|f(t)|,|g(t)|,|f′(t)|,|g′(t)|,|f′′(t)|,|g′′(t)|}≤c1;(A2)
there exists a constantc2>0 such that ∥vn,R∥≤c2 and ∥vn,I∥≤c2 for all n=0,1,2,…;(A3)
the setΦ0={W∣(∂E(W)/∂wmR)=0,(∂E(W)/∂wmI)=0,(∂E(W)/∂vR)=0,(∂E(W)/∂vI)=0,m=1,…,M} contains only finite points.Theorem 3.1.
Suppose that Assumptions (A1) and (A2) are valid and that{Wn} are the weight vector sequence generated by
(2.14)–(2.16) with arbitrary initial values W0.
If η≤c8,
where c8 is a constant defined in (5.21) below, then one has
(i)
E(Wn+1)≤E(Wn),n=0,1,2,…;(ii)
limn→∞∥(∂E(Wn)/∂wmR)∥=0, limn→∞∥(∂E(Wn)/∂wmI)∥=0,    limn→∞∥(∂E(Wn)/∂vR)∥=0,
and limn→∞∥(∂E(Wn)/∂vI)∥=0,0≤m≤M. Furthermore, if Assumption (A3) also holds, then there exists a point  W⋆∈Φ0 such that
(iii)
limn→∞Wn=W⋆.The monotonicity of the error functionE(W) during the learning process is shown in the
statement (i). The statement (ii) indicates the convergence of the gradients
for the error function with respect to the real parts and the imaginary parts
of the weights. The statement (iii) points out that if the number of the
stationary points is finite, the sequence {Wn} will converge to a local minimum of the error
function.


## 4. Numerical Example

In this section, we illustrate the convergence behavior of BSCBP algorithm by using a simple numerical example. The well-known XOR problem is a benchmark in literature of neural networks. As in [5], the training samples of the encoded XOR problem for CVNN are presented as follows:(4.1){z1=−1−i,d1=1},{z2=−1+i,d2=0},{z3=1−i,d3=1+i},{z4=1+i,d4=i}.This example uses a network with one input neuron,
three hidden neurons, and one output neuron. The transfer function istansig(⋅) in MATLAB, which is a commonly used sigmoid
function. The learning rate η is set to be 0.1.
We carry out the test with the initial components of the weights stochastically
chosen in [−0.5,0.5]. Figure 2 shows that the gradient tends to zero and the square error decreases monotonically as the number of iteration increases and at last tends to a constant. This supports our theoretical findings.Figure 2
Convergence behavior of BSCBP algorithm for solving XOR problem. (sum of gradient norms =∑m=1M(∥Δwmn,R∥2+∥Δwmn.I∥2)+∥Δvn,R∥2+∥Δvn,I∥2.)


## 5. Proofs

In this
section, we first present two lemmas; then, we use them to prove the main
theorem.Lemma 5.1.
Suppose that the functionℰ:ℝ2M(L+1)→ℝ1 is continuous and differentiable on a compact
set Φ⊂ℝ2M(L+1)
and that Φ1={θ∣(∂ℰ(θ)/∂θ)=0} contains only finite points. If a sequence {θn}n=1∞⊂Φ satisfies(5.1)limn→∞∥θn+1−θn∥=0,limn→∞∥∂ℰ(θn)∂θ∥=0,then there exists a point θ⋆∈Φ1 such that limn→∞θn=θ⋆.Proof.
This result is
almost the same as [13, Theorem 14.1.5], and the detail of the proof is omitted.For any1≤q<Q,1≤m≤M and n=0,1,2,…,
write(5.2)Umn,q=Umn,q,R+iUmn,q,I=(wmn,R−wmn,I)⋅(xqyq)+i(wmn,Iwmn,R)⋅(xqyq),Hmn,q=Hmn,q,R+iHmn,q,I=f(Umn,q,R)+if(Umn,q,I),Hn,q,R=(H1n,q,R,…,HMn,q,R)T,Hn,q,I=(H1n,q,I,…,HMn,q,I)T,Sn,q=Sn,q,R+iSn,q,I=(vn,R−vn,I)⋅(Hn,q,RHn,q,I)+i(vn,Ivn,R)⋅(Hn,q,RHn,q,I),ψn,q,R=Hn+1,q,R−Hn,q,R,ψn,q,I=Hn+1,q,I−Hn,q,I.Lemma 5.2.
Suppose Assumptions (A1) and (A2)
hold, then for any1≤q≤Q and n=0,1,2,…,
one has
(5.3)|Oq,R|≤c0,|Oq,I|≤c0,∥Hn,q,R∥≤c0,∥Hn,q,I∥≤c0,(5.4)|μqR′(t)|≤c3,|μqI′(t)|≤c3,|μqR′′(t)|≤c3,|μqI′′(t)|≤c3,t∈ℝ1,(5.5)max{∥ψn,q,R∥2,∥ψn,q,I∥2}≤c4∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2),(5.6)∑q=1Q(μqR′(Sn,q,R)(Δvn,R−Δvn,I)⋅(Hn,q,RHn,q,I)+μqI′(Sn,q,I)(Δvn,IΔvn,R)⋅(Hn,q,RHn,q,I))=−1η(∥Δvn,R∥2+∥Δvn,I∥2),(5.7)∑q=1Q(μqR′(Sn,q,R)(vn,R−vn,I)⋅(ψn,q,Rψn,q,I)+μqI′(Sn,q,I)(vn,Ivn,R)⋅(ψn,q,Rψn,q,I))≤(c5−1η)∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2),(5.8)∑q=1Q(μqR′(Sn,q,R)(Δvn,R−Δvn,I)⋅(ψn,q,Rψn,q,I)+μqI′(Sn,q,I)(Δvn,IΔvn,R)⋅(ψn,q,Rψn,q,I))≤c6(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2),(5.9)12∑q=1Q(μqR′′(t1n,q)(Sn+1,q,R−Sn,q,R)2+μqI′′(t2n,q)(Sn+1,q,I−Sn,q,I)2)≤c7(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2), where ci(i=0,3,…,7) are constants independent of n and q,
each t1n,q∈ℝ1 lies on the segment between Sn+1,q,R and Sn,q,R,
and each t2n,q∈ℝ1 lies on the segment
between Sn+1,q,I and Sn,q,I.Proof.
The validation
of (5.3) can be easily got by (2.4)–(2.6) when the set of samples are fixed and
Assumptions (A1) and (A2) are satisfied. By (2.8), we have(5.10)μqR′(t)=g′(t)(g(t)−Oq,R),μqI′(t)=g′(t)(g(t)−Oq,I),μqR′′(t)=g′′(t)(g(t)−Oq,R)+(g′(t))2,μqI′′(t)=g′′(t)(g(t)−Oq,I)+(g′(t))2,1≤q≤Q,t∈ℝ1.Then (5.4)
follows directly from Assumption (A1) by defining c3=c1(c1+c0)+(c1)2.
It follows from (5.2), Assumption (A1),
the Mean-Value Theorem and the Cauchy-Schwartz Inequality that for any 1≤q≤Q and n=0,1,2,…, (5.11)∥ψn,q,R∥2=∥Hn+1,q,R−Hn,q,R∥2=∥(f(U1n+1,q,R)−f(U1n,q,R)⋮f(UMn+1,q,R)−f(UMn,q,R))∥2=∥(f′(s1n,q)(U1n+1,q,R−U1n,q,R)⋮f′(sMn,q)(UMn+1,q,R−UMn,q,R))∥2=∑m=1M(f′(smn,q)(Δwmn,R⋅xq−Δwmn,I⋅yq))2≤2c1∑m=1M((Δwmn,R⋅xq)2+(Δwmn,I⋅yq)2)≤2c1∑m=1M(∥Δwmn,R∥2∥xq∥2+∥Δwmn,I∥2∥yq∥2)≤c4∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2), where c4=2c1max1≤q≤Q{∥xq∥2,∥yq∥2} and each smn,q is on the segment between Umn+1,q,R and Umn,q,R for m=1,…,M.
Similarly we can get (5.12)∥ψn,q,I∥≤c4∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2). Thus, we have (5.5).
By (2.10), (2.11), (2.16), and (5.2), we have (5.13)∑q=1Q(μqR′(Sn,q,R)(Δvn,R−Δvn,I)⋅(Hn,q,RHn,q,I)+μqI′(Sn,q,I)(Δvn,IΔvn,R)⋅(Hn,q,RHn,q,I))=∑q=1Q(μqR′(Sn,q,R)Hn,q,R⋅Δvn,R+μqI′(Sn,q,I)Hn,q,I⋅Δvn,R−μqR′(Sn,q,R)Hn,q,I⋅Δvn,I+μqI′(Sn,q,I)Hn,q,R⋅Δvn,I)=∂E(Wn)∂vR⋅Δvn,R+∂E(Wn)∂vI⋅Δvn,I=−1η(∥Δvn,R∥2+∥Δvn,I∥2).
Next, we prove (5.7). By (2.2), (2.4), (5.2), and Taylor's formula, for any 1≤q≤Q, 1≤m≤M, and n=0,1,2,…,
we have(5.14)Hmn+1,q,R−Hmn,q,R=f(Umn+1,q,R)−f(Umn,q,R)=f′(Umn,q,R)(Umn+1,q,R−Umn,q,R)+12f′′(tmn,q,R)(Umn+1,q,R−Umn,q,R)2,(5.15)Hmn+1,q,I−Hmn,q,I=f(Umn+1,q,I)−f(Umn,q,I)=f′(Umn,q,I)(Umn+1,q,I−Umn,q,I)+12f′′(tmn,q,I)(Umn+1,q,I−Umn,q,I)2,where tmn,q,R is an intermediate point on the line segment
between the two points Umn+1,q,R and Umn,q,R,
and tmn,q,I between the two points Umn+1,q,I and Umn,q,I.
Thus according to (2.12), (2.13), (2.16), (5.2), (5.14), and (5.15), we have(5.16)∑q=1Q(μqR′(Sn,q,R)(vn,R−vn,I)⋅(ψn,q,Rψn,q,I)+μqI′(Sn,q,I)(vn,Ivn,R)⋅(ψn,q,Rψn,q,I))=∑q=1Q∑m=1M(μqR′(Sn,q,R)vmn,Rf′(Umn,q,R)(Δwmn,R−Δwmn,I)⋅(xqyq)−μqR′(Sn,q,R)vmn,If′(Umn,q,I)(Δwmn,IΔwmn,R)⋅(xqyq)+μqI′(Sn,q,I)vmn,If′(Umn,q,R)(Δwmn,R−Δwmn,I)⋅(xqyq)+μqI′(Sn,q,I)vmn,Rf′(Umn,q,I)(Δwmn,IΔwmn,R)⋅(xqyq))+δ1=∑m=1M((∑q=1Q[μqR′(Sn,q,R)(vmn,Rf′(Umn,q,R)xq−vmn,If′(Umn,q,I)yq)+μqI′(Sn,q,I)(vmn,If′(Umn,q,R)xq+vmn,Rf′(Umn,q,I)yq)])⋅Δwmn,R+(∑q=1Q[μqR′(Sn,q,R)(−vmn,Rf′(Umn,q,R)yq−vmn,If′(Umn,q,I)xq)+μqI′(Sn,q,I)(−vmn,If′(Umn,q,R)yq+vmn,Rf′(Umn,q,I)xq)])⋅Δwmn,I)+δ1=−1η∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+δ1,where(5.17)δ1=12∑q=1Q∑m=1M(μqR′(Sn,q,R)vmn,Rf′′(tmn,q,R)((Δwmn,R−Δwmn,I)⋅(xqyq))2−μqR′(Sn,q,R)vmn,If′′(tmn,q,I)((Δwmn,IΔwmn,R)⋅(xqyq))2+μqI′(Sn,q,I)vmn,If′′(tmn,q,R)((Δwmn,R−Δwmn,I)⋅(xqyq))2+μqI′(Sn,q,I)vmn,Rf′′(tmn,q,I)((Δwmn,IΔwmn,R)⋅(xqyq))2). Using Assumptions (A1) and (A2), (5.4), and triangular inequality, we immediately get(5.18)δ1≤|δ1|≤c5∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2),where c5=2Qc1c2c3max1≤q≤Q{∥xq∥2+∥yq∥2}.
Now, (5.7) results from (5.16) and (5.18).
According to (5.2), (5.4), and (5.5), we have(5.19)∑q=1Q(μqR′(Sn,q,R)(Δvn,R−Δvn,I)⋅(ψn,q,Rψn,q,I)+μqI′(Sn,q,I)(Δvn,IΔvn,R)⋅(ψn,q,Rψn,q,I))≤c3∑q=1Q(∥(Δvn,R−Δvn,I)∥∥(ψn,q,Rψn,q,I)∥+∥(Δvn,IΔvn,R)∥∥(ψn,q,Rψn,q,I)∥)≤12c3∑q=1Q(∥(Δvn,R−Δvn,I)∥2+∥(ψn,q,Rψn,q,I)∥2+∥(Δvn,IΔvn,R)∥2+∥(ψn,q,Rψn,q,I)∥2)=c3∑q=1Q(∥Δvn,R∥2+∥Δvn,I∥2+∥ψn,q,R∥2+∥ψn,q,I∥2)≤c3∑q=1Q(∥Δvn,R∥2+∥Δvn,I∥2+2c4∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2))≤c6(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2),(5.20)12∑q=1Q(μqR′′(t1n,q)(Sn+1,q,R−Sn,q,R)2+μqI′′(t2n,q)(Sn+1,q,I−Sn,q,I)2)≤c32∑q=1Q((Sn+1,q,R−Sn,q,R)2+(Sn+1,q,I−Sn,q,I)2)≤c32∑q=1Q(((Δvn,R−Δvn,I)⋅(Hn+1,q,RHn+1,q,I)+(vn,R−vn,I)⋅(ψn,q,Rψn,q,I))2+((Δvn,IΔvn,R)⋅(Hn+1,q,RHn+1,q,I)+(vn,Ivn,R)⋅(ψn,q,Rψn,q,I))2)≤c3Qmax{(c0)2+(c2)2}(∥Δvn,R∥2+∥Δvn,I∥2+∥ψn,q,R∥2+∥ψn,q,I∥2)≤c7(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2),where c6=Qc3max{1,2c4} and c7=Qc3max{(c0)2+(c2)2}max{1,2c4}.
So we obtain   (5.8) and (5.9).Now, we are ready to prove Theorem3.1
in terms of the above two lemmas.Proof of Theorem3.1.
(i) By (5.6)–(5.9) and the
Taylor's formula, we have(5.21)E(Wn+1)−E(Wn)=∑q=1Q(μqR(Sn+1,q,R)−μqR(Sn,q,R)+μqI(Sn+1,q,I)−μqI(Sn,q,I))=∑q=1Q(μqR′(Sn,q,R)(Sn+1,q,R−Sn,q,R)+μqI′(Sn,q,I)(Sn+1,q,I−Sn,q,I)+12μqR′′(t1n,q)(Sn+1,q,R−Sn,q,R)2+12μqI′′(t2n,q)(Sn+1,q,I−Sn,q,I)2)=∑q=1Q(μqR′(Sn,q,R)(Δvn,R−Δvn,I)⋅(Hn,q,RHn,q,I)+μqI′(Sn,q,I)(Δvn,IΔvn,R)⋅(Hn,q,RHn,q,I)+μqR′(Sn,q,R)(vn,R−vn,I)⋅(ψn,q,Rψn,q,I)+μqI′(Sn,q,I)(vn,Ivn,R)⋅(ψn,q,Rψn,q,I)+μqR′(Sn,q,R)(Δvn,R−Δvn,I)⋅(ψn,q,Rψn,q,I)+μqI′(Sn,q,I)(Δvn,IΔvn,R)⋅(ψn,q,Rψn,q,I)+12μqR′(t1n,q)(Sn+1,q,R−Sn,q,R)2+12μqI′′(t2n,q)(Sn+1,q,I−Sn,q,I)2)≤−1η(∥Δvn,R∥2+∥Δvn,I∥2)+(c5−1η)∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+c6(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2)+c7(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2)≤(c8−1η)(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2),where c8=c5+c6+c7, t1n,q∈ℝ1 is on the segment between Sn+1,q,R and Sn,q,R,
and t2n,q∈ℝ1 is on the segment between Sn+1,q,I and Sn,q,I.
Then we have(5.22)E(Wn+1)≤E(Wn)−(1η−c8)(∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2).Obviously, by choosing the
learning rate η to satisfy that(5.23)0<η<1c8,then we have(5.24)E(Wn+1)≤E(Wn),n=0,1,2,….
(ii) According to (2.16), we have(5.25)∑m=1M(∥Δwmn,R∥2+∥Δwmn,I∥2)+∥Δvn,R∥2+∥Δvn,I∥2=η2(∑m=1M(∥∂E(Wn)∂wmR∥2+∥∂E(Wn)∂wmI∥2)+∥∂E(Wn)∂vR∥2+∥∂E(Wn)∂vI∥2).
Combining with (5.21), we have(5.26)E(Wn+1)≤E(Wn)−α(∑m=1M(∥∂E(Wn)∂wmR∥2+∥∂E(Wn)∂wmI∥2)+∥∂E(Wn)∂vR∥2+∥∂E(Wn)∂vI∥2)≤…≤E(W0)−α∑k=0n(∑m=1M(∥∂E(Wk)∂wmR∥2+∥∂E(Wk)∂wmI∥2)+∥∂E(Wk)∂vR∥2+∥∂E(Wk)∂vI∥2),where α=(η−c8η2).
Since E(Wn+1)≥0,
there holds that(5.27)α∑k=0n(∑m=1M(∥∂E(Wk)∂wmR∥2+∥∂E(Wk)∂wmI∥2)+∥∂E(Wk)∂vR∥2+∥∂E(Wk)∂vI∥2)≤E(W0).Let n→∞,
then(5.28)α∑k=0∞(∑m=1M(∥∂E(Wk)∂wmR∥2+∥∂E(Wk)∂wmI∥2)+∥∂E(Wk)∂vR∥2+∥∂E(Wk)∂vI∥2)≤E(W0)<∞.So there holds that(5.29)limn→∞(∑m=1M(∥∂E(Wn)∂wmR∥2+∥∂E(Wn)∂wmI∥2)+∥∂E(Wn)∂vR∥2+∥∂E(Wn)∂vI∥2)=0,which implies
that(5.30)limn→∞∥∂E(Wn)∂wmR∥=limn→∞∥∂E(Wn)∂wmI∥=0,0≤m≤M,(5.31)limn→∞∥∂E(Wn)∂vR∥=limn→∞∥∂E(Wn)∂vI∥=0.
(iii) Write(5.32)θ=((w1R)T,…,(wMR)T,(w1I)T,…,(wMI)T,(vR)T,(vI)T)T,then E(W) can be looked as a function of θ,
which is denoted as ℰ(θ).
That is to say(5.33)E(W)≡ℰ(θ). Obviously, ℰ(θ) is a continuously differentiable real-valued
function and(5.34)∂ℰ(θ)∂vR=∂E(W)∂vR,∂ℰ(θ)∂vI=∂E(W)∂vI,∂ℰ(θ)∂wmR=∂E(W)∂wmR,∂ℰ(θ)∂wmI=∂E(W)∂wmI,m=1,…,M.Let(5.35)θn=((w1n,R)T,…,(wMn,R)T,(w1n,I)T,…,(wMn,I)T,(vn,R)T,(vn,I)T)T, then by (5.30)
and (5.31),
we have(5.36)limn→∞∥∂ℰ(θn)∂wmR∥=limn→∞∥∂ℰ(θn)∂wmI∥=0,0≤m≤M,(5.37)limn→∞∥∂ℰ(θn)∂vR∥=limn→∞∥∂ℰ(θn)∂vI∥=0.Thus we have(5.38)limn→∞∥∂ℰ(θn)∂θ∥=0.We use (2.16), (5.30), and (5.31) to obtain(5.39)limn→∞∥wmn+1,R−wmn,R∥=0,limn→∞∥wmn+1,I−wmn,I∥=0,m=1,…,M,(5.40)limn→∞∥vn+1,R−vn,R∥=0,limn→∞∥vn+1,I−vn,I∥=0.This leads to(5.41)limn→∞∥θn+1−θn∥=0. Furthermore, from Assumption
(A3) we know that the set {θ∣(∂ℰ(θ)/∂θ)=0} contains only finite points. Thus, the
sequence {θn}n=1∞ here satisfies all the conditions needed in
Lemma 5.1. As a result, there is a θ⋆ which satisfies that limn→∞θn=θ⋆.
Since θn consists of the real and imaginary parts of Wn,
we know that there is a W⋆ such that limn→∞Wn=W⋆.
We thus complete the proof.


## 6. Conclusion

In this paper,
some convergence results of BSCBP algorithm for CVNN are presented. An up-bound
of the learning rateη is given to guarantee both the monotonicity of
the error function and the convergence of the gradients for the error function.
It is also proved that the network weights vector tends to a local minimum if
there are only finite stable points for the error function. A numerical example
is given to support the theoretical findings. Our work can help the neural
network researchers to choose the appropriate activation function and learning
rate to guarantee the convergence of the algorithm when they use BSCBP
algorithm to train CVNN. We mention that the convergence results can be
extended to a more general case that the networks have several outputs and
hidden layers.



---

*Source: 329173-2009-04-22.xml*