# Convergence of an Online Split-Complex Gradient Algorithm for Complex-Valued Neural Networks

Authors: Huisheng Zhang; Dongpo Xu; Zhiping Wang
Journal: Discrete Dynamics in Nature and Society
 (2010)

---

## Abstract

The online gradient method has been widely used in training neural networks. We consider in this paper an online split-complex gradient algorithm for complex-valued neural networks. We choose an adaptive learning rate during the training procedure. Under certain conditions, by firstly showing the monotonicity of the error function, it is proved that the gradient of the error function tends to zero and the weight sequence tends to a fixed point. A numerical example is given to support the theoretical findings.


---

## Body

## 1. Introduction

In recent years, neural networks have been widely used because of their outstanding capability of approximating nonlinear models. As an important search method in optimization theory, gradient algorithm has been applied in various engineering fields, such as adaptive control and recursive parametrical estimation [1–3]. Gradient algorithm is also a popular training method for neural networks (when used to train neural networks with hidden layers, gradient algorithm is also called BP algorithm) and can be done either in the online or in the batch mode [4]. In online training, weights are updated after the presentation of each training example, while in batch training, weights are not updated until all of the examples are inputted into the networks. As a result, batch gradient training algorithm is always used when the number of training samples is relatively small. However, in the case that a very large number of training samples are available, online gradient training algorithm is preferred.Conventional neural networks' parameters are usually real numbers for dealing with real-valued signals [5, 6]. In many applications, however, the inputs and outputs of a system are best described as complex-valued signals and processing is done in complex space. In order to solve the problem in complex domain, complex-valued neural networks (CVNNs) have been proposed in recent years [7–9], which are the extensions of the usual real-valued neural networks to complex numbers. Accordingly, there are two types of generalized gradient training algorithm for complex-valued neural networks: fully complex gradient algorithm [10–12] and split-complex gradient algorithm [13, 14]; both of which can be processed in online mode and batch mode. It has been pointed out that the split-complex gradient algorithm can avoid the problems resulting from the singular points [14].Convergence is of primary importance for a training algorithm to be successfully used. There have been extensive research results concerning the convergence of gradient algorithm for real-valued neural networks (see, e.g., [15, 16] and the references cited therein), covering both of online mode and batch mode. In comparison, the convergence properties for complex gradient algorithm are seldom investigated. We refer the reader to [11, 12] for some convergence results of fully complex gradient algorithms and [17] for those of batch split-gradient algorithm. However, to the best of our knowledge, convergence analysis of online split-complex gradient (OSCG) algorithm for complex-valued neural networks has not yet been established in the literature, and this becomes our primary concern in this paper. Under certain conditions, by firstly showing the monotonicity of the error function, we prove that the gradient of the error function tends to zero and the weight sequence tends to a fixed point. A numerical example is also given to support the theoretical findings.The remainder of this paper is organized as follows. The CVNN model and the OSCG algorithm are described in the next section. Section3 presents the main results. The proofs of these results are postponed to Section 4. In Section 5 we give a numerical example to support our theoretical findings. The paper ends with some conclusions given in Section 6.


## 2. Network Structure and Learning Method

It has been shown that two-layered CVNN can solve many problems that cannot be solved by real-valued neural networks with less than three layers [13]. Thus, without loss of generalization, this paper considers a two-layered CVNN consisting of L input neurons and 1 output neuron. For any positive integer d, the set of all d-dimensional complex vectors is denoted by ℂd and the set of all d-dimensional real vectors is denoted by ℝd. Let us write w=wR+iwI=(w1,w2,…,wL)T∈ℂL as the weight vector between the input neurons and output neuron, where wl=wlR+iwlI, wlR and wlI∈ℝ1, i=-1, and l=1,…,L. For input signals z=(z1,z2,…,zL)T=x+iy∈ℂL, where x=(x1,x2,…,xL)T∈ℝL, and y=(y1,y2,…,yL)T∈ℝL, the input of the output neuron is(2.1)U=UR+iUI=∑l=1L(wlRxl-wlIyl)+i∑l=1L(wlIxl+wlRyl)=(wR-wI)·(xy)+i(wIwR)·(xy).
Here “·” denotes the inner product of two vectors.For the convenience of using OSCG algorithm to train the network, we consider the following popular real-imaginary-type activation function [13]:(2.2)fC(U)=fR(UR)+ifR(UI)
for any U=UR+iUI∈ℂ1, where fR is a real function (e.g., sigmoid function). If simply denoting fR as f, the network output O is given by(2.3)O=OR+iOI=f(UR)+if(UI).Let the network be supplied with a given set of training examples{zq,dq}q=1Q⊂ℂL×ℂ1. For each input zq=xq+iyq(1≤q≤Q) from the training set, we write Uq=Uq,R+iUq,I as the input for the output neuron and Oq=Oq,R+iOq,I as the actual output. The square error function can be represented as follows:(2.4)E(w)=12∑q=1Q(Oq-dq)(Oq-dq)*=12∑q=1Q[(Oq,R-dq,R)2+(Oq,I-dq,I)2]=∑q=1Q[μqR(Uq,R)+μqI(Uq,I)],
where “*" signifies complex conjugate, and(2.5)μqR(t)=12(f(t)-dq,R)2,μqI(t)=12(f(t)-dq,I)2,t∈ℝ1,1≤q≤Q.The neural network training problem is to look for the optimal choicew⋆ of the weights so as to minimize approximation error. The gradient method is often used to solve the minimization problem. Differentiating E(w) with respect to the real parts and imaginary parts of the weight vectors, respectively, gives(2.6)∂E(w)∂wR=∑q=1Q[μqR′(Uq,R)xq+μqI′(Uq,I)yq],(2.7)∂E(w)∂wI=∑q=1Q[-μqR′(Uq,R)yq+μqI′(Uq,I)xq].
Now we describe the OSCG algorithm. Given initial weights w0=w0,R+iw0,I at time 0, OSCG algorithm updates the weight vector w by dealing with the real part wR and wI separately:(2.8)wmQ+q,R=wmQ+q-1,R-ηm[μqR′(wmQ+q-1,R·xq-wmQ+q-1,I·yq)xq+μqI′(wmQ+q-1,I·xq+wmQ+q-1,R·yq)yq],wmQ+q,I=wmQ+q-1,I-ηm[-μqR′(wmQ+q-1,R·xq-wmQ+q-1,I·yq)yq+μqI′(wmQ+q-1,I·xq+wmQ+q-1,R·xq)yq],m=0,1,…,q=1,2,…,Q.
For k,q=1,2,…,Q, and m=0,1,…,  denote that(2.9)UmQ+k,q,R=wmQ+k,R·xq-wmQ+k,I·yq,UmQ+k,q,I=wmQ+k,I·xq+wmQ+k,R·yq,ΔwmQ+k,R=wmQ+k,R-wmQ+k-1,R,ΔwmQ+k,I=wmQ+k,I-wmQ+k-1,I,pRm,q,k=μqR′(UmQ+k-1,q,R)xq+μqI′(UmQ+k-1,q,I)yq,pIm,q,k=-μqR′(UmQ+k-1,q,R)yq+μqI′(UmQ+k-1,q,I)xq.
Then (2.8) can be rewritten as(2.10)ΔwmQ+q,R=-ηmpRm,q,q,ΔwmQ+q,I=-ηmpIm,q,q.
Given 0<η0≤1 and a positive constant N, we choose learning rate ηm as(2.11)1ηm+1=1ηm+N,m=0,1,….
Equation (2.11) can be rewritten as(2.12)ηm=η01+Nη0m=O(1m),
and this implies that(2.13)ηm≤1Nm.
This type of learning rate is often used in the neural network training [16].For the convergence analysis of OSCG algorithm, similar to the batch version of split-complex gradient algorithm [17], we shall need the following assumptions.(A1)
There exists a constantc1>0 such that(2.14)maxt∈ℝ1{|f(t)|,|f′(t)|,|f′′(t)|}≤c1.(A2)
The setΦ0={w∣∂E(w)/∂wR=0,∂E(w)/∂wI=0} contains only finite points.


## 3. Main Results

In this section, we will give several lemmas and the main convergence theorems. The proofs of those results are postponed to the next section.In order to derive the convergence theorem, we need to estimate the values of the error function (2.4) at two successive cycles of the training iteration. Denote that(3.1)rRm,k=pRm,k,k-pRm,k,rIm,k=pIm,k,k-pIm,k,m=0,1,…,k=1,…,Q,
where pRm,k≜pRm,k,1, and pIm,k≜pIm,k,1. The first lemma breaks the changes of error function (2.4) at two successive cycles of the training iteration into several terms.Lemma 3.1.
Suppose Assumption(A1) is valid. Then one has
(3.2)E(w(m+1)Q)-E(wmQ)=-ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-ηm[(∑q=1QpRm,q)·(∑k=1QrRm,k)+(∑q=1QpIm,q)·(∑k=1QrIm,k)]+∑q=1Q(ρm,q,R+ρm,q,I),
where ρm,q,R=(1/2)μqR′′(t1m,q)(U(m+1)Q,q,R-UmQ,q,R)2, ρm,q,I=(1/2)μqI′′(t2m,q)(U(m+1)Q,q,I-UmQ,q,I)2, each t1m,q∈ℝ1 lies on the segment between U(m+1)Q,q,R and UmQ,q,R, and each t2m,q∈ℝ1 lies on the segment between U(m+1)Q,q,I and UmQ,q,I.The second lemma gives the estimations on some terms of (3.2).Lemma 3.2.
Suppose Assumptions(A1) and (A2) hold, for 0<η0≤1, then one has
(3.3)∥rRm,k∥≤c2ηm∑q=1k-1(∥pRm,q∥+∥pIm,q∥),∥rIm,k∥≤c2ηm∑q=1k-1(∥pRm,q∥+∥pIm,q∥),k=2,…,Q,(3.4)|ηm((∑q=1QpRm,q)·(∑k=1QrRm,k)+(∑q=1QpIm,q)·(∑k=1QrIm,k))|≤c3ηm2((∑q=1Q∥pRm,q∥)2+(∑q=1Q∥pIm,q∥)2),(3.5)|∑q=1Q(ρm,q,R+ρm,q,I)|≤c4ηm2((∑q=1Q∥pRm,q∥)2+(∑q=1Q∥pIm,q∥)2),
where ci(i=2,3,4) are constants and m=0,1,….From Lemmas3.1 and 3.2, we can derive the following lemma.Lemma 3.3.
Suppose Assumptions(A1) and (A2) hold, for 0<η0≤1, then one has
(3.6)E(w(m+1)Q)-E(wmQ)≤-ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+c5ηm2∑k=1Q(∥pRm,k∥2+∥pIm,k∥2)
where c5 is a constant.With the above Lemmas3.1–3.3, we can prove the following monotonicity result of OSCG algorithm.Theorem 3.4.
Let{ηm} be given by (2.11) and let the weight sequence {wmQ} be generated by (2.8). Then under Assumption (A1), there are positive numbers Ñ and η̃ such that for any N>Ñ and 0<η0<min{1,η̃} one has
(3.7)E(w(m+1)Q)≤E(wmQ),∀m≥0.To give the convergence theorem, we also need the following estimation.Lemma 3.5.
Let{ηm} be given by (2.11). Then under Assumption (A1), there are the same positive numbers Ñ and η̃ chosen as Theorem 3.4 such that for any N>Ñ and 0<η0<min{1,η̃} one has
(3.8)∑m=1∞1m(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)<∞.The following lemma gives an estimate of a series, which is essential for the proof of the convergence theorem.Lemma 3.6 (see [16]).
Suppose that a series(3.9)∑n=1∞an2n
is convergent and an≥0. If there exists a constant c6>0 such that
(3.10)|an-an+1|<c6n,
then
(3.11)limn→∞an=0.The following lemma will be used to prove the convergence of the weight sequence.Lemma 3.7.
Suppose that the functionℰ:ℝ2L→ℝ1 is continuous and differentiable on a compact set Φ⊂ℝ2L and that Φ1={θ∣∂ℰ(θ)/∂(θ)=0} contains only finite points. If a sequence {θn}n=1∞⊂Φ satisfies
(3.12)limn→∞∥θn+1-θn∥=0,limn→∞∥∂ℰ(θn)∂θ∥=0,
then there exists a point θ⋆∈Φ1 such that limn→∞θn=θ⋆.Now we are ready to give the main convergence theorem.Theorem 3.8.
Let{ηm} be given by (2.11) and let the weight sequence {wn} be generated by (2.8). Then under Assumption (A1), there are positive numbers Ñ and η̃ such that for any N>Ñ and 0<η0<min{1,η̃} one has
(3.13)limn→∞∥∂E(wn)∂wR∥=0,limn→∞∥∂E(wn)∂wI∥=0.
Furthermore, if Assumption (A2) also holds, then there exists a point w⋆∈Φ0 such that
(3.14)limn→∞wn=w⋆.


## 4. Proofs

Proof of Lemma3.1.
Using Taylor's formula, we have(4.1)μqR(U(m+1)Q,q,R)-μqR(UmQ,q,R)=μqR′(UmQ,q,R)(U(m+1)Q,q,R-UmQ,q,R)+12μqR′′(t1m,q)(U(m+1)Q,q,R-UmQ,q,R)2=μqR′(UmQ,q,R)(w(m+1)Q,R-wmQ,R-w(m+1)Q,I+wmQ,I)·(xqyq)+12μqR′′(t1m,q)((w(m+1)Q,R-wmQ,R-w(m+1)Q,I+wmQ,I)·(xqyq))2,
where t1m,q lies on the segment between U(m+1)Q,q,R and UmQ,q,R. Similarly we also have a point t2m,q between U(m+1)Q,q,I and UmQ,q,I such that
(4.2)μqI(U(m+1)Q,q,I)-μqI(UmQ,q,I)=μqI′(UmQ,q,I)(U(m+1)Q,q,I-UmQ,q,I)+12μqI′′(t2m,q)(U(m+1)Q,q,I-UmQ,q,I)2=μqI′(UmQ,q,I)(w(m+1)Q,I-wmQ,Iw(m+1)Q,R-wmQ,R)·(xqyq)+12μqI′′(t2m,q)((w(m+1)Q,I-wmQ,Iw(m+1)Q,R-wmQ,R)·(xqyq))2.
From (2.8) and (2.10) we have
(4.3)w(m+1)Q,R-wmQ,R=∑q=1QΔwmQ+q,R=-ηm∑q=1QpRm,q,q,w(m+1)Q,I-wmQ,I=∑q=1QΔwmQ+q,I=-ηm∑q=1QpIm,q,q.
Combining (2.4), (2.9), (3.1), and (4.1)–(4.3), then we have
(4.4)E(w(m+1)Q)-E(wmQ)=∑q=1Q([μqR(U(m+1)Q,q,R)-μqR(UmQ,q,R)]+[μqI(U(m+1)Q,q,I)-μqI(UmQ,q,I)])=∑q=1Q(μqR′(UmQ,q,R)[-ηm∑k=1QpRm,k,k·xq+ηm∑k=1QpIm,k,k·yq]+μqI′(UmQ,q,I)[-ηm∑k=1QpIm,k,k·xq-ηm∑k=1QpRm,k,k·yq])+∑q=1Q(ρm,q,R+ρm,q,I)=∑q=1Q([μqR′(UmQ,q,R)xq+μqI′(UmQ,q,I)yq]∑k=1Q(-ηmrRm,k-ηmpRm,k)+[-μqR′(UmQ,q,R)yq+μqI′(UmQ,q,I)xq]∑k=1Q(-ηmrIm,k-ηmpIm,k))+∑q=1Q(ρm,q,R+ρm,q,I)=-ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-ηm[(∑q=1QpRm,q)·(∑k=1QrRm,k)+(∑q=1QpIm,q)·(∑k=1QrIm,k)]+∑q=1Q(ρm,q,R+ρm,q,I),
where
(4.5)ρm,q,R=12μqR′′(t1m,q)(U(m+1)Q,q,R-UmQ,q,R)2,ρm,q,I=12μqI′′(t2m,q)(U(m+1)Q,q,I-UmQ,q,I)2.Proof of Lemma3.2.
From (2.5) and Assumption (A1) we know that functions μqR(t), μqI(t), μqR′(t), μqI′(t), μqR′′(t), and μqI′′(t)(1≤q≤Q) are all bounded. Thus there is a constant c7 such that
(4.6)max{∥μqR(t)∥,∥μqI(t)∥,∥μqR′(t)∥,∥μqI′(t)∥,∥μqR′′(t)∥,∥μqI′′(t)∥}≤c7,t∈ℝ1,1≤q≤Q.
By (2.9), (2.10), (3.1), and the Mean-Value Theorem, for 2≤k≤Q and m=0,1,…, we have
(4.7)∥rRm,k∥=∥pRm,k,k-pRm,k∥=∥μkR′(UmQ+k-1,k,R)xk+μkI′(UmQ+k-1,k,I)yk-μkR′(UmQ,k,R)xk-μkI′(UmQ,k,I)yk∥=∥μkR′′(t3m,k)(UmQ+k-1,k,R-UmQ,k,R)xk+μkI′′(t4m,k)(UmQ+k-1,k,I-UmQ,k,I)yk∥=∥μkR′′(t3m,k)[(wmQ+k-1,R-wmQ,R)·xk-(wmQ+k-1,I-wmQ,I)·yk]xk+μkI′′(t4m,k)[(wmQ+k-1,I-wmQ,I)·xk+(wmQ+k-1,R-wmQ,R)·yk]yk∥≤c̃1[∥∑q=1k-1(-ηmrRm,q-ηmpRm,q)∥+∥∑q=1k-1(-ηmrIm,q-ηmpIm,q)∥]≤c̃1ηm∑q=1k-1[(∥pRm,q∥+∥pIm,q∥)+(∥rRm,q∥+∥rIm,q∥)],
where c̃1=c7max1≤q≤Q(∥xq∥2+∥yq∥2). Similarly we have
(4.8)∥rIm,k∥≤c̃1ηm∑q=1k-1[(∥pRm,q∥+∥pIm,q∥)+(∥rRm,q∥+∥rIm,q∥)],2≤k≤Q,m=0,1,….
In particular, as ∥rRm,1∥=∥rIm,1∥=0, for k=2, we can get
(4.9)max{∥rRm,2∥,∥rIm,2∥}≤c̃2ηm∑q=1k-1(∥pRm,q∥+∥pIm,q∥),
where c̃2=c̃1. For 3≤s≤Q, 2≤k≤s-1, suppose that
(4.10)∥rRm,k∥≤c̃sηm∑q=1k-1(∥pRm,q∥+∥pIm,q∥),∥rIm,k∥≤c̃sηm∑q=1k-1(∥pRm,q∥+∥pIm,q∥),
where c̃s are nonnegative constants. Recalling ηm≤1, then we have
(4.11)∥rRm,s∥≤c̃1ηm∑q=1s-1[(∥pRm,q∥+∥pIm,q∥)+(∥rRm,q∥+∥rIm,q∥)]=c̃1ηm[∑q=1s-1(∥pRm,q∥+∥pIm,q∥)+∑q=2s-1∥rRm,q∥+∑q=2s-1∥rIm,q∥]≤c̃1ηm[∑q=1s-1(∥pRm,q∥+∥pIm,q∥)+2∑q=2s-1(c̃qηm∑l=1q-1(∥pRm,l∥+∥pIm,l∥))]≤c̃1ηm[∑q=1s-1(∥pRm,q∥+∥pIm,q∥)+2Qηmc̃L∑l=1s-1(∥pRm,l∥+∥pIm,l∥)]≤c̃sηm∑q=1s-1(∥pRm,q∥+∥pIm,q∥)
where c̃s=1+2Qc̃L and c̃L=max{c̃1,c̃2,…,c̃s-1}. Similarly, we also have
(4.12)∥rRm,s∥≤c̃sηm∑q=1s-1(∥pRm,q∥+∥pIm,q∥).
Thus, by setting c2=max{c̃1,c̃2,…,c̃Q}, we have (3.3). Now we begin to prove (3.4). Using (3.3) and Cauchy-Schwartz inequality, we have
(4.13)|ηm((∑q=1QpRm,q)·(∑k=1QrRm,k)+(∑q=1QpIm,q)·(∑k=1QrIm,k))|≤ηm∑q=1Q∥pRm,q∥×∑k=1Q(c2ηm∑l=1k-1(∥pRm,l∥+∥pIm,l∥))+ηm∑q=1Q∥pIm,q∥×∑k=1Q(c2ηm∑l=1k-1(∥pRm,l∥+∥pIm,l∥))≤c2Qηm2∑q=1Q∥pRm,q∥×∑l=1Q(∥pRm,l∥+∥pIm,l∥)+c2Qηm2∑q=1Q∥pIm,q∥×∑l=1Q(∥pRm,l∥+∥pIm,l∥)≤c8ηm2((∑q=1Q∥pRm,q∥)2+(∑q=1Q∥pIm,q∥)2),
where c8=2c2Q. This validates (3.4). Finally, we show (3.5). Using (2.10), (3.1), (3.3), and (4.3), we have
(4.14)∥w(m+1)Q,R-wmQ,R∥=ηm∥∑k=1Q(rRm,k+pRm,k)∥≤ηm∑k=1Q∥pRm,k∥+c2ηm2∑k=1Q∑q=1k-1(∥pRm,q∥+∥pIm,q∥)≤ηm∑k=1Q(∥pRm,k∥+∥pIm,k∥)+c2Qηm2∑q=1Q(∥pRm,q∥+∥pIm,q∥)≤c9ηm∑k=1Q(∥pRm,k∥+∥pIm,k∥),
where c9=c2Q+1. Similarly we also have
(4.15)∥w(m+1)Q,I-wmQ,I∥≤c9ηm∑k=1Q(∥pRm,k∥+∥pIm,k∥).
This together with (2.9) and (4.6) leads to
(4.16)|∑q=1Q(ρm,q,R+ρm,q,I)|=12|∑q=1Q(μqR′′(t1m,q)(U(m+1)Q,q,R-UmQ,q,R)2+μqI′′(t2m,q)(U(m+1)Q,q,I-UmQ,q,I)2)|=12|∑q=1Q[μqR′′(t1m,q)((w(m+1)Q,R-wmQ,R-w(m+1)Q,I+wmQ,I)·(xqyq))2+μqI′′(t2m,q)((w(m+1)Q,I-wmQ,Iw(m+1)Q,R-wmQ,R)·(xqyq))2]|≤c10(∥w(m+1)Q,R-wmQ,R∥2+∥w(m+1)Q,I-wmQ,I∥2)≤2c9c10ηm2(∑k=1Q(∥pRm,k∥+∥pIm,k∥))2≤c4ηm2((∑q=1Q∥pRm,q∥)2+(∑q=1Q∥pIm,q∥)2),
where c10=Qc7max1≤q≤Q{∥xq∥2+∥yq∥2} and c4=4c9c10. This completes the proof.Proof of Lemma3.3.
Recalling Lemmas3.1 and 3.2, we conclude that
(4.17)E(w(m+1)Q)-E(wmQ)≤-ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+(c3+c4)ηm2((∑q=1Q∥pRm,q∥)2+(∑q=1Q∥pIm,q∥)2)≤-ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+Q(c3+c4)ηm2∑q=1Q(∥pRm,q∥2+∑q=1Q∥pIm,q∥2).
Then (3.6) is obtained by letting c5=Q(c3+c4).Proof of Theorem3.4.
In virtue of (3.6), the core to prove this lemma is to verify that
(4.18)ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)≥c5ηm2∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
In the following we will prove (4.18) by induction. First we take η0 such that
(4.19)η0(∥∑k=1QpR0,k∥2+∥∑k=1QpI0,k∥2)≥c5η02∑k=1Q(∥pR0,k∥2+∥pI0,k∥2).
For m≥0 suppose that
(4.20)ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)≥c5ηm2∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
Next we will prove that
(4.21)ηm+1(∥∑k=1QpRm+1,k∥2+∥∑k=1QpIm+1,k∥2)≥c5ηm+12∑k=1Q(∥pRm+1,k∥2+∥pIm+1,k∥2).
Notice that
(4.22)pRm+1,k-pRm,k=(μkR′(U(m+1)Q,k,R)-μkR′(UmQ,k,R))xk+(μkI′(U(m+1)Q,k,I)-μkI′(UmQ,k,I))yk=(μkR′′(t5m,k)(U(m+1)Q,k,R-UmQ,k,R))xk+(μkI′′(t6m,k)(U(m+1)Q,k,I-UmQ,k,I))yk=μkR′′(t5m,k)((w(m+1)Q,R-wmQ,R)·xk-(w(m+1)Q,I-wmQ,I)·yk)xk+μkI′′(t6m,k)((w(m+1)Q,I-wmQ,I)·xk+(w(m+1)Q,R-wmQ,R)·yk)yk,
where t5m,k lies on the segment between U(m+1)Q,k,R and UmQ,k,R, and t6m,k lies on the segment between U(m+1)Q,k,I and UmQ,k,I. Similar to (4.14), we also have the following estimation:
(4.23)∥w(m+1)Q,R-wmQ,R∥=ηm∥∑k=1Q(rRm,k+pRm,k)∥≤ηm∑k=1Q∥rRm,k∥+ηm∥∑k=1QpRm,k∥≤c2ηm2∑k=1Q∑q=1k-1(∥pRm,q∥+∥pIm,q∥)+ηm∥∑k=1QpRm,k∥≤c2Qηm2∑q=1Q(∥pRm,q∥+∥pIm,q∥)+ηm(∥∑k=1QpRm,k∥+∥∑k=1QpIm,k∥)=ηm(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)+c11ηm2∑q=1Q(∥pRm,q∥+∥pIm,q∥),∥w(m+1)Q,I-wmQ,I∥≤ηm(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)+c11ηm2∑q=1Q(∥pRm,q∥+∥pIm,q∥),
where c11=c2Q. By (4.6) and (4.22)-(4.23) we know that there are positive constants c12 and c13 such that
(4.24)∥pRm+1,k∥≤∥pRm,k∥+c12∥w(m+1)Q,R-wmQ,R∥+c13∥w(m+1)Q,I-wmQ,I∥≤∥pRm,k∥+c14ηm(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)+c15ηm2∑q=1Q(∥pRm,q∥+∥pIm,q∥),
where c14=c12+c13,c15=c11(c12+c13). Taking squares of the two sides of the above inequality gives
(4.25)∥pRm+1,k∥2≤∥pRm,k∥2+c142ηm2(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)2+c152ηm4(∑q=1Q(∥pRm,q∥+∥pIm,q∥))2+2(c14ηm∥pRm,k∥(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)+c15ηm2∥pRm,k∥∑q=1Q(∥pRm,q∥+∥pIm,q∥)+c14c15ηm3(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)∑q=1Q(∥pRm,q∥+∥pIm,q∥))≤∥pRm,k∥2+c142ηm2(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)2+c152ηm4(∑q=1Q(∥pRm,q∥+∥pIm,q∥))2+c14ηm(∥pRm,k∥2+(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)2)+c15ηm2(∥pRm,k∥2+(∑q=1Q(∥pRm,q∥+∥pIm,q∥))2)+c14c15ηm3((∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)2+(∑q=1Q(∥pRm,q∥+∥pIm,q∥))2)=∥pRm,k∥2+(c142ηm+c14+c14c15ηm2)ηm(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)2+(c14+c15ηm)ηm∥pRm,k∥2+(c152ηm3+c15ηm+c14c15ηm2)×ηm(∑q=1Q(∥pRm,q∥+∥pIm,q∥))2.
Now we sum up the above inequality over k=1,…,Q and obtain
(4.26)∑k=1Q∥pRm+1,k∥2≤∑k=1Q∥pRm,k∥2+Q(c142ηm+c14+c14c15ηm2)ηm(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)2+(c14+c15ηm)ηm∑k=1Q∥pRm,k∥2+Q(c152ηm3+c15ηm+c14c15ηm2)ηm(∑q=1Q(∥pRm,q∥+∥pIm,q∥))2≤∑k=1Q∥pRm,k∥2+2Q(c142ηm+c14+c14c15ηm2)ηm(∥∑q=1QpRm,q∥2+∥∑q=1QpIm,q∥2)+(c14+c15ηm)ηm∑k=1Q∥pRm,k∥2+2Q(c152ηm3+c15ηm+c14c15ηm2)ηm((∑q=1Q∥pRm,q∥)2+(∑q=1Q∥pIm,q∥)2)≤∑k=1Q∥pRm,k∥2+2Q(c142ηm+c14+c14c15ηm2)ηm(∥∑q=1QpRm,q∥2+∥∑q=1QpIm,q∥2)+(c14+c15ηm)ηm∑k=1Q(∥pRm,k∥2+∥pIm,k∥2)+2Q2(c152ηm3+c15ηm+c14c15ηm2)ηm(∑q=1Q∥pRm,q∥2+∑q=1Q∥pIm,q∥2).
Let
(4.27)c16=max{2Q(c142ηm+c14+c14c15ηm2),c14+c15ηm+2Q2(c152ηm3+c15ηm+c14c15ηm2)},
then
(4.28)∑k=1Q∥pRm+1,k∥2≤∑k=1Q∥pRm,k∥2+c16ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+c16ηm∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
On the other hand, from (4.22) we have
(4.29)∑k=1QpRm+1,k=∑k=1QpRm,k+∑k=1QμkR′′(t5m,k)((w(m+1)Q,R-wmQ,R)·xk-(w(m+1)Q,I-wmQ,I)·yk)xk+∑k=1QμkI′′(t6m,k)((w(m+1)Q,I-wmQ,I)·xk+(w(m+1)Q,R-wmQ,R)·yk)yk.
Similar to the deduction of (4.24), from (4.29) we have
(4.30)∥∑k=1QpRm+1,k∥≥∥∑k=1QpRm,k∥-Qc12∥w(m+1)Q,R-wmQ,R∥-Qc13∥w(m+1)Q,I-wmQ,I∥≥∥∑k=1QpRm,k∥-Qc14ηm(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)-Qc15ηm2∑q=1Q(∥pRm,q∥+∥pIm,q∥).
It can be easily verified that, for any positive numbers a, b, c, if a≥b-c,
(4.31)a2≥b2-2bc.
Applying (4.31) to (4.30) implies that
(4.32)∥∑k=1QpRm+1,k∥2≥∥∑k=1QpRm,k∥2-2∥∑k=1QpRm,k∥×(Qc14ηm(∥∑q=1QpRm,q∥+∥∑q=1QpIm,q∥)+Qc15ηm2∑q=1Q(∥pRm,q∥+∥pIm,q∥))≥∥∑k=1QpRm,k∥2-2Qc14ηm∥∑k=1QpRm,k∥2-2Qc14ηm∥∑k=1QpRm,k∥∥∑q=1QpIm,q∥-2Qc15ηm2(∑k=1Q∥pRm,k∥)2-2Qc15ηm2(∑k=1Q∥pRm,k∥)(∑q=1Q∥pIm,q∥)≥∥∑k=1QpRm,k∥2-2Qc14ηm∥∑k=1QpRm,k∥2-Qc14ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-2Qc15ηm2(∑k=1Q∥pRm,k∥)2-Qc15ηm2((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2).
Similarly, we can obtain the counterpart of (4.28) as
(4.33)∑k=1Q∥pIm+1,k∥2≤∑k=1Q∥pIm,k∥2+c16ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+c16ηm∑k=1Q(∥pRm,k∥2+∥pIm,k∥2),
and the counterpart of (4.32) as
(4.34)∥∑k=1QpIm+1,k∥2≥∥∑k=1QpIm,k∥2-2Qc14ηm∥∑k=1QpIm,k∥2-Qc14ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-2Qc15ηm2(∑k=1Q∥pIm,k∥)2-Qc15ηm2((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2).
From (4.28) and (4.33) we have
(4.35)∑k=1Q∥pRm+1,k∥2+∑k=1Q∥pIm+1,k∥2≤∑k=1Q∥pRm,k∥2+∑k=1Q∥pIm,k∥2+2c16ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+2c16ηm∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
From (4.32) and (4.34) we have
(4.36)∥∑k=1QpRm+1,k∥2+∥∑k=1QpIm+1,k∥2≥∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2-4Qc14ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4Qc15ηm2((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2).
Using (2.11) and (4.36), we can get
(4.37)1ηm+1(∥∑k=1QpRm+1,k∥2+∥∑k=1QpIm+1,k∥2)≥1ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+N(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4Qc14(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4NQc14ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4Qc15ηm((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2)-4NQc15ηm2((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2).
Multiplying (4.37) with ηm+12 gives
(4.38)ηm+1(∥∑k=1QpRm+1,k∥2+∥∑k=1QpIm+1,k∥2)≥ηm+12ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+Nηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4Qc14ηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4NQc14ηmηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4Qc15ηmηm+12((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2)-4NQc15ηm2ηm+12((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2).
Using (4.20) and (4.35), we obtain
(4.39)c5ηm+12(∑k=1Q∥pRm+1,k∥2+∑k=1Q∥pIm+1,k∥2)≤ηm+12ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+2c5c16ηmηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+2c5c16ηmηm+12∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
Combining (4.38) and (4.39) we have
(4.40)ηm+1(∥∑k=1QpRm+1,k∥2+∥∑k=1QpIm+1,k∥2)≥c5ηm+12(∑k=1Q∥pRm+1,k∥2+∑k=1Q∥pIm+1,k∥2)+Nηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4Qc14ηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4NQc14ηmηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-4Qc15ηmηm+12((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2)-4NQc15ηm2ηm+12((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2)-2c5c16ηmηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-2c5c16ηmηm+12∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
Thus to validate (4.21) we only need to prove the following inequality:
(4.41)Nηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)≥4Qc14ηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+4NQc14ηmηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+4Qc15ηmηm+12((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2)+4NQc15ηm2ηm+12((∑k=1Q∥pRm,k∥)2+(∑k=1Q∥pIm,k∥)2)+2c5c16ηmηm+12(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)+2c5c16ηmηm+12∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
Recalling (4.20) and  ηm≤η0(m≥0), it is easy to see that (N/6)ηm+12(∥∑k=1QpRm+1,k∥2+∥∑k=1QpIm+1,k∥2)≥ each term of (4.41) can be assured for N≥Ñ and 0≤η0≤min{1,η̃} by setting
(4.42)Ñ=max{24Qc14,24Qc15c5,12c16},η̃=min{23Qc14,2c163c15,c5c16Ñ12,∥∑k=1QpR0,k∥2+∥∑k=1QpI0,k∥2c5∑k=1Q∥pR0,k∥2+∥pI0,k∥2}.
This, thus, validates (4.41). As a result, (4.18) and (3.7) are proved.Proof of Lemma3.5.
From Lemma3.3 we have
(4.43)E(wmQ)-E(w(m+1)Q)≥ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-c5ηm2∑k=1Q(∥pRm,k∥2+∥pIm,k∥2).
Sum the above inequalities up over m=1,…,M, then
(4.44)E(wQ)-E(w(M+1)Q)≥∑m=1M(ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-c5ηm2∑k=1Q(∥pRm,k∥2+∥pIm,k∥2)).
Note that E(w(M+1)Q)≥0 for M≥0. Setting M→∞, we have
(4.45)∑m=1∞(ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)-c5ηm2∑k=1Q(∥pRm,k∥2+∥pIm,k∥2))≤E(wQ)<∞.
Using (2.9) and (4.6), we can find a constant c17 such that
(4.46)∑k=1Q(∥pRm,k∥2+∥pIm,k∥2)≤c17.
This together with (2.13) leads to
(4.47)∑m=1∞(c5ηm2∑k=1Q(∥pRm,k∥2+∥pIm,k∥2))≤∑m=1∞c5c17N21m2<∞.
Thus from (4.45) and (4.47) it holds that
(4.48)∑m=1∞ηm(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)<∞.
Recalling ηm=O(1/m) from (2.12) gives
(4.49)∑m=1∞1m(∥∑k=1QpRm,k∥2+∥∑k=1QpIm,k∥2)<∞.Proof of Lemma3.6.
This lemma is the same as Lemma2.1 of [16].Proof of Lemma3.7.
This result is almost the same as Theorem14.1.5 in [18], and the details of the proof are omitted.Proof of Theorem3.8.
Using (2.9), (4.6), (4.14), and (4.15), we can find a constant c18 such that
(4.50)∥w(m+1)Q,R-wmQ,R∥≤c18ηm,∥w(m+1)Q,I-wmQ,I∥≤c18ηm.
From (2.6) and (4.22) we have
(4.51)∂E(w(m+1)Q)∂wR-∂E(wmQ)∂wR=∑k=1Q([μkR′(U(m+1)Q,k,R)-μkR′(UmQ,k,R)]xk+[μkI′(U(m+1)Q,k,I)-μkI′(UmQ,k,I)]yk)=∑k=1Q(μkR′′(t5m,k)((w(m+1)Q,R-wmQ,R)·xk-(w(m+1)Q,I-wmQ,I)·yk)xk+μkI′′(t6m,k)((w(m+1)Q,I-wmQ,I)·xk+(w(m+1)Q,R-wmQ,R)·yk)yk),
where t5m,k and t6m,k are defined in (4.22). Thus, from (4.6), (4.50), and Cauchy-Schwartz inequality, there exists a constant c19 such that for any vector e∈ℝL(4.52)||∂E(w(m+1)Q)∂wR·e|-|∂E(wmQ)∂wR·e||≤|∂E(w(m+1)Q)∂wR·e-∂E(wmQ)∂wR·e|≤c19∥e∥(∥w(m+1)Q,R-wmQ,R∥+∥w(m+1)Q,I-wmQ,I∥)≤2c18c19∥e∥ηm≤2c18c19∥e∥N1m.
Using (2.6), (2.9), and Lemma 3.5, we have
(4.53)∑m=1∞(1m|∂E(wmQ)∂wR·e|2)=∑m=1∞(1m|(∑k=1QpRm,k)·e|2)≤∥e∥2∑m=1∞(1m∥∑k=1QpRm,k∥2)<∞.
From (4.52), (4.53), and Lemma 3.6 it holds that
(4.54)limm→∞|∂E(wmQ)∂wR·e|=0.
Since e is arbitrary in ℝL, we have
(4.55)limm→∞∂E(wmQ)∂wR=0.
Therefore, when q=0, we complete the proof of limm→∞(∂E(wmQ+q)/∂wR)=0, and we can similarly show that limm→∞(∂E(wmQ+q)/∂wR)=0 for q=1,…,Q. Thus, we have shown that
(4.56)limn→∞∂E(wn)∂wR=0.
In a similar way, we can also prove that
(4.57)limn→∞∂E(wn)∂wI=0.
Thus, (3.13) is obtained from (4.56) and (4.57).
Next we begin to prove (3.14). Using (2.10), we have
(4.58)∥ΔwmQ+q,R∥=ηm∥pRm,q,q∥.
Similar to (4.46), we know that ∥pRm,q,q∥ is bounded. Recalling (2.13) makes us conclude that
(4.59)limm→∞∥wmQ+q,R-wmQ+q-1,R∥=limm→∞∥ΔwmQ+q,R∥=limm→∞ηm∥pRm,q,q∥=0,q=1,…,Q,
which implies that
(4.60)limn→∞∥wn,R-wn-1,R∥=0.
Similarly, we have
(4.61)limn→∞∥wn,I-wn-1,I∥=0.
Write
(4.62)θ=((wR)T,(wI)T)T,θn=((wn,R)T,(wn,I)T)T,
then the square error function E(w) can be looked as a real-valued function ℰ(θ). Thus from (4.56), (4.57), (4.60), and (4.61) we have
(4.63)limn→∞∂ℰ(θn)∂θ=0,limn→∞∥θn-θn-1∥=0.
Furthermore, from Assumption (A2) we know that the set {θ∣∂ℰ(θ)/∂(θ)=0} contains only finite points. Thus, the sequence {θn}n=1∞ here satisfies all the conditions needed in Lemma 3.7. As a result, there is a θ⋆ which satisfies that limn→∞θn=θ⋆. Since θn consists of the real and imaginary parts of wn, we know that there is a w⋆ such that limn→∞wn=w⋆. We, thus, complete the proof of (3.14).


## 5. Numerical Example

In this section we illustrate the convergence behavior of the OSCG algorithm by using a simple numerical example. The well-known XOR problem is a benchmark in literature of neural networks. As in [13], the training samples of the encoded XOR problem for CVNN are presented as follows:(5.1){z1=-1-i,d1=1},{z2=-1+i,d2=0},{z3=1-i,d3=1+i},{z4=1+i,d4=i}.This example uses a network with two input nodes (including a bias node) and one output node. The transfer function is tansig(·) in MATLAB, which is a commonly used sigmoid function. The parameter η0 is set to be 0.1 and N is set to be 1. We carry out the test with the initial components of the weights stochastically chosen in [-0.5,0.5]. Figure 1 shows that the gradients tend to zero and the square error decreases monotonically as the number of iteration increases and at last tends to a constant. This supports our theoretical analysis.Figure 1
Convergence behavior of OSCG algorithm for solving XOR problem (sum of gradient norms=∥∂E(wn)/∂wR∥+∥∂E(wn)/∂wI∥).


## 6. Conclusion

In this paper we investigate some convergence properties of an OSCG training algorithm for two-layered CVNN. We choose an adaptive learning rate in the algorithm. Under the condition that the activation function and its up to the second-order derivative are bounded, it is proved that the error function is monotonely decreasing during the training process. With this result, we further prove that the gradient of the error function tends to zero and the weight sequence tends to a fixed point. A numerical example is given to support our theoretical analysis. We mention that those results are interestingly similar to the convergence results of batch split-complex gradient training algorithm for CVNN given in [17]. Thus our results can also be a theoretical explanation for the relationship between the OSCG algorithm and the batch split-complex algorithm. The convergence results in this paper can be generalized to a more general case, that is, multilayer CVNN.



---

*Source: 829692-2010-03-21.xml*