# Without Diagonal Nonlinear Requirements: The More GeneralP-Critical Dynamical Analysis for UPPAM Recurrent Neural Networks

Authors: Xi Chen; Huizhong Mao; Chen Qiao
Journal: Mathematical Problems in Engineering
 (2013)

---

## Abstract

Continuous-time recurrent neural networks (RNNs) play an important part in practical applications. Recently, due to the ability of assuring the convergence of the equilibriums on the boundary line between stable and unstable, the study on the critical dynamics behaviors of RNNs has drawn especial attentions. In this paper, a new asymptotical stable theorem and two corollaries are presented for the unified RNNs, that is, the UPPAM RNNs. The analysis results given in this paper are under the generallyP-critical conditions, which improve substantially upon the existing relevant critical convergence and stability results, and most important, the compulsory requirement of diagonally nonlinear activation mapping in most recent researches is removed. As a result, the theory in this paper can be applied more generally.


---

## Body

## 1. Introduction

Neural networks have been rapidly developing into an important technology for about 30 years, and they can be grouped into two main types according to the networks’ structure, that is, feed-forward neural networks (FNNs) and recurrent neural networks (RNNs). RNNs are the neural networks with feedback loops and by which the output of each neuron in the monolayer would feedback to the input of other neurons. RNNs are the dynamic systems that their states will vary with time pasting by. The crucial foundation of the RNNs consists in their dynamical properties, such as the global convergence, asymptotic stability, and exponential stability. Therefore, the analysis of such dynamical behaviors is the first and necessary step for any practical design and application of RNNs, such as recognition classification, adaptive control, and optimization.In recent years, for different model individuals, considerable efforts have been devoted to the analysis on the stability of RNNs without and with delay (see, e.g., [1–18] and the references therein). In order to generalize those results, the authors of [19] point out that most of the exponential stability results are given under the condition that one discriminate matrix defined by the RNNs, which is denoted by M(L,Γ), is positive. Here, M(L,Γ)=L-1Γ-(ΓW+WTΓ)/2, and L=diag{l1,l2,…,lN} with each li>0 being the Lipschitz constant of gi, G(x)=(g1(x),g2(x),…,gN(x))T, x∈ℝN is the activation operator of the network, Γ is an arbitrary positive define diagonal matrix, and W is the weight matrix of the network. When another discriminate matrix which has the similar form as M(L,Γ) is negative, the RNNs are of exponential unstability. Further, they present the concept of the special critical condition and some special critical convergence analysis for two kinds of RNN models. In [20, 21], the general critical condition is defined. From which, we can see that the critical condition is really an essential gap between stable and unstable for the given RNNs; that is, on one side of this gap, the RNNs are sure to be stable and on the other side, the RNNs are certainly unstable. And, when the RNNs fall into this gap, there exist stable trajectories as well as unstable ones. Studying the dynamics behaviors of RNNs under the critical condition is called as the critical analysis. The goal of the critical analysis is to find the least restrictions to assure the most stability of RNNs, which is usually corresponding to the design of the connection matrices (which is contained in the critical condition). To extend the application fields, especially to loosen the design of the connection matrices, it is obvious quite important to study the critical dynamics of RNNs. Since studying the general critical dynamics is quite different, for studying conveniently, the general critical analysis is often replaced by the P-critical study.On the other hand, in order to discover some more in-depth characters other than the bound and monotonically nondecreasing properties of the common used activation operators, [22] provides two novel concepts: the uniformly pseudoprojection antimonotone (UPPAM) operator and the UPPAM RNNs. The UPPAM operator can formalize most of the activation operators, and the UPPAM RNNs can be the representation of most of the RNNs individuals too. Thus, the UPPAM RNNs are called as the unified RNNs. Further, it is guessed that only the study of the dynamics of the UPPAM RNNs may achieve the outcome that we can discriminate the similarity and redundant of the dynamics results among those known RNNs individuals, and which is affirmed following in [22, 23] for discrete-time RNNs as well as for continuous-time RNNs, respectively.For all that, we should notice one more important thing: no matter for all kinds of the RNNs individuals or for the UPPAM RNNs, almost all of the dynamics conclusions (including those latest critical results) are based on a hypothesis that the activation operators are diagonally nonlinear; that is, the nonlinear activation operatorG:ℝN→ℝN is defined by G(x)=(g1(x1),g2(x2),…,gN(xN))T (here each gi is a one-dimensional nonlinear function). The cause of the diagonally nonlinear requirement in dynamics analysis is due to the fact that when considering the derivative of a constructed energy function, an inner product is always produced and it is hard to deal with it if one do not use the diagonally nonlinear property. It is obvious that this requirement for the activation operator is quite strict and goes against the biological basis as well as applications.In the current paper, we are devoted to remove the diagonally nonlinear requirement of the activation operators in dynamics analysis and answer the question that for the unified recurrent neural networks (i.e., the UPPAM RNNs), what dynamics behavior will happen under theP-critical condition (which is defined as M(L,Γ)+P≥0 with P being an arbitrary nonnegative matrix). By using the uniformly antimonotonicity as well as the pseudoprojection property and by combining the Lyapunov functional method and the LaSalle invariance principle, we get the global convergence and asymptotic stability theorem and some corollaries, which improve the recent dynamics results in three aspects. Firstly, the theorem and corollaries obtained in this paper are for the general RNNs, so they can be applied directly to CNN, BSB, BCOp-type, and other common used specific models. Secondly, the P-critical analysis is more general than the special critical analysis, let alone those noncritical analyses. Thirdly, and most importantly, the requirement of the diagonal nonlinear for activation operator, which is the restrictive condition for activation operator, is removed.


## 2. Basic Definition

Static RNNs and local field RNNs typically represent two fundamental modeling approaches in current neural network research, which are, respectively, modeled by(1)τdxdt=-x+G(Wx+q),x(0)=x0∈ℝN,(2)τdydt=-y+WG(y)+q,y(0)=y0∈ℝN,
where x=(x1,x2,…,xN)T is the neural state vector, y=(y1,y2,…,yN)T is the local field vector, W=(ωij)N×N is the synaptic weight matrix, τ is a positive constant, q is a fixed external bias vector, and G:ℝN→ℝN is the nonlinear activation operator.We now recall some notion and notations (taking system (1) as an example). A constant vector x* is said to be an equilibrium state of system (1), if x* is a fixed point of the operator T(x):=G(Wx+q), for all x∈ℝN. x* is said to be stable if any trajectory of system (1) can stay within a small neighborhood of x* whenever the initial state x0 is close to x*, and it is said to be attractive if there is a neighborhood Δ(x*), called the attraction basin of x*, such that any trajectory of system (1) initialized from a state in Δ(x*) will approach to x* as time goes to infinity. x* is said to be globally asymptotically stable onΔ(x*) if it is both stable and attractive, with the attraction basin Δ(x*). System (1) is said to be globally convergent onΘ if for every initial point x0∈Θ, x(t,x0) converges to an equilibrium state of system (1) (the limit of x(t,x0) may not be the same for different x0).Here we give some definitions about the activation operatorG:ℝN→ℝN and the nonlinear operator T. Denote the range of G by R(G).Definition 1 (see [22]).
The nonlinear activation operatorG:ℝN→ℝN is said to be an α-uniformly antimonotonous operator if there is a constant positive number α such that
(3)〈G(x)-G(y),x-y〉≥α∥G(x)-G(y)∥2,∀x,y∈ℝN.In [22], it is shown that most of the common nonlinear activation operators have the uniformly antimonotonous properties, such as the nearest point projection operator, the linear saturating operator, the signum operator, and the pseudoprojection operator.Definition 2 (see [22]).
An operatorG is said to be a pseudoprojection if there exists a positive define diagonal matrix B=diag{β1,β2,…,βN} such that BR(G)∈ℝN and G(x)=GBG(x), for all x∈ℝN. Then G is called as B-projection.Obviously, all the projection operators are pseudoprojection operators (hereB=I, and I refers to the identity matrix), and most of the concrete activation operators diversely appeared in RNNs are pseudoprojection operators.In [22], when the operator G has both the pseudoprojection and uniformly antimonotonous properties, it is called as uniformly pseudoprojection, antimonotonous (UPPAM) operator. Specially, we say G is a (B,α)-UPPAM whenever it is a B-projection and α-uniformly antimonotonous operator. It is worthwhile to note that the UPPAM operator provides a very appropriate, unified framework within which most of the known RNN models can be embedded and uniformly studied [21–23].The following definition of the nonlinear norm is similar to that of the matrix norm.Definition 3 (see [20]).
Suppose thatT:Ψ⊆ℝN→Y⊆ℝN is a nonlinear operator, A is a nonsingular N×N matrix, and x~∈Ψ is a given vector. One calls L as the nonlinear norm if it is defined as follows:
(4)L∥·∥(T,A,x~,Ψ)=supx≠x~,x∈Ψ∥ATx-ATx~∥∥Ax-Ax~∥.Obviously, for any given matrixB, L∥·∥(B,I,x~,Ψ)=∥B∥. Additionally, for any constant β>0, one has L∥·∥(βT,A,x~,Ψ)=βL∥·∥(T,A,x~,Ψ).Definition 4.
One takesl as the minimum Lipschitz constant of operator F:ℝn→ℝ, which is defined as follows:
(5)l=supt,s∈ℝn,t≠s|F(t)-F(s)|∥t-s∥2,∀x,y∈ℝn.Without loss of generality, throughout this paper, we assume that eachl>0. Here, let li be the minimum Lipschitz constant of gi and ℒ=diag{l1,l2,…,lN}. The matrix ℒ is said to be the minimum Lipschitz matrix of operator G=(g1,g2,…,gN)T.


## 3. The Global Convergence Theorems of RNNs

In this section, the global convergence and asymptotic stability theorem and corollaries for RNNs with UPPAM operators of both systems (1) and (2) will be established under the P-critical condition. We consider the networks of form (1) first.Suppose thatG:ℝN→ℝN is the nonlinear activation operator. For any v∈R(G), define T(v)=G(Wv+q) and Fix(T) as being the fixed point set of T(v). Then by Brouwer’s fixed point theorem, T has at least one fixed point v*∈Fix(T). As a result, the equilibrium state set of (1) is not empty.Since for anyN×N positive definite matrix A, there exists an orthogonal matrix K (KTK=I) such that KTAK=diag{λ1,λ2,…,λN} (here λi>0 is the eigenvalue of A), and if we define B=Kdiag{λ11/2,λ21/2,…,λN1/2}KT, then it is clear that B=BT, B2=A and B is invertible. Such a matrix B is denoted by Deco(A); that is, B=Deco(A).Following is the global convergence and asymptotic stability theorem of system (1). Suppose that Θ=R(G) is a bounded, closed, and convex subset of ℝN.Theorem 5.
LetG:ℝN→Θ be a (B,α)-UPPAM operator, and let each gi be monotonically increasing and continuous. If there is a nonnegative definite matrix P, such that M(ℒ,I)+P≥0, and for a v*∈Fix(T), L∥·∥2(T,Q,v*,Θ)≤1 where Q=Deco(ℒ-1+P), then RNN system (1) is globally convergent on Θ when Fix(T) is disconnected. Furthermore, when v* is the unique equilibrium point of (1), then v* is globally asymptotically stable on Θ.Proof.
Denote the trajectory starting fromx0∈Θ by x(t), Let y(t)=Wx(t)+q, z(t)=G(y(t)) and u(t)=z(t)-x(t). It is obvious that u(t)=τdx(t)/dt.
Define(6)V1(x(t))=-τ·xT(t)(W+WT)x(t),V2(x(t))=τ·xT(t)(ℒ-1+B2)x(t),V3(x(t))=-2τ·xT(t)q,V4(x(t))=τ∑i=1N‍∫(y0)iyi(t)‍gi(s)ds,V5(x(t))=12τ{-(y(t)-x(t))T(y(t)-x(t))+y(t)Ty(t)+x(t)T(I-ℒ-1)x(t)(y(t)-x(t))T},V6(x(t))=τxT(t)(ℒ-1+P)x(t)-2xT(t)(ℒ-1+P)v*.
It is easy to get their derivatives, that is,
(7)V˙1(x(t))=-2〈(W+WT)x(t),u(t)〉,V˙2(x(t))=〈(ℒ-1+B)x(t),u(t)〉,V˙3(x(t))=-2〈q,u(t)〉,V˙4(x(t))=〈z(t),Wu(t)〉,V˙5(x(t))=-〈x(t),(ℒ-1-W)u(t)〉+〈y(t),u(t)〉,V˙6(x(t))=2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉.
Define(8)E(x(t))=∑i=16‍Vi(x(t))
as the Lyapunov energy function, and a direct calculation is that
(9)dE(x(t))dt=-2〈(W+WT)x(t),u(t)〉+〈(ℒ-1+B)x(t),u(t)〉-〈q,u(t)〉+〈z(t),Wu(t)〉-〈x(t),(ℒ-1-W)u(t)〉+〈y(t),u(t)〉+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=〈((ℒ-1+B)-2(W+WT))x(t),u(t)〉-2〈q,u(t)〉+〈ℒ-1z(t),u(t)〉-〈z(t),(ℒ-1-W)u(t)〉-〈x(t),(ℒ-1-W)u(t)〉+〈y(t),u(t)〉+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=-2〈y(t),u(t)〉+〈Bx(t),u(t)〉+〈ℒ-1x(t),u(t)〉-2〈x(t),Wu(t)〉+〈ℒ-1z(t),u(t)〉-〈z(t),(ℒ-1-W)u(t)〉-〈x(t),(ℒ-1-W)u(t)〉+〈y(t),u(t)〉+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=-〈y(t)-Bx(t),u(t)〉-〈ℒ-1x(t),u(t)〉+〈x(t)-z(t),(ℒ-1-W)u(t)〉+〈ℒ-1z(t),u(t)〉+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=-〈y(t)-Bx(t),u(t)〉+〈x(t)-z(t),W(x(t)-z(t))〉+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=-〈y(t)-Bx(t),u(t)〉+〈z(t)-x(t),(W+WT2)(z(t)-x(t))〉+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=-〈y(t)-Bx(t),u(t)〉-uT(t)((ℒ-1+P)-W+WT2)u(t)+uT(t)(ℒ-1+P)u(t)+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉.
Since ℒ-1+P-(W+WT)/2=M(ℒ,I)+P≥0, then
(10)-uT(t)((ℒ-1+P)-W+WT2)u(t)≤0.
Thus
(11)dE(x(t))dt≤-〈(y(t)-Bx(t)),u(t)〉+uT(t)(ℒ-1+P)u(t)+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉.
Let Q=Deco(ℒ-1+P), then we have
(12)uT(t)(ℒ-1+P)u(t)+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=〈u(t),Q2u(t)〉+2〈Q2x(t),u(t)〉-2〈Q2v*,u(t)〉=〈u(t)+2x(t)-2v*,Q2u(t)〉=〈z(t)+x(t)-2v*,Q2u(t)〉=〈((z(t)-v*)-(x(t)-v*))(z(t)-v*)+(x(t)-v*),Q2((z(t)-v*)-(x(t)-v*))〉=〈z(t)-v*,Q2(z(t)-v*)〉-〈x(t)-v*,Q2(x(t)-v*)〉.
From v*∈Fix(T), we know v*=G(Wv*+q). And then (12) is equal to
(13)uT(t)(ℒ-1+P)u(t)+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉=〈z(t)-G(Wv*+q),Q2(z(t)-G(Wv*+q))〉-〈x(t)-v*,Q2(x(t)-v*)〉=〈(G(Wx(t)+q)-G(Wv*+q))Q2G(Wx(t)+q)-G(Wv*+q),Q2(G(Wx(t)+q)-G(Wv*+q))〉-〈x(t)-v*,Q2(x(t)-v*)〉=〈Q(G(Wx(t)+q)-G(Wv*+q)),Q(G(Wx(t)+q)-G(Wv*+q))〉-〈Q(x(t)-v*),Q(x(t)-v*)〉=∥Q(G(Wx(t)+q)-G(Wv*+q))∥2-∥Q(x(t)-v*)∥2.
Furthermore, on noting that L∥·∥2(T,Q,v*,Θ)≤1, one can get that
(14)L∥·∥2(T,Q,v*,Θ)=supx≠v*,x∈Θ∥QTx-QTv*∥2∥Qx-Qv*∥2=supx≠v*,x∈Θ∥QG(Wx+q)-QG(Wv*+q)∥2∥Qx-Qv*∥2≤1.
This, combined with (11) and (13), implies that
(15)uT(t)(ℒ-1+P)u(t)+2〈(ℒ-1+P)x(t),u(t)〉-2〈(ℒ-1+P)v*,u(t)〉≤0,(16)dE(x(t))dt≤〈y(t)-Bx(t),u(t)〉.
In addition, since x(t)∈Θ=R(G) and G is a (B,α)-UPPAM operator, then x(t)=GB(x(t)) and
(17)〈y(t)-Bx(t),u(t)〉=〈y(t)-Bx(t),z(t)-x(t)〉=〈y(t)-Bx(t),G(y(t))-x(t)〉=〈y(t)-Bx(t),G(y(t))-G(Bx(t))〉≥α∥G(y(t))-G(Bx(t))∥2.
Combining with (16), we have
(18)dE(x(t))dt≤0.
The equal sign holds if and only if G(y(t))=G(Bx(t)); that is, z(t)-x(t)=0.
Furthermore, sincex(t)∈Θ is bounded and Fix(T) is disconnected, then by LaSalle invariance principle in [24], we know that RNN model (1) is globally convergent on Θ. And when Fix(T)={v*}, then it is easy to deduce that v* is both attractive and stable on Θ since Θ is bounded; that is, v* is globally asymptotically stable on Θ. Thus, Theorem 5 is proved.Theorem5 gives the global convergence and asymptotic stability result of UPPAM RNNs without diagonal nonlinear requirement under the P-critical condition, while it is not quite easy to judge the condition that L∥·∥2(T,Q,v*,Θ)≤1. To improve it, we present the following corollary.Corollary 6.
Assume thatG:ℝN→Θ is a (B,α)-UPPAM operator with each gi being monotonically increasing and continuous. If there exists a nonnegative diagonal matrix P, such that M(ℒ,I)+P≥0 and ∥Q∥2·∥WQ-1∥2≤α (here Q=Deco(ℒ-1+P)), then RNN model (1) is globally convergent on Θ when Fix(T) is disconnected. Moreover, when v* is the unique equilibrium point of (1), then v* is globally asymptotically stable on Θ.Proof.
Assume thatv*∈Fix(T); then
(19)∥QTx-QTv*∥2=∥QG(Wx+q)-QG(Wv*+q)∥2=∥Q(G(Wx+q)-G(Wv*+q))∥2≤∥Q∥2·∥G(Wx+q)-G(Wv*+q)∥2.
Since G is α-uniformly antimonotonous, we know that for any x,y∈ℝN,
(20)〈x-y,G(x)-G(y)〉=∥x-y∥·∥G(x)-G(y)∥·cosθ≥α·∥G(x)-G(y)∥2.
So it is clear that
(21)α·∥G(x)-G(y)∥≤∥x-y∥,∀x≠y.
Then, we have
(22)∥G(Wx+q)-G(Wv*+q)∥2≤1α∥(Wx+q)-(Wv*+q)∥2=1α∥W(x-v*)∥2.
And (19) can conclude that
(23)∥QTx-QTv*∥2≤1α∥Q∥2·∥W(x-v*)∥2=1α∥Q∥2·∥WQ-1Q(x-v*)∥2≤1α∥Q∥2·∥WQ-1∥2·∥Q(x-v*)∥2.
Clearly, when∥Q∥2·∥WQ-1∥2≤α, then it can be deduced that
(24)L∥·∥2(T,Q,v*,Θ)=supx≠v*,x∈Θ∥QTx-QTv*∥2∥Qx-Qv*∥2≤1.
Corollary 6 is then proved from Theorem 5.Correspondingly, we can deduce the critical global convergence and asymptotical stability conclusions for RNN system (2).Corollary 7.
Assume thatG:ℝN→Θ is a (B,α)-UPPAM operator with each gi being monotonically increasing and continuous. If there exists a nonnegative diagonal matrix P such that M(ℒ,I)+P≥0 and one of the following conditions holds, then RNN model (2) is globally convergent on (WΘ+q) when Fix(T) is disconnected. Moreover, when y* is the unique equilibrium point of (2), then y* is globally asymptotically stable on (WΘ+q).(i)
For av*∈Fix(T), the nonlinear norm L∥·∥2(T,Q,v*,Θ)≤1, where Q=Deco(ℒ-1+P);(ii)
For the weight matrixW, ∥Q∥2·∥WQ-1∥2≤α, where Q=Deco(ℒ-1+P).Proof.
For any trajectoryy(t) of (2) starting from y0∈W(Θ)+q, let y0=Wx0+q with x0∈Θ and let x(t) be the solution of (1) with initial value x(0)=x0; then by the uniqueness of solution of differential equations, it is easy to verify that y(t)≡Wx(t)+q and there exists an equilibrium state of (1), denoted by v*, such that y*=Wv*+q. Thus, by [25], the convergence of y(t) to an equilibrium state of (2) can be shown by studying the asymptotic behavior of x(t). Then, the conclusion of Corollary 7 readily follows from Theorem 5 and Corollary 6.Remark 8.
In this section, we have presented theP-critical convergence as well as the stability results for the uniformly pseudoprojection antimonotone RNNs. In detail, we obtain the global convergence and the asymptotic stability for the static UPPAM RNNs under the conditions that either the nonlinear norm defined by the networks is less than 1, or the matrix norms given by the networks satisfy one bounded requirement. And the corresponding results for the local field UPPAM RNNs are discussed.
It should be noticed that in the achieved theorem as well as the two corollaries here, the diagonally nonlinear requirement of the activation operators has been removed directly, which is a basic hypothesis in nearly all of the dynamics analysis obtained before. Thus, we improve most of the existing results for RNNs, either for those ones under the noncritical conditions or for those under the critical conditions (see, e.g., [6, 13, 19, 21, 25–28] and the references therein). In addition of this, we know from [20] that the P-critical analysis of RNNs can give the basic argument between stability and unstability of RNNs, so the discussion of the P-critical dynamics analysis for RNNs without diagonally nonlinear requirement is quite meaningful both in theory and in applications. Further, since the UPPAM RNNs can formalize most of the existing RNNs individuals, thus the analysis results of dynamics behaviors for UPPAM RNNs may achieve the unified conclusions for RNNs, and which can discriminate the similarity and redundant of the dynamics results among the known RNNs individuals. In particular, the achieved results here can be applied directly to many RNNs models and can improve deeply the main results of those models, for example, the Cellular Neural Networks (CNNs) [7–11], the Brain-State-in-Box Neural Networks (BSB NNs) [29, 30], the BCOp-type RNNs [31], and other commonly used specific individuals.


## 4. Illustrative Examples

In this section, we provide several illustrative examples to demonstrate the validity of the convergence and stability results formulated in the previous section.Example 1.
Consider the following RNN:(25)dx1(t)dt=-x1(t)+g1(0.5x1(t)+1.5x2(t)+3),dx2(t)dt=-x2(t)+g2(-0.5x1(t)+0.5x2(t)+1.5x3(t)+4),dx3(t)dt=-x3(t)+g3(-0.5x2(t)+0.5x3(t)+1.5x4(t)-4),dx4(t)dt=-x4(t)+g4(-0.5x3(t)+0.5x4(t)),
where each gi(s)={1,s>1s,-1<s≤1-1,s≤-1 (i=1,2,3,4).In this example, it is easy to find that the activation operatorG:ℝ4→Θ=[-1,1]4 and that the minimum Lipschitz constant of gi=1, (i=1,2,3,4). So we get that ℒ=I. In addition, the unique equilibrium state is x*=(1,1,-1,1)T.For any positive diagonal matrixΓ, it is easy to verify that M(ℒ,Γ)=Γ-(ΓW+WTΓ)/2 is not positive and, further, not nonnegative. That is, all of the noncritical and critical conclusions in the literature (see, e.g., [19, 27, 32]) cannot be used here. But Theorem 5 can be applied to this example. Actually, the projection operator G is a (I,1)-UPPAM operator. Letting P=diag{3,3,3,3}, we have
(26)M(ℒ,I)+P=(0.5110.5110.5110.5)+P=(3.5113.5113.5113.5).It is obvious thatM(ℒ,I)+P>0 and Q=Deco(ℒ-1+P)=diag{2,2,2,2}. Letting T(v)=G(Wv+q) for any v∈Θ, one can get that Fix(T)={(1,1,-1,1)T}. For any v*∈Fix(T), we want to show that
(27)L∥·∥2(T,Q,v*,Θ)=supx≠v*,x∈Θ∥QG(Wx+q)-Qv*∥2∥Qx-Qv*∥2≤1.
On noting that for any x=(x1,x2,X3,x4)T∈Θ,
(28)Wx+q=(0.5x1+1.5x2+3-0.5x1+0.5x2+1.5x3+4-0.5x2+0.5x3+1.5x4-4-0.5x3+0.5x4)=x¯.
And -1≤xi≤1 (i=1,2,3,4), so 0.5x1+1.5x2+3≥1, -0.5x1+0.5x2+1.5x3+4≥1,  -0.5x2+0.5x3+1.5x4-4≤-1; then (G(x¯))1=(G(x¯))2=1, (G(x¯))3=-1. Then,
(29)∥QG(Wx+q)-Qv*∥22=∥Q(x¯-v*)∥22=|q4(x¯4-v4*)|2=4(-0.5x3+0.5x4-1)2.
On the other hand,(30)∥Qx-Qv*∥22=∑i=14‍(qi(xi-vi*))2≥4((x3+1)2+(x4-1)2).
From (29) and (30), it shows that
(31)∥Qx-Qv*∥22-∥QG(Wx+q)-Qv*∥22≥4((x3+1)2+(x4-1)2-(-0.5x3+0.5x4-1)2)=4x32+8x3+4+4x42-8x4+4-x32-x42-4+2x3x4-4x3+4x4=(x3+x4)2+2x32+2x42+4x3-4x4+4=(x3+x4)2+2(x3+1)2+2(x4-1)2≥0.
So the inequality
(32)∥QG(Wx+q)-Qv*∥2∥Qx-Qv*∥2≤1
always holds.According to Theorem5, system (25) is globally asymptotically stable on Θ. The following Figure 1 depicts the time responses of state variables of the system with random initial point starting from Θ, which confirm that the proposed condition in Theorem 5 ensures the globally asymptotical stability of the RNNs.Figure 1
Transient behaviors of RNN in system (25) with random initial point x0∈[-1,1]4.Example 2.
Consider the following RNN of system (2):
(33)dydt=-y+WG(y)+q,
where y∈ℝ3, G(y)=argminz∈Θ∥y-z∥2 and Θ={v∈ℝ3:∥v∥2≤1}. The weight matrix and the external bias vector are defined as follows:
(34)W=(0.7-0.30.10.10.8-0.20.1-0.20.5),q=(-0.2-0.30.3).Obviously, this example is established on a general projection operator, and all the diagonally nonlinear conclusions in the literature [20, 28, 32] cannot be used here. But the Corollary 7 established in Section 3 can be applied to Example 2. Actually, in this example B=I and α=1. And it could be proofed that ℒ=I, that is because when Θ is the unit sphere in ℝ3 and G=(g1,g2,g3)T, gi is defined as follows:
(35)gi(y)={yi,∥y∥2≤1yi∥y∥2,∥y∥2>1,y∈Θ,i=1,2,3.
Let v≠u be two arbitrary points in ℝ3, we know that |g1(v)-g1(u)|≤∥G(v)-G(u)∥2. In addition, ∥G(v)-G(u)∥2≤α∥v-u∥2=∥v-u∥2 (here α=1), so |g1(v)-g1(u)|≤∥v-u∥2, and then l1≤1. On the other hand, taking v=(δ,1,0)T, u=(-δ,1,0)T, then |g1(v)-g1(u)|=(1+δ2)-1/2(∥v-u∥2). For ∀ε>0, ∃0<δ<(1+ε)-2-1, ∋|g1(v)-g1(u)|>(1-ε)(∥v-u∥2). So
(36)l1=supv,u∈ℝ3,v≠u|g1(v)-g1(u)|∥v-u∥2=1.
And then we get ℒ=I.TakingP=diag{0,0.44,0}, then M(ℒ,I)+P≥0 and Q=Deco(ℒ-1+P)=diag{1,1.2,1}. It is easy to figure out that ∥Q∥2·∥WQ-1∥2≈0.9882<1. According to Corollary 7, system (33) is globally convergent on W(Θ)+q. Figure 2 depicts the time responses of state variables of system (33) with random initial point starting from W(Θ)+q.Transient behaviors of RNN in system (33) with random initial point x0∈W(Θ)+q.
(a)
(b)Example 3.
Consider the following RNN of system (1):
(37)dxdt=-x+G(WX+q),
where x∈ℝ3, G(x)=argminz∈Θ∥x-0.8z∥2 and Θ={v∈ℝ3:∥v∥2≤1}. The weight matrix and the external bias vector is defined as follows:
(38)W=(0.4-0.30.20.20.5-0.2-0.10.20.6),q=(0.2-0.50.3).Like Example2, all the diagonally nonlinear conclusions in the literature [20, 28, 32] cannot be used here. But this example can be proved to be globally convergent by Corollary 6. Here, we find that 0.8G is equal to the activation operator in Example 2. Furthermore, it is easy to get that G(x)=G(0.8I)G(x) and 〈0.8G(x)-0.8G(y),x-y〉≥∥0.8G(x)-0.8G(y)∥22 which equals 〈G(x)-G(y),x-y〉 ≥ 0.8∥G(x)-G(y)∥22 with each x,y∈Θ. So the activation operator G here is a (0.8I,0.8)-UPPAM. And just like the proof in Example 2, we can get ℒ=1.25I.TakingP=diag{0.2,0.41,0.41}, then M(ℒ,I)+P≥0 and Q=Deco(ℒ-1+P)=diag{1,1.1,1.1}. It is easy to figure out that ∥Q∥2·∥WQ-1∥2≈0.6782 < 0.8=α. According to Corollary 6, system (37) is globally convergent on Θ. Figure 3 depicts the time responses of state variables of system (37) with random initial point starting from Θ.Figure 3
Transient behaviors of RNN in system (37) with random initial point x0∈[-1,1]3.


## 5. Conclusion

Two basic dynamics behaviors, global convergence and asymptotical stability of both static and local field RNNs with UPPAM operators, have been studied under theP-critical condition. It has been proved that when the nonlinear norm determined by the network is bounded, then RNN with UPPAM operator possesses convergent and stable properties in the sense that a discriminant matrix M(ℒ,I)+P is nonnegative definite, where M(ℒ,I) is a matrix related to the network and P is an arbitrary nonnegative definite matrix. Compared with the existing dynamics analysis, the results in this paper extend most of the dynamics conclusions achieved. The requirements of net type, critical form and activation operator’s character in the available literatures have been dearly relaxed. Some typical RNNs with UPPAM activation operators, such as most of the CNNs, BSB, and BCOp-type networks, can apply the theory obtained here to judge the dynamical behavior directly. The significance of the results obtained here not only lies in providing some further cognizance on the essentially dynamical behavior of RNNs, but also in enlarging the application field of them.In this paper, we only achieved the global convergence and asymptotical stability result for UPPAM RNNs and did not discuss another important dynamics behavior, that is, the exponential stability for UPPAM RNNs, and this is under our current investigation.



---

*Source: 760293-2013-11-30.xml*