# Hopfield Neural Networks with Unbounded Monotone Activation Functions

Authors: Nasser-eddine Tatar
Journal: Advances in Artificial Neural Systems
 (2012)

---

## Abstract

For the Hopfield Neural Network problem we consider unbounded monotone nondecreasing activation functions. We prove convergence to zero in an exponential manner provided that we start with sufficiently small initial data.


---

## Body

## 1. Introduction

Of concern is the following system:(1)xi′(t)=-ai(t)xi(t)+∑j=1m‍bij(t)fj(xj(t))+ci(t),i=1,…,m,
where ai(t)≥0, bij(t),ci(t), i,j=1,…,m are continuous functions, and fj are the activation functions which will be assumed continuous and bounded by some nondecreasing (and possibly unbounded functions).This system appears in Neural Network theory [1, 2]. As is well-known, Neural Networks are an important tool in business intelligence. Their architecture differs from the one of standard computers in that it consists of a large number of processors (neurons) with high connections between them. In contrast to computers with a single processor, (Artificial) Neural Networks perform their computations in parallel.Just as the human brain, the neurons receive weighted signals from the neurons in the input layer, sum up these inputs and test against a threshold value. Then they decide to fire or not.The applications are numerous, we may cite few: modelling soil behavior, design of tunnels, image processing, graph flow, data deconvolution, energy demand forecasting, ecosystem evaluation, scheduling optimization, targeted marketing, medical diagnosis, time series analysis, and stock market.Neural Networks are able to analyze and evaluate many phenomena in real world business as well as in industry. Some of their advantages over the conventional computers are forecasting, strategy planning, and predicting many phenomena.Different methods have been used by many authors to study the well-posedness and the asymptotic behavior of solutions [3–20]. In particular, a lot of efforts are devoted in improving the set of conditions on the different coefficients involved in the system as well as the class of activation functions. Regarding the latter issue, the early assumptions of boundedness, monotonicity, and differentiability have been all relaxed to merely a global Lipschitz condition. Since then, it seems that, this assumption has not been weakened further considerably although there is a great need for that [21]. A slightly weaker condition: xigi(xi)>0, xi≠0 and there exist λi>0 such that λi=supxi≠0(gi(xi)/xi), where gi(xi)=fi(xi)-fi(xi*) and xi* is the equilibrium, has been used in [22–24] (see also [25–27]).Here we assume that the activation functionsfj are bounded by continuous monotone nondecreasing functions gj, that is,
(2)|fj(x)|≤gj(|x|),j=1,…,m.
The functions gj are not necessarily Lipschitz continuous and they may be unbounded (like power type functions with powers bigger than one). We can also consider activation functions with discrete delays as is explained below. We prove that, for sufficiently small initial data, solutions decay to zero exponentially.The local existence and existence of equilibria is standard (see, the Gronwall-type Lemma1 below) and the global existence follows from the estimation in our theorem below. However, the uniqueness of the equilibrium is not trivial. Here, as we are concerned with the convergence to zero rather than stability of equilibrium, the uniqueness of equilibrium is put aside.The next section contains the statement and proof of our result as well as a crucial lemma we will be using.


## 2. Exponential Convergence

In this section it is proved that solutions converge to zero in an exponential manner when the activation functions are (or bounded by) continuous nondecreasing and unbounded functions. To this end we need a lemma due to Bainov and Simeonov [3].LetI⊂R and let g1,g2:I→R∖{0}. We write g1∝g2 if g2/g1 is nondecreasing in I.Lemma 1.
Leta(t) be a positive continuous function in J:=[α,β),kj(t,s),j=1,…,n are nonnegative continuous functions for α≤s≤t<β which are nondecreasing in t for any fixed s,gj(u),j=1,…,n are nondecreasing continuous functions in R+, with gj(u)>0 for u>0 and u(t) is a nonnegative continuous functions in J. If g1∝g2∝⋯∝gn in (0,∞), then the inequality
(3)u(t)≤a(t)+∑j=1n‍∫αt‍kj(t,s)gj(u(s))ds,t∈J
implies that
(4)u(t)≤ωn(t),α≤t<β0,
where ω0(t):=sup0≤s≤ta(s),
(5)ωj(t):=Gj-1[Gj(ωj-1(t))+∫αt‍kj(t,s)ds],j=1,…,n,Gj(u):=∫uju‍dxgj(x),u>0(uj>0,j=1,…,n),
and β0 is chosen so that the functions ωj(t),j=1,…,n are defined for α≤t<β0.For the statement of our theorem we will need the following notation:(6)a(t):=min1≤i≤m{ai(t)},(7)ω0(t):=x(0)+∑i=1m‍∫0t‍exp[∫0s‍a(σ)dσ]|ci(s)|ds,(8)ωj(t):=Gj-1[(∑i=1m‍|bij(s)|)dsGj(ωj-1(t))+∫0t‍exp[∫0s‍a(σ)dσ](∑i=1m‍|bij(s)|)ds],(9)Gj(u):=∫uju‍dxgj(x),u>0(uj>0,j=1,…,m).Theorem 2.
Assume thatfj satisfy |fj(x)|≤gj(|x|),j=1,…,m for some continuous nondecreasing (and possibly unbounded) functions gj, j=1,…,m in R+, with gj(u)>0 for u>0. Assume further that ai(t)≥0,bij(t), ci(t),i,j=1,…,m are continuous functions. If g1∝g2∝⋯∝gm in (0,∞) then, there exists β0>0 such that
(10)x(t)≤ωm(t)exp[-∫0t‍a(s)ds],0≤t<β0,
where x(t):=∑i=1m‍|xi(t)|.Proof.
From (1) and our assumption on fj we see that
(11)D+|xi(t)|≤-ai(t)|xi(t)|+∑j=1m‍|bij(t)|gj(|xj(t)|)+ci(t),t>0,i=1,…,m,
or
(12)D+x(t)≤-min1≤i≤m{ai(t)}x(t)+∑i,j=1m‍|bij(t)|gj(|xj(t)|)+∑i=1m‍ci(t),t>0,
where D+ denotes the right Dini derivative. Hence
(13)D+x(t)≤-a(t)x(t)+∑i,j=1m‍|bij(t)|gj(|xj(t)|)+∑i=1m‍ci(t),t>0.
In virtue of (13) we derive
(14)D+{x(t)exp[∫0t‍a(s)ds]}≤exp[∫0t‍a(s)ds]∑i,j=1m‍|bij(t)|gj(x(t))+exp[∫0t‍a(s)ds]∑i=1m‍ci(t),
and thereafter (see [28])
(15)x~(t)≤x(0)+∑j=1m‍∫0t‍{(∑i=1m‍|bij(s)|)exp[∫0s‍a(σ)dσ]×(∑i=1m‍|bij(s)|)gj(x~(s))exp[∫0s‍a(σ)dσ]}ds+∑i=1m‍∫0t‍exp[∫0s‍a(σ)dσ]|ci(s)|ds,t>0,
where
(16)x~(t):=x(t)exp[∫0t‍a(s)ds].
Applying Lemma 1 we obtain the existence of β0 such that
(17)x~(t)≤ωm(t),0≤t<β0,
where ω0(t),ωj(t), and Gj(u),j=1,…,m are as defined in (7)–(9).Remark 3.
To have global existence we needβ0=∞ and this is possible when
(18)∫0∞‍exp[∫0s‍a(σ)dσ](∑i=1m‍|bik(s)|)ds≤∫ωk-1∞‍dzgk(z),k=1,…,m.Remark 4.
Assuming thatωn(t) grows up at most polynomially, we see that the rate is exponential.Remark 5.
Note here that our assumptions in the previous remarks involve a smallness condition on the initial data.


## 3. Applications

Using Kirchhoff's law, Hopfield demonstrated that electrical circuits could behave as a small Neural Network. His original system has the form:(19)Ciduidt=∑i=1m‍Tijvj-uiRi+Ii,i=1,…,m,vj=gj(uj),j=1,…,m,
where Ci>0: Capacity, Ri>0: Resistance, Ii: Bias (external action on the ith neuron),  ui: Input (voltage) of the ith neuron, vi: Output of the ith neuron, Tij: The coupling constants of the jth neuron with the ith  neuron, and gj(uj): Activation functions.T
i
j are called elements of the weight matrix or connection matrix. This matrix describes the strength of connection between neurons. The expression 1/Ri is sometimes called the feedback factor.The functionsgj(uj) are nonlinear functions characterizing the response of the ith neuron to changes in its state. Typical activation functions are the “Step function”, the “Sign function”, the “Gaussian” function, the “Hyperbolic function”, and the “Exponential type function”. However, it has been established that many other activation functions arise in practice which are not of these forms. Therefore there is a need to enlarge these classes of functions to more general ones.In Neural Network Theory researchers are rather interested in designing models which are globally asymptotically stable. That is, the models must have a unique equilibrium which attracts all the solutions. Of course the rate of convergence is extremely important and it is preferable to have an exponential convergence rate. In the present work (for the case of variable coefficients) we prove that if solutions start close enough to zero then they will be attracted by zero. Our theorem shows that solutions remain bounded by(20)ωm(t)exp[-∫0t‍a(s)ds],
as long as t<β0 defined as a bound for the interval of existence of the ωj's (see (8)). In Remark 3 we gave a sufficient condition ensuring the existence of the ωj's for all time. That is conditions for which β0=+∞. It follows then that, under these conditions, the states actually converge to zero as t goes to infinity with an exponential rate in case ωm(t) does not grow too fast and ∫0t‍a(s)ds→∞ as t→∞.The example below represents a possible practical situation for which our argument applies. Again we establish a sufficient explicit condition leading to exponential convergence to zero provided that the initial data are small enough.Example 6.
Consider the special (but common) functionsgj(x)=xni,ni>1,i=1,…,m. The order g1∝g2∝⋯∝gm means n1≤n2≤⋯≤nm. Clearly, in this case Gj(x)=(x1-nj/(1-nj))-(x01-nj/(1-nj)), Gj-1(z)=[x01-nj-(nj-1)z]-(1/(nj-1)), and for t>0(21)ωj(t)={(∑i=1m‍|bij(s)|)dsωj-11-nj(t)-(nj-1)∫0t‍exp[∫0s‍a(σ)dσ]×(∑i=1m‍|bij(s)|)ds}-(1/(nj-1)).
The value β0 will be the largest value of t for which
(22)ωj-1nj-1(t)∫0t‍exp[∫0s‍a(σ)dσ](∑i=1m‍|bij(s)|)ds<1nj-1,
for all j=1,…,m. As we are interested in the long time behavior of solutions it is necessary that these conditions hold for all t. Our theorem then implies that solutions are bounded by the expression
(23)ωm(t)exp[-∫0t‍a(s)ds],
which provides us with an exponential decay under some fairly reasonable assumptions.


### 3.1. Discrete Delays

The case where we have discrete delays in the activation functions, that is,(24)xi′(t)=-ai(t)xi(t)+∑j=1m‍bij(t)fj(xj(t-τij))+ci(t),i=1,…,m,
where τij are different finite delays, can be treated similarly. We use the following functional
(25)Ξ(t):=∑i,j=1m‍e-∫0t‍a(s)ds∫t-τijt‍e∫0s+τij‍a(σ)dσ×|bij(s+τij)|gj(|xj(s)|)ds
to get rid of the delayed terms and replace them by terms without delays.


## 3.1. Discrete Delays

The case where we have discrete delays in the activation functions, that is,(24)xi′(t)=-ai(t)xi(t)+∑j=1m‍bij(t)fj(xj(t-τij))+ci(t),i=1,…,m,
where τij are different finite delays, can be treated similarly. We use the following functional
(25)Ξ(t):=∑i,j=1m‍e-∫0t‍a(s)ds∫t-τijt‍e∫0s+τij‍a(σ)dσ×|bij(s+τij)|gj(|xj(s)|)ds
to get rid of the delayed terms and replace them by terms without delays.



---

*Source: 571358-2012-12-31.xml*