# Horseshoe Chaos in a 3D Neural Network with Different Activation Functions

Authors: Fangyan Yang; Song Tang; Guilan Xu
Journal: Discrete Dynamics in Nature and Society
 (2013)

---

## Abstract

This paper studies a small neural network with three neurons. First, the activation
function takes the sign function. Although the network is a simple hybrid system
with all subsystems being exponentially stable, we find that it can exhibit very
complex dynamics such as limit cycles and chaos. Since the sign function is a limit
case of sigmoidal functions, we find that chaos robustly exists with some different
activation functions, which implies that such chaos in this network is more related to its
weight matrix than the type of activation functions. For chaos, we present a rigorous
computer-assisted study by virtue of topological horseshoe theory.


---

## Body

## 1. Introduction

Since substantial evidence of chaos is found in biological studies of natural neuronal systems, researchers have realized that chaos is much helpful for neural networks escaping the local minima and may play an essential role in the storage and retrieval of information [1–3]. Thus, a thorough investigation on chaotic dynamics of neural networks is significant for neural networks studies, which has become a popular research field in recent decades. A lot of artificial neural networks have been proposed in order to realize chaotic and hyperchaotic attractors [4–12].Generally, neural networks in real world have very high dimension, which is too hard to study. Fortunately, research in anatomy and physiology shows that neurons in biological brains are grouped together into functional circuits [13, 14]. This implies that a first step before studying chaos in high-dimensional neural networks should be to have detailed investigations on chaos in low-dimensional networks with only a few nerons [15–19].The nonlinearity of neural systems usually comes from the activation functions, which is the reason causing chaos. There are many types of sigmoid functions used in literature, such as the hyperbolic tangent function, piecewise linear functions, the Logistic (sigmoid) function, and the sign functions. So, we are interested in whether chaos in a neural network can take place with all these functions or whether chaos can take place for any type sigmoidal activation function.In order to answer the two questions, this paper will take a limit of the sigmoidal functions by zooming out the input scale and study a small Hopfield neural network (HNN) with hard switches. Such sign function is not only extremely easy to implement, but also of dynamical and biological significance in gene regulatory networks [4, 5]. An interesting phenomenon we find in this paper is that the HNN can demonstrate chaos, although it is a switching system that only consisted of stable subsystems; such chaos still exists even when we replace the sign function with many other activation functions.The following paper is organized as follows. Section2 presents the HNN and demonstrates its chaotic behavior with different activation functions; Section 3 first recalls some theoretical results of topological horseshoe and then presents computer-assisted proof of the existence of chaos; Section 4 draws conclusions.


## 2. Chaos in the Small Network with Different Activation Functions

The Hopfield neural network is described by [20]
(1)x˙i=-aixi+∑j=1n‍wijfi(xi)+bi,i=1,2,…,n,
where fi is a sigmoidal activation function and W=(wij) is an n×n matrix, called weight matrix or connection matrix describing the strength of connections between neurons.In this paper, we only consider three neurons, that is,n=3 and take bi=0, then the small network can be written with the following equation in vector form:
(2)x˙=-diag(a)x+Wf(x),
where a=[a1,a2,a3]T, f(x)=[f(x1),f(x2),f(x3)]T.Now leta=[1,1,2]T, f(xi)=sign(xi), i=1,2,3, and
(3)W=(0.30.80.4-0.970.31+p0.050.23-0.20.198),
where p is a parameter. Then we can easily solve the isolated equilibrium points of (2) while the parameter p from -∞ to +∞, as illustrated in Table 1. Since (2) is symmetric with respect to the origin, its equilibria appears in pairs, and the origin is always an equilibrium. Numerical computation suggests that the origin is always unstable. Since f only takes value of -1, 0, and 1, it is easy to see from the equation that the other equilibria are always exponentially stable.Table 1
Equilibrium points of (2) as we adjust p from -∞ to +∞.
Region ofp
Number of isolated equilibrium points
Stable
Unstable
p
≤
0.61
1
0
1
0.61
<
p
<
0.71
3
2
1
0.71
≤
p
5
4
1To explore complex dynamics in (2), we simulate the system with p<0.61 then numerically find that there are limit circles and strange attractors, as illustrated with the bifurcation plot in Figure 1. The dynamics relies greatly on its initial condition; that is, different initial value may result in different kind of dynamics. During our simulation, we chose x(0)=[0.2,0.3,-0.4]T. The bifurcation diagram indicates that the system (2) is very likely to be chaotic when p is near to zero.Figure 1
The bifurcation plot of (2) on the crossplane x2=0.After careful numerical computation, we obtain an invariant set called attractor in the sense that almost every trajectory with initial point near this set tends to this set, while this set contains a trajectory, that is, dense in it, as illustrated in Figure2 for p=0. It seems a chaotic attractor. In the next section, we will prove that this attractor is indeed chaotic with the topological horseshoe theory.Figure 2
The phase portrait of system (2) and the poincaré plane P:x3=0.2.Since the sign function is a limit of most sigmoidal functions with large-scale input, we in this section will study existence of chaos for the HNN (1) with other sigmoidal activation function. For this purpose, we considerate the following equations:
(4)x˙=-diag(a)x+Wf(kx),
where k is a positive scale factor. Replacing x with k-1x, we have the equivalent system
(5)x˙=-diag(a)x+kWf(x).So for any type sigmoidal activation functionf(x) with its output range from -1 to 1, if k is great enough, f(kx) will be sufficiently close to the discontinuous sign function. The robustness of (2) suggests that (4) could exhibit chaos with the same weight matrix (3). Taking the consideration of the bias, (1) may exhibit chaos for any type sigmoidal activation function when n=3.To illustrate this fact, we will give examples with a hyperbolic tangent function, a Logistic function, a piecewise linear function, and a very complicated function.Case 1.
ftakes the hyperbolic tangent function. Let f(x)=tanh(x). In order to make the neural system (5) chaos, we take k=60. Then we get an attractor shown in Figure 3. The Lyapunov exponents are 0.156, 0.000, and -1.4273. The first one is positive showing that the attractor is most likely chaotic.Figure 3
The phase portrait of (5) with f(x)=tanh(x) and k=60.Case 2.
f takes the Logistic (sigmoid) function. Take f(x)=1/(1+e-x)=0.5tanh(x)+0.5, then tanh(x)=2f(x)+1. From the previous subsection, we have a Hopfield neural network with the Logistic (sigmoid) activation function:
(6)x˙=-diag(a)x+2kWf(x)+b.Here,b=kW[1,1,1]T is the bias.Case 3.
f is a piecewise linear function. Let f(x)=0.5(|x+1|-|x-1|). In order to make the neural system (5) chaos, we also take k=60. Then we get an attractor shown in Figure 4. The three Lyapunov exponents are 0.169, 0.000, and -1.450. The first one is positive suggesting the attractor is chaotic.Figure 4
The phase portrait of (5) with f(x)=0.5(|x+1|-|x-1|) and k=60.Case 4.
f is a complicated piecewise linear function. For the system (4) and the weight matrix (3) at p=0, we take a more complicated f(xi) randomly as shown in Figure 5. Let k=100, a chaotic attractor appears as shown in Figure 6. Since the activation function is too complicate, it is not easy to compute the Lyapunov exponent of the attractor in Figure 6 with enough accuracy; so, in next section we will prove that this attractor is indeed chaotic with the topological horseshoe theory.Figure 5
The plot of the piecewise functionf(xi).Figure 6
The attractor we observed and the poincaré planeP:x3=0.2.It is clear to see from Figures3 to 6 that although their activation functions are much different, the chaotic attractor exists in all four cases and they all look similar. All the above four cases suggest that the existence of chaos in this neural network is robust on the type of activation functions. In another word, the complex dynamics of a small neural network should be more related to its weight matrix.


## 3. Computer-Assisted Proof of Chaos

The existence of a topological horseshoe is recognized as one of the most important signatures of chaos. With the horseshoe theory, one can not only prove chaos rigorously but also reveal the mechanism of chaos with its invariant set. So we will present computer-assisted proof of chaos in the HNN in this section. First, let us recall a theorem on topological horseshoe and the m-shift and then present our main results.LetX be a metric space, Q is a compact subset of X, and g:Q→X is map satisfying the assumption that there exist m mutually disjoint compact subsets Q1,Q2,…,Qm of Q, the restriction of g to each Qi, that is, g∣Qi is continuous.Definition 1.
Letγ be a compact subset of Q, such that for each 1≤i≤m, γi=γ∩Qi is nonempty and compact; then γ is called a connection with respect to Q1,Q2,…,Qm. Let F be a family of connections γs with respect to Q1,Q2,…,Qm satisfying property: γ∈F⇒g(γi)∈F. Then F is said to be g-connected family with respect to Q1,Q2,…,Qm.Theorem 2.
Suppose that there exists ag-connected family with respect to Q1,Q2,…,Qm. Then there exists a compact invariant set K⊂Q, such that K is semiconjugate to m-shift.In this theorem, them-shift is also called Bernoulli shift sometimes, denoted by σ:Σm→Σm, where Σm is the collection of all bi-infinite sequences
(7)s={…,s-n,…,s-1;s0,s1,…,sn,…},si∈{0,1,…,m-1},
and the shift map σ is defined as
(8)σ(s)={…,s-n+1,…,s0;s1,s2,…,sn+1,…}.It is well known thatΣm is a Cantor set, which is compact, totally disconnected, and perfect. As a dynamical system defined on Σm, σ has a countable infinity of periodic orbits consisting of orbits of all periods, an uncountable infinity of aperiodic orbits, and a dense orbit. A direct consequence of these three properties is that the dynamics generated by the shift map is sensitive to initial conditions. Since g is topologically semiconjugate to σ, which means that there exists a continuous surjection h:Σm→X such that f∘h=h∘σ, g must be also sensitive to initial conditions. Mathematically, the complexity of the system g can be measured by its topological entropy, which roughly means the exponential growth rate of the number of distinguishable orbits as time advances. As another result of the semiconjugate, the topological entropy of g, denoted by ent(g), is not less then m. When m>1, ent(g)>0, therefore the system is chaotic. For more details of the above symbolic dynamics and horseshoe theory, we refer the reader to [21–24].In what follows we will study existence of horseshoes embedded in the attractor in Figure2. For this purpose, we will utilize the technique of cross section and the corresponding Poincaré map. Consider the section plane P:x3=0.2, as shown in Figure 2. The Poincaré map π:P→P is chosen as follows. For each x∈P, π(x) is taken to be the second return point in P under the flow with the initial condition x.To find the horseshoe, we use the efficient method proposed in [21, 25] which has been implemented with a MATLAB toolbox called “a toolbox for finding horseshoes in 2D map” (download: http://www.mathworks.com/matlabcentral/fileexchange/14075). The method is so powerful that it has been successfully applied in a number of chaotic systems [26–29], a fractional-order system [30],even a hyperchaotic system [31].After many attempts, we carefully pick two quadrangles|A1B1C1D1| and |A2B2C2D2| in this section plane P with their vertices as follows:
(9)A1=(0.28570,-0.40254,0.2),B1=(0.28700,-0.40195,0.2),C1=(0.28631,-0.40313,0.2),D1=(0.28502,-0.40370,0.2),A2=(0.28880,-0.40116,0.2),B2=(0.28886,-0.40114,0.2),C2=(0.28821,-0.40234,0.2),D2=(0.28816,-0.40237,0.2).By means of interval analysis, we compute the Poincaré map on|A1B1C1D1| and |A2B2C2D2| and then have the following statement.Theorem 3.
For the Poincaré mapπ corresponding to the cross sections |A1B1C1D1| and |A2B2C2D2|, there exists a closed invariant set Λ⊂Q≜|A1B1C1D1|∪|A2B2C2D2| for which π∣Λ is semiconjugate to the 2-shift map.Proof.
To prove this statement, we will find two disjoint compact subsetsQ1 and Q2 of Q, such that the existence of a π-connected family can be easily derived.
The first subsetQ1 takes |A1B1C1D1| as shown in Figure 7, and the Poincaré map sends this subset to its image |A1′B1′C1′D1′| as follows:
(10)A1D1⟶A1′D1′,B1C1⟶B1′C1′,
showing that A1D1 is mapped left of the side A1D1 and B1C1 is mapped right of the side B2C2. In this case, we say that the image π(|A1B1C1D1|) lies wholly across the quadrangles |A1B1C1D1| and |A2B2C2D2| with respect to A1D1 and B2C2.
The second subsetQ2 takes |A2B2C2D2| as shown in Figure 8, and the Poincaré map sends this subset to its image |A2′B2′C2′D2′| as follows:
(11)A2D2⟶A2′D2′,B2C2⟶B2′C2′,
showing that A2D2 is mapped left of the side A1D1 and B2C2 is mapped right of the side B2C2. In this case, we say that the image π(|A2B2C2D2|) lies wholly across the quadrangles |A1B1C1D1| and |A2B2C2D2| with respect to A1D1 and B2C2.
Generally, (2) can be regarded as a switching system consisting of eight very simple continuous subsystems and 12 quarter planes called switching planes; for detail, see [5]. In this system, every trajectory in the neighborhood of the attractor transversely intersects with the switching plane, and the Poincaré map π can be regarded as a composition of a series of continuous submaps by the subsystems. Since each subsystem is linear, it is easy to prove that for the quadrangle region |A1B1C1D1| or |A2B2C2D2| all submaps are continuous; consequently, π∣Q is continuous.
Note that the subsets|A1B1C1D1| and |A2B2C2D2| are mutually disjointed. It is easy to see from the whole acrossness of π(|A1B1C1D1|) and π(|A2B2C2D2|) with respect to A1D1 and B2C2 that there exists a π-connected family with respect to Q1=|A1B1C1D1| and Q2=|A2B2C2D2|. In view of Theorem 2, this means that the Poincaré map π is semiconjugate to 2-shift map.
It is easy to see from Theorem3 that the entropy of π is not less than log2, so the attractor in Figure 2 is a chaotic attractor indeed.
So we can prove the existence of chaos in the attractor shown in Figure6 by the same way in Section 3. The new vertices of the two subsets |A1B1C1D1| and |A2B2C2D2| are
(12)A1=(0.28589,-0.40149,0.2),B1=(0.28766,-0.40075,0.2),C1=(0.28730,-0.40122,0.2),D1=(0.28555,-0.40196,0.2),A2=(0.28999,-0.39985,0.2),B2=(0.29007,-0.39982,0.2),C2=(0.28996,-0.40002,0.2),D2=(0.28988,-0.40005,0.2).
The images of|A1B1C1D1| and |A2B2C2D2| are shown in Figure 9, from which we can see easily that the images of |A1B1C1D1| and |A2B2C2D2| across simultaneously |A1B1C1D1| and |A2B2C2D2|, that is, analogous to Figures 7 and 8. From Theorem 2, we infer that there exists a β-connected family with respect to |A1B1C1D1| and |A2B2C2D2|, where β is the corresponding Poincaré map. From the topological horseshoe theory we have ent(π)⩾log2 after similar arguments. The positive entropy suggests that the attractor in Figure 6 is chaotic indeed.Figure 7
The imageπ(|A1B1C1D1|) lies wholly across |A1B1C1D1| and |A2B2C2D2|.Figure 8
The imageπ(|A2B2C2D2|) lies wholly across |A1B1C1D1| and |A2B2C2D2|.Figure 9
The new topological horseshoe.


## 4. Conclusions

In this paper, we have studied a 3D Hopfield neural network with the sign activation function. Computer simulation shows that this HNN can exhibit chaotic attractors and limit cycles with respect top. In order to verify the chaotic behavior, we present a computer-assisted verification for the existence of horseshoes imbedded in this system. We also show evidence that chaos could possibly be exhibited by the HNN with any type of sigmoidal activation function. In another word, such chaos should be more related to its weight matrix than the type of activation functions. In addition, since the HNN is a switching system only consisting of stable subsystems, this fact suggests that the dynamics of a hybrid system could be much more complex than we used to think. This may be of interest to researchers of neural networks, nonlinear dynamics, and so on.



---

*Source: 430963-2013-04-28.xml*