# Storage Capacities of Twin-Multistate Quaternion Hopfield Neural Networks

Authors: Masaki Kobayashi
Journal: Computational Intelligence and Neuroscience
 (2018)

---

## Abstract

A twin-multistate quaternion Hopfield neural network (TMQHNN) is a multistate Hopfield model and can store multilevel information, such as image data. Storage capacity is an important problem of Hopfield neural networks. Jankowski et al. approximated the crosstalk terms of complex-valued Hopfield neural networks (CHNNs) by the 2-dimensional normal distributions and evaluated their storage capacities. In this work, we evaluate the storage capacities of TMQHNNs based on their idea.


---

## Body

## 1. Introduction

A complex-valued Hopfield neural network (CHNN) is a multistate model of Hopfield neural network. CHNNs can deal with multilevel information and have been applied to the storage of image data [1–7]. They have also been extended using Clifford algebra, which includes a complex field, hyperbolic algebra, and quaternion field. Several models of hyperbolic Hopfield neural network have been proposed [8–11]. Isokawa et al. [12] proposed quaternion Hopfield neural networks (QHNNs) employing the split activation function. Minemoto et al. [13] studied QHNNs using the polar-represented activation function. Several other models of QHNNs have also been proposed [14–17]. In this work, we study the twin-multistate quaternion Hopfield neural networks (TMQHNNs) [18]. The neuron of a TMQHNN consists of a pair of complex-valued multistate neurons. The TMQHNN requires only half the connection weight parameters of the CHNN.Storage capacity is an important issue in Hopfield neural networks. When a Hopfield neural network is given a training pattern, the weighted sum input is decomposed into main and crosstalk terms. The main term enables the Hopfield neural network to memorize the training patterns. The crosstalk term interferes with the storage of training patterns. The storage capacity of conventional Hopfield neural network has been investigated by evaluating the crosstalk term [19]. Jankowski et al. [1] and Kobayashi [20] applied this technique to CHNNs and rotor Hopfield neural networks (RHNNs), respectively. The RHNN is an extension of the CHNN using vectors and matrices [21]. In this work, we provide the Hebbian learning rule for TMQHNNs and evaluate the storage capacity based on Jankowski’s concept. In the case of TMQHNNs, the cross talk term is decomposed into two complex parts. By evaluating both parts, we determine the storage capacity of TMQHNNs. The theory suggests that TMQHNNs have half the storage capacities of CHNNs. In addition, we compared the storage capacities of CHNNs and TMQHNNs by computer simulation.The rest of this paper is organized as follows: Sections2 and 3 introduce CHNNs and TMQHNNs, respectively. Section 4 provides the Hebbian learning rule for the TMQHNN and evaluates the storage capacity. It also contains descriptions of the computer simulations conducted to verify our analysis. Section 5 concludes this paper.


## 2. Complex-Valued Hopfield Neural Networks

The CHNNs are briefly described [1]. Let za and wab be the state of neuron a and the connection weight from neuron b to neuron a, respectively. The weighted sum input to neuron a is given by(1)Ia=∑b=1NCwabzb,where NC is the number of neurons. For the resolution factor K, we define θK=π/K. For the weighted sum, input I=rexpiθ, where r≥0 and 0≤θ<2π; the complex-valued multistate activation function is defined by(2)fCI=10≤θ<θK,exp2iθKθK≤θ<3θK,⋮⋮exp2K−1iθK2K−3θK≤θ<2K−1θK,12K−1θK≤θ<2π.We define the set of neuron states and denote it as follows:S=1,exp2iθK,…,exp2K−1iθK. The connection weights must satisfy the following conditions:(3)wab¯=wba,(4)waa≥0.Then, the CHNN converges to a fixed point.Letzp=z1p,z2p,…,zNCp 1≤p≤PC be the pth training pattern, where PC is the number of training patterns. The Hebbian learning rule is defined as(5)wab=1NC∑p=1PCzapzbp¯.Then, the connection weights satisfywab¯=wba. Giving the qth training pattern to the CHNN, the weighted sum input to neuron a is(6)Iaq=∑b=1NCwabzbq(7)=zaq+1NC∑p≠q∑b=1NCzapzbp¯zbq.The second term of (7) is referred to as the crosstalk term. The crosstalk term interferes with the storage of training patterns. We define(8)Aaq=1NC∑p≠q∑b=1NCzapzbp¯zaq¯zbq.Then, we have(9)Iaq=zaq1+Aaq.Ifarg1+Aaq<θK, then we have fCIaq=zaq. Therefore, if arg1+Aaq<θK for all a, the qth training pattern is a fixed point. We regard NCAaq as the summation of PCNC random variables of V for simplicity, although the summation consists of exactly P−1NC terms. The real and imaginary parts of each random variable have the equal variance σ and do not have correlations. Setting PC=αNC, NCAaq is regarded as the summation of αNC2 random variables. Then, we have(10)1αAaq=1αNC∑p≠q∑b=1NCzapzbp¯zaq¯zbq.LetXA and YA be the real and imaginary parts of 1/αAaq, respectively. From the central limit theorem, we have(11)XA,YA∼12πσ2exp−XA2+YA22σ2.ForAaq=XA′+YA′i, we have(12)XA′,YA′∼12απσ2exp−XA′2+YA′22ασ2.


## 3. Twin-Multistate Quaternion Hopfield Neural Networks

A quaternion is expressed byq=q0+q1i+q2j+q3k using real numbers q0,q1,q2, and q3. The imaginary units i,j, and k satisfy the following properties:(13)i2=j2=k2=−1,ij=−ji=k,jk=−kj=i,ki=−ik=j.The quaternions satisfy the associative and distributive laws. For a complex numberc, we have the important equality:(14)jc=c¯j.Puttingx=q0+q1i and y=q2+q3i, the quaternion q is described as q=x+yj. For quaternions x+yj and x′+y′j, the addition and multiplication are described as(15)x+yj+x′+y′j=x+x′+y+y′j,x+yjx′+y′j=xx′−yy′¯+xy′+x′¯yj.The conjugate ofq is defined as(16)q¯=q0−q1i−q2j−q3k=x¯−yj.Then, we have the equality(17)qq¯=xx¯+yy¯=x2+y2.In the TMQHNNs, the neuron states and connection weights are represented by quaternions. These neuron states and connection weights are denoted in the same way as those of CHNNs. The number of neurons in a TMQHNN is denoted asNQ. The weighted sum input to neuron a is given by(18)Ia=∑b=1NQwabzb.For the weighted sum, inputI=Ix+Iyj, the activation function is defined as(19)fQI=fCIx+fCIyj.Therefore, the set of neuron states isS+Sj. The connection weights must satisfy conditions (3) and (4). Then, the TMQHNN converges to a fixed point.


## 4. Storage Capacity of Twin-Multistate Quaternion Hopfield Neural Networks

We provide the Hebbian learning rule for TMQHNNs. Letzp=z1p,z2p,…,zNQp 1≤p≤PQ be the pth training pattern, where PQ is the number of training patterns. The Hebbian learning rule is given by(20)wab=12NQ∑p=1PQzapzbp¯.Then, the connection weights satisfyw¯ab=wba. Giving the qth training pattern to the TMQHNN, the weighted sum input to neuron a is(21)Iaq=∑b≠awabzbq(22)=zaq+12NQ∑p≠q∑b=1NQzapzbp¯zbq.The second term of (22) is also referred to as the crosstalk term and interferes the storage of training patterns. We decompose the quaternion zap into a pair of complex numbers by zap=xap+yapj to investigate the storage capacity. Then, we have(23)Iaq=xaq+yaqj+12NQ∑p≠q∑b=1NQxap+yapjxbp−ybpjxbq+ybqj=xaq+12NQ∑p≠q∑b=1NQxapxbpxbq+xapybpybq¯−yapxbp¯ybq¯+yapybp¯xbq+yaqj+12NQ∑p≠q∑b=1NQxapxbpybq−xapybpxbq¯−yapxbp¯xbq¯+yapybp¯ybqj.We define(24)Baq=12NQ∑p≠q∑b=1NQxapxbpxbq+xapybpybq¯−yapxbp¯ybq¯+yapybp¯xbqxaq¯,Caq=12NQ∑p≠q∑b=1NQxapxbpybq−xapybpxbq¯−yapxbp¯xbq¯+yapybp¯ybqyaq¯.Then, we have(25)Iaq=xaq1+Baq+yaq1+Caqj.Ifarg1+Baq<θK and arg1+Caq<θK, then we have fQIaq=zaq. We regard 2NQBaq and 2NQCaq as the summations of 4NQPQ random variables of S. Then, Baq and Caq follow the same distributions. Thus, we can discuss only Baq. If the TMQHNN is used instead of the CHNN, NC=2NQ is required, since a twin-multistate quaternion neuron consists of two complex-valued multistate neurons. Setting PQ=βNC=2βNQ, 2NQBaq is regarded as the summation of 8βNQ2 random variables, and we have(26)12βBaq=122βNQ∑p≠q∑b=1NQxapxbpxbq+xapybpybq¯−yapxbp¯ybq¯+yapybp¯xbqxaq¯.LetXB and YB be the real and imaginary parts of 1/2βBaq, respectively. From the central limit theorem, we have(27)XB,YB∼12πσ2exp−XB2+YB22σ2.PuttingBaq=XB′+YB′i, we have(28)XB′,YB′∼14βπσ2exp−XB′2+YB′24βσ2.We require the same distributions for (12) and (22) and obtained α=2β. Thus, the CHNN has double the storage capacity of the TMQHNN.Computer simulations were conducted to verify our analysis.K was varied from 4−12 in steps of 2, and P was varied from 1−20. For each K and P, 100 sets of training patterns were generated randomly; the number of trials was 100. The CHNN and TMQHNN attempted to store the training patterns by the Hebbian learning rule. If all the training patterns were fixed, the trial was regarded as successful, otherwise, as failed. Figure 1 shows the simulation results. The horizontal and vertical axes indicate the number of training patterns and success rate, respectively. The simulation results showed that the storage capacity of the TMQHNN was a bit larger than half that of the CHNN.Figure 1
The simulation result for storage capacities.


## 5. Conclusions

A TMQHNN needs only half the connection weight parameters of CHNN and is expected to have a smaller storage capacity. However, the storage capacity had not yet been analyzed. In this work, we defined the Hebbian learning rule for TMQHNNs and analyzed their storage capacity. The analysis demonstrated that a TMQHNN had half of the storage capacity of a CHNN. In addition, a computer simulation was conducted to verify our analysis. The simulation results confirmed our analysis. In future, we intend to study the storage capacity using different methods [19, 22, 23].



---

*Source: 1275290-2018-11-01.xml*