Journal of the Physical Society of Japan
Vol. 78, No. 3, March, 2009, 033001#2009 The Physical Society of JapanLETTERS

Parallel Dynamics of Continuous Hopﬁeld Model Revisited
Kazushi M IMURA
Graduate School of Information Sciences, Hiroshima City University, Hiroshima 731-3194(Received November 26, 2008; accepted December 25, 2008; published February 25, 2009)

We have applied the generating functional analysis (GFA) to the continuous Hopﬁeld model. We havealso conﬁrmed that the GFA predictions in some typical cases exhibit good consistency with computer
simulation results. When a retarded self-interaction term is omitted, the GFA result becomes identical tothat obtained using the statistical neurodynamics as well as the case of the sequential binary Hopﬁeldmodel.
KEYWORDS: generating functional analysis, statistical neurodynamics, exact solution, continuous Hopfield model,
zero-temperature Langevin dynamics
DOI: 10.1143/JPSJ.78.033001

In the ﬁeld of information theory, various important
inference problems are described by continuous nonlinear
systems which include disorder. Some examples are codedivision multiple-access (CDMA),1–4) error-correcting codes,5–10) lossy compression,11–16) and computer vision.17)
These techniques play an important role in our modern life.
Equilibrium statistical mechanics is used to evaluate theperformance of information processing problems that areformulated using statistical models with a large degree offreedom. These problems can be reformulated as a computation of marginal probabilities on factor graphs. Some
iterative methods, which can be used to calculate marginal
probabilities approximately, e.g., belief propagation, give us
eﬀective algorithms as iterative equations with parallel
update. In general, the theoretical treatment of the dynamics
of disordered systems is complicated compared with that ofequilibrium statistical mechanics. It’s important to understand the dynamical property of iterative algorithms suchas a belief-propagation-based one in order to improve its
performance.
In this study, we treat the continuous Hopﬁeld model18)
with the zero-temperature parallel dynamics as a typicalexample. There are many studies using the generating
functional analysis (GFA)19) of the Hopﬁeld model withvarious dynamics. The GFA allows us to study the dynamics
of an inﬁnitely large disordered system in an exact way.
For instance, Crisanti and Sompolinsky analyzed theasymmetric spherical-state Hopﬁeld model with Langevindynamics.20) Crisanti and Sompolinsky also analyzed an
asymmetric Ising system with sequential-updated Glauber
dynamics.21) Rieger analyzed the binary-state Hopﬁeld
model with sequential-updated Glauber dynamics.22)
Gardner et al. analyzed the Hopﬁeld model with the zerotemperature parallel dynamics.23) Their approach can beused to calculate magnetization after several time steps but
not to treat continuous states.
Kawamura et al. have found the sequential binary
Hopﬁeld model shows chaotic behaviors.24,25) In the caseof the sequential binary Hopﬁeld model with the Glauberdynamics, Kawamura and Okada have shown that the GFA
result and the statistical neurodynamics (SND)26) result are
identical.27) Bollé et al. have investigated the relationshipE-mail: mimura@hiroshima-cu.ac.jp

between the GFA and the SND using the binary Hopﬁeld
model with the Glauber dynamics.28) Hashiguchi andKawamura have studied the sequential continuous Hopﬁeld
model by using the SND.29) However, the relationshipbetween the GFA and the SND in the continuous Hopﬁeld
model with parallel dynamics has remained unclear so far.
Therefore, we here study this relationship. We apply theLangevin dynamics20) at zero temperature as the parallel
dynamics to treat the continuous state. This dynamics has
been applied to analysis of the batch minority games,30) etc.
To conﬁrm the validity of our analysis, we have performedsome computer simulations for some typical conditions.
We consider a nonlinear system that consists of N
continuous variables i ðtÞ 2 R at time t. In this paper, wetreat the following updating rule:i ðt þ 1Þ ¼ f ðui ðtÞÞ;ui ðtÞ  hi ððtÞÞ þ i ðtÞ

ð1Þ

where f : R ! R is an arbitrary function that called atransfer function. Here, ui ðtÞ is called a local ﬁeld, thatincludes the function hi ððtÞÞ of ðtÞ ¼ y ð1 ðtÞ; . . . ; N ðtÞÞand a time-dependent external ﬁeld i ðtÞ to calculate aresponse function. The operator y denotes the transpose. We
analyze the transient dynamics in the thermodynamical
limit, i.e., N ! 1. The Glauber dynamics can apply only
the Ising system. Therefore, we ﬁrst deﬁne the following
deterministic updating rule for the variable i ðtÞ at timet as½i ðt þ 1ÞjðtÞ ¼ ½i ðt þ 1Þ  f ðui ðtÞÞ;

ð2Þ

which can be regarded as the Langevin dynamics20) at zerotemperature, where  denotes the Dirac delta fuction. Thisdynamics represents Markovian dynamics, so the pathprobability p½ð0Þ; . . . ; ðtÞ is simply given by the productsofQNthe individual transition probabilities ½ðs þ 1ÞjðsÞ ¼i¼1 ½i ðs þ 1ÞjðsÞ of the chain:t1Yp½ð0Þ; . . . ; ðtÞ ¼ p½ð0Þ½ðs þ 1ÞjðsÞ: ð3Þ
s¼0If we assume that we know the initial microscopic state
precisely,
it allows for a factorized distribution as p½ð0Þ ¼
QNi¼1 pi ½i ð0Þ.
To analyze the transient dynamics of the continuous
nonlinear system, we deﬁne a generating functional Z½  asfollows:

033001-1

LETTERS

J. Phys. Soc. Jpn., Vol. 78, No. 3"ZZ½  ¼Rðtþ1ÞN"tY#(dðsÞ p½ð0Þ; . . . ; ðtÞs¼0

 exp i

K. M IMURAtX

 exp i

þ Q^ ðs; s0 Þh^ðsÞh^ðs0 Þ þ K^ ðs; s0 ÞðsÞh^ðs0 Þ

ð4Þ

s¼0

ðsÞ ¼ y ð 1 ðsÞ; . . . ;where ðsÞ ¼ y ð1 ðsÞ; . . . ; N ðsÞÞ and19)
one can obtain from Z½ N ðsÞÞ. In a familiar way,
all averages of interest by diﬀerentiation, i.e., spinaverages (or overlap), correlation-functions, and responsefunctions:@Z½ mi ðsÞ ¼ hi ðsÞi ¼ i lim;
!0 @ i ðsÞ@Z½ ;!0 @ i ðsÞ@ j ðs0 Þ@hi ðsÞi@Z½ Gij ðs; s0 Þ ¼¼ i lim;
0!0 @ i ðsÞ@j ðs0 Þ@j ðs Þ

½q^ðs; s0 ÞðsÞðs0 Þ

s¼0 s0 ¼0#ðsÞ  ðsÞ ;t1 Xt1Xþit1
Xh^ðsÞfhðsÞ  i ðsÞ  k^ðsÞ1i g

s¼0

i

t1XðsÞm^ ðsÞ1i  i

s¼0and¼

Cij ðs; s0 Þ ¼ hi ðsÞj ðs0 Þi ¼  lim

1lnNtX)!
ðsÞ i ðsÞ;Zx x^ y x^(N Xt1X
 exp i½x^ ðsÞx ðsÞ þ y^ ðsÞy ðsÞ  x ðsÞy ðsÞ¼2 s¼0

ð5Þ

where h i and    denote the average with respect to the pathprobability p½ð0Þ; . . . ; ðtÞ and the average over thedisorder that is included in the generating functional Z½ ,respectively.
In the continuous Hopﬁeld model, the local ﬁeld is deﬁned
as follows:
NN
X
1Xui ðsÞ Jij j ðsÞ þ i ðsÞ; Jij ¼i j ; ð6ÞNj¼1 ð6¼iÞ
¼1

N Xt1 X
t11X

2 ¼2 s¼0 s0 ¼0½x^ ðsÞQðs; s0 Þx^ ðs0 Þ

þ x^ ðsÞKðs; s0 Þy^ ðs0 Þ þ y^ ðsÞKðs; s0 Þx^ ðs0 Þþ y^ ðsÞqðs; s0 Þy^ ðs0 Þ ;

t1
Y
dhðsÞ
pﬃﬃﬃﬃﬃﬃ ;
2s¼0

where  is the loading rate. The pattern components fi gfollow an identical and independent distribution

x ¼t1 Y
Nt1 Y
NYYdx ðsÞdx^ ðsÞ
pﬃﬃﬃﬃﬃﬃ ; x^ ¼
pﬃﬃﬃﬃﬃﬃ ;2
2s¼0 ¼2
s¼0 ¼211pði Þ ¼ ði  1Þ þ ði þ 1Þ:y ¼t1 Y
Nt1 Y
NYYdy ðsÞdy^ ðsÞpﬃﬃﬃﬃﬃﬃ ; y^ ¼
pﬃﬃﬃﬃﬃﬃ :22s¼0 ¼2
s¼0 ¼22

ð7Þ

Performing an appropriate gauge transformation, we canput 1i ¼ 1 ð8iÞ without a loss of generality. We ﬁrst
separate the localR Q
ﬁeld Qui ðsÞ by inserting the followingt1N^i ðsÞ expfiu^i ðsÞ½ui ðsÞ 
-function:
1¼s¼0
i¼1 dui ðsÞ duPN
j¼1 Jij j ðsÞ  i ðsÞg. Averaging the generating functional
over the disorder fi g, we separate the relevant
PN one-time andtwo-timeorderparameters:
mðsÞ¼
ð1=NÞ
i¼1 i ðsÞ, kðsÞ ¼
PP00
ð1=NÞ PNi¼1 u^i ðsÞ, qðs; s0 Þ ¼ ð1=NÞ Ni¼1 i ðsÞPi ðs Þ, Qðs; s Þ ¼N
N00
ð1=NÞ i¼1 u^i ðsÞu^i ðs Þ, and Kðs; s Þ ¼ ð1=NÞ i¼1 i ðsÞu^i ðs0 Þ.
We then obtain the disorder-averaged generating functionalZZ ½  ¼ dm dm^ dk dk^ dq dq^ dQ dQ^ dK dK^ eNðþþÞ ; ð8Þ
where¼it1X½m^ ðsÞmðsÞ þ k^ðsÞkðsÞ  mðsÞkðsÞþi

s¼0 s0 ¼00^

Rt½q^ðs; s0 Þqðs; s0 Þ þ Q^ ðs; s0 ÞQðs; s0 Þþ K ðs; s ÞKðs; s0 Þ;#Z "YZ
Nt1X
¼lndðsÞ p½ð0Þ h h^N i¼1
Rtþ1 s¼0()
t1Y½ðs þ 1Þ  f ðhðsÞÞ
s¼0h^ ¼t1 ^YdhðsÞ
pﬃﬃﬃﬃﬃﬃ ;2s¼0

The disorder-averaged generating functional is for N !
1 dominated by a saddle-point. Applying the saddle-point
method, we arrive at the following result. Some macroscopic
variables are found to vanish at the relevant physical saddlepoint. The remaining variables can all be expressed in termsof three macroscopic observables, namely, the overlaps mðsÞ,the single-site correlation functions Cðs; s0 Þ and the singlesite response functions Gðs; s0 Þ. We then arrive at thefollowing saddle-point equations:mðsÞ ¼ hhðsÞii;Cðs; s0 Þ ¼ hhðsÞðs0 Þii;
ð12Þ 1=2
100hhðsÞðR vÞðs Þii; for s > sGðs; s0 Þ ¼0;
for s  s0 .
The average over the eﬀective path measure is given by#ZZ "Ythhgð; vÞii DvdðsÞ gð; vÞs¼0t1 Xt1
X

ð11Þ

with
h ¼

2

ð10Þ

s¼0

Rtþ1

 p½ð0Þt1
Y

s¼0

½ðs þ 1Þ  f ðuðsÞÞ;

ð13Þ

s¼0

ð9Þ

with
uðsÞ ¼ mðsÞ þ ðsÞ þpﬃﬃﬃvðsÞ þ ðÞðsÞ;R ¼ ð1  GÞ1 Cð1  y GÞ1 ;

ð14Þ

1

 ¼ ð1  GÞ G;
1and Dv  j2 Rj1=2 dv eð1=2ÞvR v . Here, C, G, R and  arematrices having matrix elements Cðs; s0 Þ, Gðs; s0 Þ, Rðs; s0 Þ,033001-2

LETTERSand ðs; s0 Þ, respectively.  is the vector y ðð0Þ; . . . ; ðtÞÞ.
The terms ðR1 vÞðsÞ and ðÞðsÞ denote the sth element of
the vectors R1 v and , respectively. The matrix 1 denotes
a unit matrix. Equations (12)–(14) completely describe thedynamics of this continuous nonlinear system. In the limit
where t ! 1, the term ðÞðsÞ in eq. (14) can be regardedas the Onsager reaction term in equilibrium statisticalmechanics.
In most applications in the ﬁeld of information theory, theinitial state is usually given by a discrete distribution having
the form
KX
p½ð0Þ ¼ck ðð0Þ  k Þ;
ð15Þ

K. M IMURA

1.2
1
overlap m(t)J. Phys. Soc. Jpn., Vol. 78, No. 3

0.8
0.6
0.4

f(x)
GFA
simulation

0.2

x

00

12

k¼1

P
where ck 0 ð8kÞ, Kk¼1 ck ¼ 1 and k 2 R ð8kÞ. When theinitial probability is given by the form of eq. (15), the spin
integral included in eq. (13) becomes a simple summationand the average over the eﬀective path measure can besimpliﬁed toZKXhhgð; vÞii ¼
Dv
ckRt
k¼1

 gð; vÞ:
ð16Þ

45Fig. 1. The ﬁrst few time steps of the dynamics of the binary Hopﬁeld
model predicted using the GFA with  ¼ 0:12 (circles and solid lines)and  ¼ 0:20 (circles and dashed lines). The transfer function isf ðxÞ ¼ sgnðxÞ. Computer simulations (crosses) were performed with
N ¼ 10;000 from 10 experiments.

1overlap m(t)

f(x)

GFA
simulation

0.8

ð0Þ¼ k ;ð1Þ¼f ðuð0ÞÞ;...;ðtÞ¼f ðuðt1ÞÞ

The spin integral in eq. (13) can be performed in this system,
therefore the eﬀective single-spin dynamics ðs þ 1Þ ¼
f ðuðsÞÞ appears in eq. (16). The initial state is chosenstochastically from the initial distribution of eq. (15), but
dynamics is deterministic. The deterministic path of theinitial state ð0Þ ¼ k appears with a probability ck . The
average over the eﬀective path measure can be calculated
from K deterministic paths.
To validate the GFA, we performed computer simulations. We considered a case with the initial distribution
p½ð0Þ ¼ ð1=2Þ½1 þ mð0Þ½ð0Þ  1 þ ð1=2Þ½1 
mð0Þ½ð0Þ þ 1 and ðsÞ ¼ 0. The ﬁrst few time steps ofthe dynamics of the binary Hopﬁeld model, which has thetransfer function f ðxÞ ¼ sgnðxÞ, predicted using the GFA
with  ¼ 0:12 and 0.20 are shown in Fig. 1. The computer
simulations were performed with N ¼ 10;000 from 10experiments. Excellent agreement was found between thepredictions of the GFA and the computer simulation results.
The ﬁrst few time steps of the dynamics of the continuous
Hopﬁeld model predicted using the GFA with  ¼ 0:12 and0.20 are shown in Fig. 2. The transfer function is f ðxÞ ¼
 tanh½ ðx þ kÞ tanhð xÞ tanh½ ðx  kÞ with
¼ 5 andk ¼ 1. Note that the parameter
is not the inverse
temperature but an arbitrary parameter. In this case, computer simulations were also performed with N ¼ 10;000
from 10 experiments. When the transfer function f iscontinuous, the GFA exhibited excellent consistency withthe computer simulation results.
The relationship between the GFA and the SND is asfollows. The SND analysis ignores the Onsager reaction
term.26) In the binary-state parallel-update sequential processing Hopﬁeld model, which does not have the Onsager
reaction term, Kawamura and Okada showed that the GFA
result was identical to the SND result.27) According to their
discussion, if we neglect the Onsager reaction term, we canshow that GFA result is equivalent to the SND result. Whenwe neglect the Onsager reaction term in the GFA result of

3time t

x0.6
0.4
0.2
0-0.2
01

2345time t
Fig. 2. The ﬁrst few time steps of the dynamics of the continuous
Hopﬁeld model predicted using the GFA with  ¼ 0:12 (circles and solidlines) and  ¼ 0:20 (circles and dashed lines). The transfer function isf ðxÞ ¼  tanh½ ðx þ kÞ tanhð xÞ tanh½ ðx  kÞ with ¼ 5 and k ¼ 1.
Computer simulations (crosses) were performed with N ¼ 10;000 from10 experiments.

the continuous Hopﬁeld model, which has the Onsager
reaction term, the response function becomesZZ@Dvðs1ÞdðsÞ ðsÞ½ðsÞGðs; s0 Þ ¼@ðs0 Þ R
Rpﬃﬃﬃ
 f ðmðs  1Þ þ ðs  1Þ þ vðs  1ÞÞ; ð17Þ
whereDvðs  1Þ  ½2 Rðs  1; s  1Þ1=2
1 vðs  1Þ2
 exp :2 Rðs  1; s  1ÞThis means that the response function becomes zero except
for Gðs; s  1Þ. Therefore, the ðs; s0 Þth elementof thematrix Gn has the simple form Gn ðs; s0 Þ ¼Qn10s;s þn ¼0 Gðs  ; s   1Þ. When the Onsager reaction
term is neglected, eq. (14) can be represented in therecurrence relation form
Rðs; s0 Þ ¼ Cðs; s0 Þ þ Gðs; s  1ÞGðs0 ; s0  1ÞRðs  1; s0  1Þ033001-3

LETTERS

J. Phys. Soc. Jpn., Vol. 78, No. 3

þ

þ

s1X

Cð ; s0 ÞsX¼0¼ þ1s0X

Cðs; Þ¼0(B) No. 18700230 from the Ministry of Education, Culture,Sports, Science and Technology of Japan.

Gð ;  1Þ

s0 1XGð ;  1Þ;¼ þ1
1K. M IMURA

ð18Þ

Pt

by using the identity ð1  GÞ ¼ 1 þ n¼1 Gn . The orderparameters becomeZmðsÞ ¼ Dz f ðu~ðs  1ÞÞ;Z0Cðs; s Þ ¼ Du Dv Dz f ðuðs  1; s0  1ÞÞ f ðuðs0  1; s  1ÞÞ;
ð19Þ
Z1
Gðs; s  1Þ ¼ pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ Dz z f ðu~ðs  1ÞÞ;Rðs  1; s  1Þpﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
whereu~ðsÞ  mðsÞ þ z pRðs;sÞ and uðs; s0 Þ  mðsÞ þpﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
0u ½Rðs; sÞ  Rðs; s Þ þ z Rðs; s0 Þ. This result obtainedusing eqs. (18) and (19) coincides with the SND result.26)
Namely, the SND result can be derived from the GFA result
by neglecting the Onsager reaction term.
In summary, we applied the generating functional analysis
to the continuous Hopﬁeld model. The predictions of theGFA are in excellent agreement with computer simulation
results for any conditions and transfer function. When theretarded self-interaction term is omitted, the GFA result
becomes identical to the SND result. The GFA can also beapplied to any continuous nonlinear system having the formof eq. (2) which includes some belief-propagation-basedalgorithms, e.g., multiuser detection problems in mobile
communication systems.
Acknowledgment
This work was partially supported by a Grant-in-Aid for
Scientiﬁc Research for Encouragement of Young Scientists

1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
12)
13)
14)
15)
16)
17)
18)
19)
20)
21)
22)
23)
24)
25)
26)
27)
28)
29)
30)

033001-4

T. Tanaka: Europhys. Lett. 54 (2001) 540.
T. Tanaka and M. Okada: IEEE Trans. Inf. Theory 51 (2005) 700.
Y. Kabashima: J. Phys. A 36 (2003) 11111.
K. Mimura and M. Okada: J. Phys. A 38 (2005) 9917.
N. Sourlas: Nature 339 (1989) 693.
Y. Kabashima and D. Saad: Europhys. Lett. 44 (1998) 668.
Y. Kabashima, T. Murayama, and D. Saad: Phys. Rev. Lett. 84 (2000)
1355.
Y. Kabashima and D. Saad: J. Phys. A 37 (2004) R1.
H. Nishimori and K. Y. M. Wong: Phys. Rev. E 60 (1999) 132.
A. Montanari and N. Sourlas: Eur. Phys. J. B 18 (2000) 107.
T. Murayama and M. Okada: J. Phys. A 36 (2003) 11123.
T. Murayama: Phys. Rev. E 69 (2004) 035105.
T. Hosaka, Y. Kabashima, and H. Nishimori: Phys. Rev. E 66 (2002)
066126.
T. Hosaka and Y. Kabashima: Physica A 365 (2006) 113.
K. Mimura and M. Okada: Phys. Rev. E 74 (2006) 026108.
F. Cousseau, K. Mimura, and M. Okada: Phys. Rev. E 78 (2008)
021124.
K. Tanaka: J. Phys. A 35 (2002) R81.
J. J. Hopﬁeld: Proc. Natl. Acad. Sci. U.S.A. 79 (1982) 2554.
A. C. C. Coolen: cont-mat/0006011.
A. Crisanti and H. Sompolinsky: Phys. Rev. A 36 (1987) 4922.
A. Crisanti and H. Sompolinsky: Phys. Rev. A 37 (1988) 4865.
H. Rieger, M. Schreckenberg, and J. Zittartz: Z. Phys. B 72 (1988)
523.
E. Gardner, B. Derrida, and P. Mottishaw: J. Phys. (Paris) 48 (1987)
741.
M. Kawamura, R. Tokunaga, and M. Okada: Europhys. Lett. 62 (2003)
657.
M. Kawamura, R. Tokunaga, and M. Okada: Phys. Rev. E 70 (2004)
046210.
M. Okada: Neural Networks 8 (1995) 833.
M. Kawamura and M. Okada: J. Phys. A 35 (2002) 253.
D. Bolle, J. B. Blanco, and T. Verbeiren: J. Phys. A 37 (2004) 1951.
K. Hashiguchi and M. Kawamura: Denshi Joho Tsushin Gakkai
Ronbunshi D 89 (2006) 2123 [in Japanese].
T. Galla: J. Stat. Mech. (2005) P01002.