Deep Learning and Physics 2019@YITP

From Complex to Simple:
Hierarchical Free-energy LandscapeRenormalized in Deep Neural Networks
Cybermedia Center & Department of Physics,
Osaka Univ.

Hajime Yoshino
H. Yoshino, arXiv1910.09918
submitted to SciPost Phys
referee round deadline Nov. 26 th… comments arewelcome!

[Q] Why deep neural networks
works?
e.g.

data size
610⌧# of parameters
810

G. Carleo, et. al, arXiv:1903.10563v1

Common sense
over ﬁtting
1. poor generalization

Empirical observations
on deep networks
1. generalization is not bad

Why???
2. many local minimum
learning is diﬃcult(glassy dynamics)

2. learning is not too diﬃcult

Why???

History: Spinglasses, Hopﬁeld model, error
correcting codes,....
• Edwards-Anderson model(1975)

H=？

Jij si sj

Jij > 0Jij < 0<i,j>si = ±1(i = 1, 2, . . . , N )Quenched random spin-spin interactions

Jij = 02
2
Jij = J

Ground state: disordered？
A lot of energetic degeneracies due to frustration• Hopfield model (1982) : associative memorysi = ±1

Firing of neurons

Jij

Hebb rule

synaptic weight1
Jij = pM µ=1embedded patterns• error correcting code： a statistical inference problem
send Jij

receiver tries to reconstruct siMX⇠iµ ⇠jµµ⇠i = ±1

Sourlas (1989)

3. Other soft materials

;

Now that I’ve introduced colloids, let’s discus
varying degrees.

1987

1991

1999

2001

2008
āğĪĊĦĭĲĳšũŨűůťūŪŅ
3.1 Emulsions

Emulsions are similar to colloids, but rather thdroplets of one liquid, mixed into a second im
water. Surfactant
molecules are necessary tohé®ëêÃĲGÝùDüë
when two droplets come together and form aĠĎĪý,kÞç
shown in Fig. 3.1, anda sketch showing a
Mayonnaiseisacommonexample of an emul
Ĳ ¦× éëäç{¦ì¡egg yolks as the surfactant, with extra ingredie

From spin glass to structural glass

10 µm

p-spin to RFOT

Kirkpatrick-Thirumalai Wolyness (1989)

more on replicas

Franz-Parisi (1995), Monasson (1995), Mezard-Parisi (1999)

d!1

ĕēĂ
Fig. 3.1
ĳťŪĲhemulsioa trans
droplets
glycerorefracti
dropletssurfacta
surfactabottoms
ER We

ŌĸĲŗĸ
ŊĸĲŏ

f(èîĸĴĲğħėÇčÃ¬=³ĝ

Parisi-Zamponi (2010), Charbonneau-Kurchan-Parisi-Urbani-Zamponi (2014)<latexit sha1_base64="6R4WA9A0JTWCWI9zsDIx8f7WTbo=">AAACcXichVHNSiNBEP4y6290Na4XxcuQoAgLoUZF1z3JevHoz0YFI2Fm7GjjZHqY6YTNBl/AB9DDHvwBQfExvOwL7CGPIB6z4MWDlUlEliVaDV1VX9VX/XW3E3gy0kT1hPGhq7unt68/OTD4cWg4NfJpM1Ll0BU5V3kq3HbsSHjSFzkttSe2g1DYJccTW87hcrO+VRFhJJX/XVcDsVuy931ZlK6tGcrvmXmtzLz0i7paSGUou0jW4rxl/h9YWYotg7atqtQ18tiDgosyShDwoTn2YCPitQMLhICxXdQYCzmScV3gCEnmlrlLcIfN6CHv+5zttFGf8+bMKGb/4A7FTMVTjmBikv7QDTXoN93SPT11nFaLpzTVVNk7La4ICsPHYxuP77JK7DUOXllvqtYo4kusVrL6IEaa93Bb/MrP08bG1/XJ2hRd0gPrv6A63fEN/Mpf92pNrP9Ckr/g5Z3NzsHmTNaazc6szWWWvrU/ow8TSGOaX3wBS1jBKnJ8boATnOE80TDGDdNIt1qNRJszin/M+PwMVVCPLA==</latexit>»éCj·íeúĲ;íëÒ
Rainone-Urbani-Yoshinno-Zamponi(2015)
èî[çÛë

Yoshino-Mezard (2010), Yoshino-Zamoponi (2014),
Jin-Yoshino (2017), Jin-Urbani-Zamponi-Yoshino (2018)şŢĸĲŊĸĲŘĸĲŕÈŏšŮŪĲ

S3… and back to p-spins… but with M ! 1 spin components
and without quenched disorder
S0

S2

S1

S2

S0

<latexit sha1_base64="f6fwXjV2lpbD35eA2wSGmDwXznU=">AAACcXichVHNSiNBEP4yrj+b9Se6F2Uvg0ERhFCjosaT6MWL4M9GBSMyM3Zi42R6mOkEY8gL7APoYQ/uCoLLPoYXX8CDjyB7zMJe9mBlEhERtRq6qr6qr/rrbifwZKSJ7hJGx4fOru6ej8lPvX39A6nBoa1IlUNX5FzlqXDHsSPhSV/ktNSe2AlCYZccT2w7R8vN+nZFhJFU/lddDcReyS76siBdWzOUXzXzWpl56Rd0dT+VpkyWrOysZb4MrAzFlkbb1lTqCnkcQMFFGSUI+NAce7AR8dqFBULA2B5qjIUcybguUEeSuWXuEtxhM3rEe5Gz3Tbqc96cGcXsY+5QzFQ8pQ4TY3RLv6hBN/Sb7un/q9Nq8ZSmmip7p8UVwf7At+HNf++ySuw1Dp9Yb6rWKGA+VitZfRAjzXu4LX7l5KyxubAxVhunC/rD+n/SHV3zDfzKX/dyXWx8R5K/4PGdzdeDramMNZ2ZWp9JLy61P6MHXzCKCX7xOSxiBWvI8bkBTnGOH4mGMWKYxmir1Ui0OZ/xzIzJByYljxU=</latexit>

H. Yoshino, SciPost Phys. 4 (6), 040 (2018)

S4

S0

S1

S3

Glass order parameter with replicas
symmetry breaking ﬁeld

replicas a=1,2,…n
replica 1

replica 2

H[✏] =nX

H(sa )

a=1

X

✏ab

a,b

N
X

Sia Sibi=1

F [ˆ✏] = ln TrS e
1 X D a bE
Qab ⌘Si Si =N i=1
N

Overlap
crystal

We are interested with
G[Q̂] = F [ˆ✏] + NX✏ab Qabab@G[Q̂]
✏ab =
@Qab

=0
Q̂=Q̂SPlim

lim Qabliquidglass

Qab = 0H[ˆ✏]

1 @F [ˆ✏]N @✏ab✏ab !0+ N !1

Qab > 0spontaneous breaking of ergodicity

Explicit RSB : Parisi-Virasoro (1989)

Replica symmetry braking and ultra-metricity
breaking of ergodicity & permutation symmetry
overlap matrix

distance

Qab =G. Parisi (1979)abﬁrst found in the
SK model for spinglasscoverlap

ultra-metricity Q(a, b) = min(Q(a, c), Q(b, c))qEA = lim Qab

Self-overlap

b!aEdwards-Anderson (EA) order parameter<latexit sha1_base64="GZjFwLgB+JxI2TeI0dcstFSRoKc=">AAACh3ichVFNTxRBEH0MoLCiLHgx8TJxg/G01LJmARMTwJh4ZIEFEpZMZoYGOsxMDzO9G3EyVw/+AQ6eJCHR4FX/gBf/gAd+gvEIiRcP1nwkxBilOul+/ape9etuJ/RkrInOB4zBoeEbN0dGK7fGbt8Zr05MrseqF7mi4ypPRZuOHQtPBqKjpfbEZhgJ23c8seEcPMvyG30RxVIFa/ooFNu+vRfIXenamimrah5aSTfyzeeL6dOuJ30rccyuVqadmm0rsZ3UqtaoPt+iBs2YBWi1StBsmo065VFDGcuq+h5d7EDBRQ8+BAJoxh5sxDy20AAhZG4bCXMRI5nnBVJUWNvjKsEVNrMHPO/xbqtkA95nPeNc/ZIrFCsVd0lhYoq+0Qe6oK90Rt/p1z+7JXmXzM0Rr06hFaE1/ube6s9rVT6vGvtXqv+61tjFXO5WsvswZ7J7uIW+/+r4YvXJylTykE7oB/t/R+f0hW8Q9C/d07ZYeYtK8QVZXD3432B9pt5o1pvtx7WFpfIzRnAfD/CIX3wWC3iBZXT43Nf4iE/4bIwa00bLmCtKjYFScxd/hLH4G9UUlos=</latexit>Rammal-Toulouse-Virassoro (1986)
Toulouse-Dehaene-Changeaux (1986)

“Disorder-free” vector p-spin models on tree
S1
S
3
H.Yoshino, SciPost Phys. 4 (6), 040 (2018)S
2

1
2 S
M
Si = (Si , Si , . . .0, Si )

S0

p=3
c=4

S0

|Si 2 | = M

i = 1, 2, . . . , N

continuous or Ising

S4

Hamiltonian

Siµ = ±1

H=

a)

“gap”

S2

r⌅ =

# of factor nodes

XS1
V (r⌅ ) .
⌅S3
Locally tree like latticewith connectivity cb)c = ↵M

M
X

1
µ
µ
µ
p
S1(⌅) S2(⌅) · · · Sp(⌅)
M µ=1

N⌅ = N c/p = N M ↵/p < N p /p!

“Inter-mediate sparseness “: high connectivity but not “global coupling”

and the Hamiltonian

s}) =

!

“Vectorial” constraint satisfaction problems

δ(si , sj ) .

(i,j)∈E(1)

gy contribution for neighbors with different colors,e.
The ground state energy (the energy at zeroly if the graph is q-colorable. This transforms theatistical physics model. Usually, two types of random
gular ensemble all points are connected to exactly
yi case the connectivity has a Poisson distribution.

standard discrete coloring

efs and Surveys
r of studies haveion of the so-called
9]. It is a powerfuldely accepted but— equivalent to the[2]. Its main idea
aph is locally treenown in physics as
ctly any model ond as “mean field” Figure 1. Example: a proper 3lized [10] that an coloring of a small graph.
d independently inBelief Propagationy also stands for) as the probability that the spin i has color c in absencen j has on the properties of the spin i), BP reads→jantiferromagnetic Potts model"k∈N (i)\{j}#

1 − ψck→i

$

H=

X
i,j

(2)

qi ,qjnt and the notation k ∈ N (i) \ {j} means the set ofoint of these equations, the complete beliefs in presence
y give, for each vertex, the probability of each color fromthe number of solutions, can be computed. A simpler(WP), restricts itself to frozen variables (i.e. to variables
e constraints). However, WP does not allow to computetence, but is definitely simpler to handle.
hat these methods developed for trees could not bem graphs because of a non-trivial phenomenon called“continuous” version

θ“repulsive” vectorial spin model

H=

X
i,j

V

✓

Si · Sjp
M◆

V (r) = lim ✏r2 ✓( r)
✏!1

M !1
<latexit sha1_base64="f6fwXjV2lpbD35eA2wSGmDwXznU=">AAACcXichVHNSiNBEP4yrj+b9Se6F2Uvg0ERhFCjosaT6MWL4M9GBSMyM3Zi42R6mOkEY8gL7APoYQ/uCoLLPoYXX8CDjyB7zMJe9mBlEhERtRq6qr6qr/rrbifwZKSJ7hJGx4fOru6ej8lPvX39A6nBoa1IlUNX5FzlqXDHsSPhSV/ktNSe2AlCYZccT2w7R8vN+nZFhJFU/lddDcReyS76siBdWzOUXzXzWpl56Rd0dT+VpkyWrOysZb4MrAzFlkbb1lTqCnkcQMFFGSUI+NAce7AR8dqFBULA2B5qjIUcybguUEeSuWXuEtxhM3rEe5Gz3Tbqc96cGcXsY+5QzFQ8pQ4TY3RLv6hBN/Sb7un/q9Nq8ZSmmip7p8UVwf7At+HNf++ySuw1Dp9Yb6rWKGA+VitZfRAjzXu4LX7l5KyxubAxVhunC/rD+n/SHV3zDfzKX/dyXWx8R5K/4PGdzdeDramMNZ2ZWp9JLy61P6MHXzCKCX7xOSxiBWvI8bkBTnGOH4mGMWKYxmir1Ui0OZ/xzIzJByYljxU=</latexit>

H. Yoshino, SciPost Phys. 4 (6), 040 (2018)

continuous RSB:
hierarchical clustering of solutions
SciPost Phys. 4, 040 (2018)

connectivity

g

(S

12

m

in

10

0.8
0.7
0.6

q=0

glass

0.4

0.1

2

Liquid

q=0

0
0

10-3

0.2

0.4

1

10-5

q(x) = x 
 = 1.415726...

0
0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x
-0.2

10-2

10-4

0
4

-0.4

10-1

0.5

0.2

Ja

6

6.7
6.71
6.72
6.722
6.724
6.726
power law

0.3

m

8

100

0.9

q(x)

14

b)

1

a)

1

AT
-U

↵ = c/M

q(x)

N
SA
T)

SciPost Phys. 4, 040 (2018)

10-4

10-3

10-2

x

10-1

100

Figure 9:
The q(x) function of the hardcore model with p = 2,=
0 for which ↵c=
1.5708.. and ↵j=6.732...
a)Same jamming
criticality↵ = 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 4.0, 5.0, 6.0, 6.5, 6.6, 6.7 from the bottom
hard
spheresto the top. b) The straight line represents the power lawas
fit ax  with = 1.4157,
andperceptron(p=1)
the same exponent as that for the hardspheres [16].

Franz-Parisi(2016),
at T = 0 (hardcore limit). We obtained the result using thescheme explained
in sec. 8.4.3
The phase diagram of the p = 2 body hardcore model within thereplicaFranz-Parisi-Sevlevtogether with the inputs for the hardcore case shown in sec.
10.2.1. As can be seen in Fig. 9a), wecurve
found the continuous RSB solution with nonzero Urbani-Zamponi
q(x) function emerges(2017)
continuouslyc (RS) ansatz: q > 0 RS solution emerges continuously at the lower
starting from the AT line ↵ = ↵c ( ) as expected.
resents ↵ = ↵c ( ) given by Eq. (235). The value of the order parameter
Using the scheme explained in sec. 10.2.4 we obtained the jamming line ↵ = ↵j ( ) of theq ! 1 approaching the upper solid line ↵ = ↵j ( ) given by Eq. (237),
whichhardcore model and the result is displayed in Fig. 8 at the bottom. It is also shown in Fig. 6
ming line within the RS ansatz. The lower curve coincides with the
AT we
line
where
can see that the RS ansatz overestimates the jamming line. Quite interestingly weclustering = glass transition

“Design space” of a multilayer-perceptron network“Esemble of random perceptrons”

Shunʼichi Amari (1971)

LS0,1<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit>

S2,1

S1,1

S3,1

<latexit sha1_base64="(null)">(null)</latexit>

S0,2<latexit sha1_base64="(null)">(null)</latexit>

S1,2<latexit sha1_base64="(null)">(null)</latexit>

S0,3

N

<latexit sha1_base64="(null)">(null)</latexit>

S0,N
input

S1,N
hiddenstate of neurons with respect to M diﬀerent patterns
1
2MSi,l = (Si,l , Si,l , . . . , Si,l )<latexit sha1_base64="IQ1ZI7jiQ/Wy8+UMKWaYvXyVZfA=">AAACvnichVFNS9xAGH5Mq9WtH9v2IvQSulgUlmWyFloKwmIvvRTUdV3B6JJkZ9fBJBOTyaIN+QP+gR56aqHQ0t/QUy/9Az149iS2Jwu99NB3s0FRUd+BmWee933eeWbGDlwRKcYOh7Q7d4dH7o2OFe6PT0xOFR88XItkHDq84UhXhuu2FXFX+LyhhHL5ehByy7Nd3rR3XvXzzR4PIyH9VbUf8E3P6vqiIxxLEdUqNhPT7uj1tJWIspsuzNYHYCsx0vIZrqZl021LFZ1Tb9I5XdfN3d3YautnrOnF6YIZeLrRKpZYhWWhXwVGDkrIY0kWP8NEGxIOYnjg8KEIu7AQ0diAAYaAuE0kxIWERJbnSFEgbUxVnCosYndo7tJuI2d92vd7Rpl6jyokKSV1SaFjhv1kX9gp+8G+smP279puSdal72afVnug5UFr6mC6/vdWlUerwva56kbXCh28yNwKch9kTP8ezkDfe/vutP5yZSZ5yj6yE/L/gR2y73QDv/fH+bTMV96jQF9gXH7wq2CtWjHmK9XlZ6XaYv4Zo3iMJ5ilF3+OGl5jCQ069xuO8Au/tZrW0TxNDkq1oVzzCBdC2/sPJVqrwQ==</latexit>µSi,l = ±1

outputN, M ! 1
M
↵=
N

S1

J1

<latexit sha1_base64="QoVpjNLS1LOP88I7//d20UgzJjY=">AAACaHichVFNLwNBGH66vuujiwPi0mgqTs1sSYiTcHGktSpBmt01mHS7u9ndNmHjD7g4Ik4kIuJnuPgDDn4CjpW4OHh3u4nQ4J3MzDPPvM87z8zojik8n7GnhNTW3tHZ1d2T7O3rH0jJg0Prnl1zDa4atmm7G7rmcVNYXPWFb/INx+VaVTd5Sa8shfulOnc9YVtr/oHDt6vaniV2haH5RKnFcqAcleUMy7Eo0q1AiUEGcazY8g22sAMbBmqogsOCT9iEBo/aJhQwOMRtIyDOJSSifY4jJElboyxOGRqxFRr3aLUZsxatw5pepDboFJO6S8o0suyR3bIGe2B37Jl9/ForiGqEXg5o1pta7pRTx6PF939VVZp97H+p/vTsYxdzkVdB3p2ICW9hNPX1w9NGcb6QDSbZFXsl/5fsid3TDaz6m3G9ygsXSNIHKD+fuxWs53PKdC6/OpNZWIy/ohvjmMAUvfcsFrCMFah0rsAJznCeeJFkaUQaa6ZKiVgzjG8hTXwCVvKLhw==</latexit>

Perceptron（McCulloch-Pitts model)<latexit sha1_base64="zr/nSDwjQTQFp4csWjiF3RnDh0c=">AAACaHichVHLLgRBFD3T3uMxjQViIzMZsZpUIyFWwkasZjBGwmTS3cqo6Fe6eyah4wdsLBErEhHxGTZ+wMInYEliY+F2TwsiuJWqunXuPbdO1dUcQ3g+Y/cJqam5pbWtvSPZ2dXdk5J7+1Y9u+bqvKjbhu2uaarHDWHxoi98g685LldNzeAlbWc+jJfq3PWEba34uw4vm2rVEltCV32CiouVQNmvyBklxyIbYT+cj1AGseVt+RIb2IQNHTWY4LDgk29AhUdjHQoYHMLKCAhzyRNRnGMfSeLWKItThkroDq1VOq3HqEXnsKYXsXW6xaDpEnMEWXbHrtgzu2XX7IG9/VoriGqEWnZp1xpc7lRSB4PLr/+yTNp9bH+y/tTsYwvTkVZB2p0ICV+hN/j1vaPn5ZmlbDDKztkT6T9j9+yGXmDVX/SLAl86RfJrA353VsdzykRuvDCZmZ2LW9GOYaQxRv89hVksII8i3StwiGOcJB4lWRqQhhqpUiLm9OObSel3RsqLfw==</latexit>

S2

J2

S0

<latexit sha1_base64="be9vAOb/HSLVDCUOLTsC6symlCs=">AAACaHichVE9SwNBEH05v+NHohYqNiEhYhU2UVCsgjZipYmXBGIId+cal1zujrtLQEP+gI2lipWCiPgzbPwDFv4EtVSwsXDuElERdZbdnX0zb/btjmrpwnEZuw9IXd09vX39A8HBoeGRUHh0LOeYdVvjsmbqpl1QFYfrwuCyK1ydFyybKzVV53m1uuLF8w1uO8I0Nt09i5dqSsUQO0JTXILktXIz1SqHY8kE8y3CfjgfoRg6tm6GL7GFbZjQUEcNHAZc8nUocGgUkQSDRVgJTcJs8oQf52ghSNw6ZXHKUAit0lqhU7GDGnT2ajo+W6NbdJo2MSOIszt2xZ7ZLbtmD+zt11pNv4anZY92tc3lVjl0MJl9/ZdVo93F7ifrT80udrDoaxWk3fIR7xVam9/YP3rOLmXizRl2zp5I/xm7Zzf0AqPxol1s8Mwpgl8b8LuTSyWSc4nUxnwsvdxpRT+mEcUs/fcC0ljFOmS6V+AQxzgJPEphaUKaaqdKgQ5nHN9Mir4DSMuLgA==</latexit><latexit sha1_base64="msQsOKxDYussk++CeQfiXTU+cko=">AAACaHichVFNLwNBGH66vqo+ujggLqIhTs1sSYhTw8WxVUWCNLtrWpPuV3a3TWrTP+DiiDiRiIif4eIPOPgJOJK4OHi73UQQvJOZeeaZ93nnmRnNMYTnM/YQkzo6u7p74r2Jvv6BwaQ8NLzh2TVX50XdNmx3S1M9bgiLF33hG3zLcblqagbf1Korrf3NOnc9YVvrfsPhu6ZasURZ6KpPVLFQCjLNkpxiaRbG5E+gRCCFKHK2fIUd7MGGjhpMcFjwCRtQ4VHbhgIGh7hdBMS5hES4z9FEgrQ1yuKUoRJbpbFCq+2ItWjdqumFap1OMai7pJzENLtn1+yF3bEb9sjef60VhDVaXho0a20td0rJw7HC278qk2Yf+5+qPz37KGMx9CrIuxMyrVvobX394PilsLQ2HcywC/ZM/s/ZA7ulG1j1V/0yz9fOkKAPUL4/90+wkUkrc+lMfj6VXY6+Io4JTGGW3nsBWawihyKdK3CEE5zGniRZGpXG26lSLNKM4EtIUx9Y84uI</latexit><latexit sha1_base64="CpYN3TX7zie2aju2flZiK+fYnXI=">AAACaHichVFNLwNBGH66vuujxQFxaTQVp+YtEuLUcHFEVSXVNLtrWhPb3c3utglN/4CLI+JEIiJ+hos/4OAn4Eji4uDtdhOhwTuZmWeeeZ93npnRbEO6HtFjSOno7Oru6e0L9w8MDkWiwyNbrlV1dJHVLcNytjXVFYY0RdaTniG2bUeoFc0QOW1/pbmfqwnHlZa56R3YolBRy6YsSV31mMpminVqFKNxSpIfsXaQCkAcQaxZ0WvsYBcWdFRRgYAJj7EBFS63PFIg2MwVUGfOYST9fYEGwqytcpbgDJXZfR7LvMoHrMnrZk3XV+t8isHdYWUMCXqgG3qle7qlJ/r4tVbdr9H0csCz1tIKuxg5Gs+8/6uq8Oxh70v1p2cPJSz6XiV7t32meQu9pa8dnrxmljYS9Wm6pBf2f0GPdMc3MGtv+tW62DhHmD8g9fO528HWbDI1l5xdn4+nl4Ov6MUkpjDD772ANFaxhiyfK3GMU5yFnpWoMqZMtFKVUKAZxbdQpj4BVPGLhg==</latexit>

S0 (t + 1) = sgn

JN<latexit sha1_base64="2O2++RHzQ/Co0/j53Yd1tJ6OqD0=">AAACx3ichVHLahRBFD1pX3F8ZNSN4KZxiMwgDNVRUIRA0I26kCTjJIF0bLo7NZMi/bKqZkgsauHWH3DhSkFE/Aw3+QEX+QQR3ESICxfe7mkQDeotqurec++5daoqKhKhNGP7U86x4ydOnpo+3Thz9tz5meaFiysqH8mY9+M8yeVaFCqeiIz3tdAJXyskD9Mo4avR9r0yvzrmUok8e6x3C76RhsNMDEQcaoKCZtQLDLNtfd3ruPPGl6mrhpl1/YQPdNsfyDA2njW+eiq1eWStr0ZpYMS8Z59Q6Pqa7+hKhJF805qHlLO2V65t3fGlGG7pTtBssS6rzD3qeLXTQm2LefMdfGwiR4wRUnBk0OQnCKForMMDQ0HYBgxhkjxR5TksGsQdURWnipDQbVqHFK3XaEZx2VNV7JhOSWhKYrqYZZ/Ye3bA9tgH9pn9+GsvU/UotezSHk24vAhmXlzuHf6XldKusfWL9U/NGgPcrrQK0l5USHmLeMIfP3t50LuzPGuusTfsC+l/zfbZR7pBNv4Wv13iy6/QoA/w/nzuo87KXNe70Z1butlauFt/xTSu4Cra9N63sID7WESfzt3DVxziu/PAyZ2xszMpdaZqziX8Zs7zn1BHsdo=</latexit>

N
X

1
p
Ji Si (t)N i=1<latexit sha1_base64="VrJZuhLJvL//qUjHzqDOU0SeTNg=">AAACaHichVFNLwNRFD0d3/XRwQKxaTTEqnktCbFq2IiFtGhJkGZmvNZL5ysz0yY0/QM2logViYj4GTb+gIWfgCWJjYU70xFEcF/ee/ede899572r2rpwPcbuI1JLa1t7R2dXtLunty8m9w8UXKvqaDyvWbrlbKiKy3Vh8rwnPJ1v2A5XDFXn62plwY+v17jjCstc8/Zsvm0oZVOUhKZ4BOWXivXlRlFOpJIssDj74XyEEggta8mX2MIOLGiowgCHCY98HQpcGptIgcEmbBt1whzyRBDnaCBK3CplccpQCK3QWqbTZoiadPZrugFbo1t0mg4x4xhnd+yKPbNbds0e2NuvtepBDV/LHu1qk8vtYuxgePX1X5ZBu4fdT9afmj2UMBtoFaTdDhD/FVqTX9s/el6dWxmvT7Bz9kT6z9g9u6EXmLUX7SLHV04R/dqA351COpmaSqZz04nMfNiKToxiDJP03zPIYBFZ5OlegUMc4yTyKMnSkDTSTJUiIWcQ30waeweA54uc</latexit>!

activation function

Sc

Statistical mechanics on the “ensemble of ﬁxed points”
fixed point patternsµSi (t) = Si

µ = 1, 2, . . . , M

M↵=
N

N, M ! 1 with fixed ↵<latexit sha1_base64="wuyXAWKQJXyT91z0o8NeC5Bg81Y=">AAACeXichVHLSgMxFD0dX7W+6mMhuFGLUl2UVAVFEEQ3LvuwKvgoM2PU4LyYSQt16A/4Ay7cqCBS/Qw3/oALP0FcKrjQhbfTAVFRb0hycnLPzUmiOYbwJGMPEaWpuaW1Ldoe6+js6u6J9/ateXbJ1XlBtw3b3dBUjxvC4gUppME3HJerpmbwde1wub6/XuauJ2xrVVYcvm2q+5bYE7oqiSrGB/JFX1STcmIhADv+llmqFuMJlmJBDP8E6RAkEEbGjl9hC7uwoaMEExwWJGEDKjxqm0iDwSFuGz5xLiER7HNUESNtibI4ZajEHtK4T6vNkLVoXa/pBWqdTjGou6Qcxhi7ZzX2zO7YDXtk77/W8oMadS8VmrWGljvFnuPB/Ou/KpNmiYNP1Z+eJfYwF3gV5N0JmPot9Ia+fHTynJ/Pjfnj7II9kf9z9sBu6QZW+UW/zPLcKWL0Aenvz/0TrE2l0tOpqexMYnEp/IoohjCKJL33LBaxggwKdO4RzlDDdeRNGVGSymQjVYmEmn58CWX6AxVkki8=</latexit><latexit sha1_base64="moARqhCvJm0Z/4cZNVEckuXMI/M=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoqPjp+CrElOQrxGTmpZVUqiiUZ5ZkKKRlVqSmKKjEJOYUZCSqxAsoG+gZgIECJsMQylBmgIKAfIEVDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKy7AqgiH6gzH2hKLYMCg6rBVYOVBp8NThisNnhp8AenadVgU0CuqQTSSRC9qQXx/F0Swd8J6soF0iUMGQhdeF1dwpDGYAF2bSbQ9QVgEZA/kiH6y6qmfw62ClKtVjNYZPAa6P6FBjcNDgN9kFf2JXlpYGrQbAYucBRYgoAxPMAxGWFGeobGesaBRsoOTtDI4GCQZlBi0ACGuDmDA4MHQwBDKNDeLoYNDDsZdjHxMhkzWTHZQJQyMUL1CDOgACZXAIDllq8=</latexit><latexit sha1_base64="buqQMfIAvnJWoco3d5kEck62VG8=">AAACeXichVHLSsNAFD2Nr1ofrXUjuKkWRVyUqQqKIBTduFHaaqtgpUziVEPTJCRpUUN/wB9wIQgKYsXPcOMPuPATxGUFF7rwNg2IinoHZs6ce8+dMzOyqam2w9hjQOro7OruCfaG+voHBsORoWjeNqqWInKKoRnWtsxtoam6yDmqo4lt0xK8ImtiSy6vtPJbNWHZqqFvOkem2K3wfV0tqQp3iCpGogWumQd8qVCyuOKu1d31ejESZwnmRewnSPogDj/SRqSBAvZgQEEVFQjocAhr4LBp7CAJBpO4XbjEWYRULy9QR4i0VaoSVMGJLdO8T7sdn9Vp3+ppe+pDqjBIaVCXOmKYYA/shjXZPbtlT+z9126u16Xl5ohWua0VZjF8MrLx+q+qQquDg0/Vn64dlLDguVXJvekxrXsobX3t+LS5sZidcCfZJXsm/xfskd3RDfTai3KVEdkzhOgLkt8f/CfIzySSs4mZzFw8tex/RhCjGMcUvfg8UlhFGjk69xDnuEYj8CaNSVPSdLtUCviaYXwJafYDwveSBA==</latexit>

Gardner volume
Elisabeth Gardner(1957-1988)“Hardcore” constraint

e

V (h)

= ✓(h)

V =<latexit sha1_base64="ot8kMOWPau9oxqoIrzVKG6sTgNo=">AAAC/HichZHNahRBEMdrRqNxjGaNF8HL4JIQwSw9G8EgBIJeRFDy4W4CmewyM1sbO5kve3oXY9M+gC+QgydFEQlefQEvvoCHPIJ4jOAlgdTMDorGJD30dPW/6ldV3e2nIc8kY7uGeebs0LnzwxesiyOXLo9Wrow1s6QnAmwESZiIFd/LMOQxNiSXIa6kAr3ID3HZ37yf+5f7KDKexE/kVoprkbce8y4PPElSu7LdnLVcHkvbclORdNpqY9bRLfVYu13hBapjuxKfy6KOEtjR6iGFaK2Vmz0TUtXdlGttYUtNDYD/h7dUnRCav8u4Ua8o9KiEfZSe3ZwUrdyjb+p2pcpqrBj2UcMpjSqUYz6pfAAXOpBAAD2IACEGSXYIHmT0rYIDDFLS1kCRJsjihR9Bg0Vsj6KQIjxSN+m/TrvVUo1pn+fMCjqgKiFNQaQN4+wb+8j22Fe2w76z/WNzqSJH3ssWrf6AxbQ9+ura0q9TqYhWCU//UCf2LKELM0WvnHpPCyU/RTDg+y+295buLo6rCfaW/aD+37Bd9oVOEPd/Bu8XcPE1WPQAzr/XfdRo1mvOdK2+cLs6d698imG4Djdgku77DszBA5iHBtU9MKrGLWPKfGm+M3fMT4NQ0yiZq/DXMD8fAg87xfA=</latexit>

Z Y
N

“Gap”

dJ
p j e
2⇡
j=1

Jj 2
2

M
Y

e

V (r µ )

µ=1

N
X
1
µ
µ
p Ji Siµr = S0N
i=1<latexit sha1_base64="P4+YCNesE0u++NLr7tbSB2gpgvQ=">AAACunichVHLThRBFD00Ijg+GGVD4mbiBONqUo0mECMJkQ1xYYBxgITGTndTA5XpF9XVE7BSP+APuHAlCTHGT3Dpxh9wwdKlssTEjQtvPxKiRL2Vqjr31D333qry01BkirGTEWv00tjl8YkrjavXrt+YbN68tZ4luQx4L0jCRG76XsZDEfOeEirkm6nkXuSHfMMfLBXnG0MuM5HEz9RhyrcjbzcWfRF4iii32ZPPtRPlZqHVdTUzleNkeeRqsWCT/9Q4fekF2jbayfalIsI4ih+osraWfMfoJxRsTLcSF9httlmHlda6COwatFHbStJ8Cwc7SBAgRwSOGIpwCA8ZjS3YYEiJ24YmThIS5TmHQYO0OUVxivCIHdC6S95WzcbkFzmzUh1QlZCmJGULM+wze8fO2Cf2nn1lP/+aS5c5il4OafcrLU/dyZfT3R//VUW0K+ydq/7Zs0If82WvgnpPS6a4RVDphy9enXUfrs3ou+yInVL/b9gJ+0g3iIffg+NVvvYaDfoA+8/nvgjWZzv2/c7s6oP24uP6KyZwG3dwj957DotYxgp6VPcDvuAbTq1Hlm8Ja1CFWiO1Zgq/maV+AVavrvM=</latexit>

1
J⌅

µ
S⌅(1)

S0,1

S2,1

S1,1

<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit>

S3,1<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅(2)

2
J⌅

S0,2

<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅

N
J⌅
<latexit sha1_base64="PzUIaUxmlcfnXo6NBtE8K0QXcG4=">AAACeXichVHNLgNRFP46/uun9bOQ2KAhYtHcthJiJWzEQqoUCTWZGReTzp+ZaRMmfQEvYGGDRKQ8ho0XsOgjiCWJBQtnpiOIlHNz7z33O+c797v3yJamOi5jtYjQ1NzS2tbeEe3s6u6JxXv71h2zZCs8r5iaaW/KksM11eB5V3U1vmnZXNJljW/IxQU/vlHmtqOaxpp7ZPGCLu0b6p6qSC5BYnxgSfS2ZU1Sis5hSbJ5ZcdbrojxRCrJAhtmv5zPUAKhZc34NbaxCxMKStDBYcAlX4MEh8YWUmCwCCvAI8wmTw3iHBVEiVuiLE4ZEqFFWvfptBWiBp39mk7AVugWjaZNzGGMsQdWZc/snt2yR/besJYX1PC1HNEu17ncEmMng6uv/7J02l0cfLH+1OxiDzOBVpW0WwHiv0Kp88vHp8+rs7kxb5xdsifSf8Fq7I5eYJRflKsVnjtD9HsDGjvr6WQqk0yvTCXm5sNWtGMIo5ig/57GHBaRRZ7uPcY5qriJvAkjwoQwWU8VIiGnHz9MyHwAAzCSow==</latexit>

µ
S⌅(N
)

S1,2

<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅
= sgn

<latexit sha1_base64="8TJBS5cQywa2sByQTfTnXTF/7rU=">AAACfnichVFNLwNBGH66vqq+igtxEQ3hoKYl0Tg1XJykVEuiNLtryqb7ZT+a1KZx9wccnEgaEQk/wsUfcPATxLESFwdvt5sIgncyM8888z7vPDMjmapiO4w9hYS29o7OrnB3pKe3r38gOjiUtw3XknlONlTD2pZEm6uKznOO4qh827S4qEkq35LKK839rQq3bMXQN52qyXc18UBXSoosOkQVo6PZPa+gubWiV5BUUS7bR65o8em1mVoxGmNx5sf4T5AIQAxBZIzoFQrYhwEZLjRw6HAIqxBhU9tBAgwmcbvwiLMIKf4+Rw0R0rqUxSlDJLZM4wGtdgJWp3Wzpu2rZTpFpW6RchyT7JFdswZ7YDfsmb3/WsvzazS9VGmWWlpuFgdOR7Jv/6o0mh0cfqr+9OyghJTvVSHvps80byG39JXjs0Z2aWPSm2KX7IX8X7Andk830Cuvcn2db5wjQh+Q+P7cP0E+GU/Mx5PrC7H0cvAVYYxhAtP03otIYxUZ5OjcE9RxizsBwpQwK8y1UoVQoBnGlxBSH/TUk20=</latexit><latexit sha1_base64="8fA3cAJF8y34tZihZo9XmxWhHFk=">AAAC8nichVFLaxRBEK4ZX3F9ZNWL4mXIEtlclp5EUIRA0It4kCTrJoF0dpjp9G6anVe6exZj039A8CzoKYqI6L/wkj+QQ36CeFzBiwdrZwfExEc13V31dX1VX3dHeSyUJuTIcU+dPnP23NT52oWLly5P169cXVNZIRnvsCzO5EYUKh6LlHe00DHfyCUPkyjm69Hgwfh8fcilEln6RO/lfCsJ+6noCRZqhIL6i3bX0KSwgaFRHLKB2i1Cye2ioTLxVD+1Ho15TzdpT4bM+NZQtSu1eWwtVUUSGLHo2y6GHtX8qS71GMm3rXl0rGLXCGv/2Kwp5iyVor+j54J6g7RIad5Jx6+cBlS2nNXfA4VtyIBBAQlwSEGjH0MICscm+EAgR2wLDGISPVGec7BQQ26BWRwzQkQHuPYx2qzQFONxTVWyGXaJcUpkejBLDskHMiIH5CP5Qn78tZYpa4y17OEeTbg8D6afX29//y8rwV3Dzi/WPzVr6MHdUqtA7XmJjG/BJvzhs5ej9r3VWXOLvCFfUf8+OSKf8Qbp8Bt7t8JXX0MNP8A//twnnbX5lr/Qml+53Vi6X33FFNyEGWjie9+BJXgIy9DBviPnhjPjNFztvnL33beTVNepONfgN3M//QSdqsPi</latexit>

N
X

1
i µ
p
J⌅
S⌅(i)
N i=1

0

S0,3
<latexit sha1_base64="(null)">(null)</latexit>

!

N

X
1
µ
SL,i (t + 1) = sgn @ p
JL,i,j sgn
N j=1<latexit sha1_base64="330hvt09FDcdy8cKr/afOeav6xU=">AAADinicjVHLbtNAFL2ueZRQaAobJDYWoSgVJhoHJF6qVAELhBBqG9JWqlvLnkxSJ34xM4koo/kB+AAWrEBCCLFiC0s2/ACLfgJiWSQ2LLhxLBCtCL2Wfe89c8/xmZkgi0IhCdkxJsxDh48cnTxWOj514uR0eebUikj7nLImTaOUrwW+YFGYsKYMZcTWMs78OIjYatC7PVxfHTAuwjR5KLczthH7nSRsh9SXCHkzxvnGpnLjvvbUfTvUVXnRmbPmlctjS3QSbbkRa8uq2+Y+VY5WrnjEpXqgtSv6sae6847exNZyJXssczeKs5ZW93I5u6v1QaV646QuOXbX7uEobaVSWKWDisZjRB07smOtf2+fYFeVcy4PO1t7k1eukBrJw9pfOEVRgSIW0/IbcKEFKVDoQwwMEpBYR+CDwGcdHCCQIbYBCjGOVZivM9BQQm4fpxhO+Ij28NvBbr1AE+yHmiJnU/xLhC9HpgWz5At5S3bJZ/KOfCU//6mlco2hl23MwYjLMm/66ZnGj/+yYswStv6wxnqW0IZrudcQvWc5MtwFHfEHT57vNm4sz6oL5BX5hv5fkh3yCXeQDL7T10ts+QWU8AKcvce9v1ip15zLtfrSlcrCreIqJuEsnIMqnvdVWIC7sAhNoMYz473xwfhoTpl187p5czQ6YRSc0/BXmHd+Ab3E+i4=</latexit>

S0,N

S1,N

input

hiddenN

1 X
p
JL 1,j,k · · · sgn
N k=1

N

1 X
µ
p
J1,l,m S0,m
(t)
N m=1

e.g.

N X
M ⇣
X
µ
E=
SL,i
i=1 µ=1(2) try to minimize the loss function
via back-propagation!!1
A

desired output

Usual strategy of learning(1) define “loss function"

output(S⇤ )µL,i⌘2e.g. SDG (stochastic gradient descent)Too much long-ranged, highly convoluted, non-linear interaction! …hard to analyze

1
J⌅

µ
S⌅(1)

S0,1

S2,1

S1,1

<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit>

S3,1<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅(2)

2
J⌅

S0,2

<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅

N
J⌅

S0,3
<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="PzUIaUxmlcfnXo6NBtE8K0QXcG4=">AAACeXichVHNLgNRFP46/uun9bOQ2KAhYtHcthJiJWzEQqoUCTWZGReTzp+ZaRMmfQEvYGGDRKQ8ho0XsOgjiCWJBQtnpiOIlHNz7z33O+c797v3yJamOi5jtYjQ1NzS2tbeEe3s6u6JxXv71h2zZCs8r5iaaW/KksM11eB5V3U1vmnZXNJljW/IxQU/vlHmtqOaxpp7ZPGCLu0b6p6qSC5BYnxgSfS2ZU1Sis5hSbJ5ZcdbrojxRCrJAhtmv5zPUAKhZc34NbaxCxMKStDBYcAlX4MEh8YWUmCwCCvAI8wmTw3iHBVEiVuiLE4ZEqFFWvfptBWiBp39mk7AVugWjaZNzGGMsQdWZc/snt2yR/besJYX1PC1HNEu17ncEmMng6uv/7J02l0cfLH+1OxiDzOBVpW0WwHiv0Kp88vHp8+rs7kxb5xdsifSf8Fq7I5eYJRflKsVnjtD9HsDGjvr6WQqk0yvTCXm5sNWtGMIo5ig/57GHBaRRZ7uPcY5qriJvAkjwoQwWU8VIiGnHz9MyHwAAzCSow==</latexit>

µ
S⌅(N
)

S1,2

<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅
= sgn

<latexit sha1_base64="8TJBS5cQywa2sByQTfTnXTF/7rU=">AAACfnichVFNLwNBGH66vqq+igtxEQ3hoKYl0Tg1XJykVEuiNLtryqb7ZT+a1KZx9wccnEgaEQk/wsUfcPATxLESFwdvt5sIgncyM8888z7vPDMjmapiO4w9hYS29o7OrnB3pKe3r38gOjiUtw3XknlONlTD2pZEm6uKznOO4qh827S4qEkq35LKK839rQq3bMXQN52qyXc18UBXSoosOkQVo6PZPa+gubWiV5BUUS7bR65o8em1mVoxGmNx5sf4T5AIQAxBZIzoFQrYhwEZLjRw6HAIqxBhU9tBAgwmcbvwiLMIKf4+Rw0R0rqUxSlDJLZM4wGtdgJWp3Wzpu2rZTpFpW6RchyT7JFdswZ7YDfsmb3/WsvzazS9VGmWWlpuFgdOR7Jv/6o0mh0cfqr+9OyghJTvVSHvps80byG39JXjs0Z2aWPSm2KX7IX8X7Andk830Cuvcn2db5wjQh+Q+P7cP0E+GU/Mx5PrC7H0cvAVYYxhAtP03otIYxUZ5OjcE9RxizsBwpQwK8y1UoVQoBnGlxBSH/TUk20=</latexit><latexit sha1_base64="8fA3cAJF8y34tZihZo9XmxWhHFk=">AAAC8nichVFLaxRBEK4ZX3F9ZNWL4mXIEtlclp5EUIRA0It4kCTrJoF0dpjp9G6anVe6exZj039A8CzoKYqI6L/wkj+QQ36CeFzBiwdrZwfExEc13V31dX1VX3dHeSyUJuTIcU+dPnP23NT52oWLly5P169cXVNZIRnvsCzO5EYUKh6LlHe00DHfyCUPkyjm69Hgwfh8fcilEln6RO/lfCsJ+6noCRZqhIL6i3bX0KSwgaFRHLKB2i1Cye2ioTLxVD+1Ho15TzdpT4bM+NZQtSu1eWwtVUUSGLHo2y6GHtX8qS71GMm3rXl0rGLXCGv/2Kwp5iyVor+j54J6g7RIad5Jx6+cBlS2nNXfA4VtyIBBAQlwSEGjH0MICscm+EAgR2wLDGISPVGec7BQQ26BWRwzQkQHuPYx2qzQFONxTVWyGXaJcUpkejBLDskHMiIH5CP5Qn78tZYpa4y17OEeTbg8D6afX29//y8rwV3Dzi/WPzVr6MHdUqtA7XmJjG/BJvzhs5ej9r3VWXOLvCFfUf8+OSKf8Qbp8Bt7t8JXX0MNP8A//twnnbX5lr/Qml+53Vi6X33FFNyEGWjie9+BJXgIy9DBviPnhjPjNFztvnL33beTVNepONfgN3M//QSdqsPi</latexit>

N
X

1
i µ
p
J⌅
S⌅(i)
N i=1

!

S0,N

S1,N

input

hiddenoutputGardner volume generalized for a multi-layer network
trace over hidden variables0

V (S(0), S(L)) = eN M S(S(0),S(l)) = @Hamiltonian with“short-ranged” interactions

H=

M X
X

µ=1 ⌅

“Hardcore” constraint

e

V (h)

= ✓(h)

L
N
Y1 Y

X

µ
l=1 i=1 Sl,i
=±1

µ
V (r⌅
)

10
A@

“Gap”

Gaussian approx.
or modiﬁed model

Z YY
N

j
dJ⌅
p e
2⇡
⌅ j=1

(J

j 2
)
⌅
2

1

Ae

N
X
1 i µ
µ
µ
p
r⌅ =
J⌅ S⌅(i) S⌅N
i=1<latexit sha1_base64="I3kOm1j0xPHvbkwIKL9Rjj68l4U=">AAAC8nichVHNThRBEK4Z/MH1hwUvEi8bNhi8bHrQBGNCQvRCPBBgXSBh2ElP04udnT+6ezZip1/AxLMJnoAYY/QtvPgCHngE43FJuHiwdnYSI6BWp7urv6rvq+ruMIuE0oQcO+7IpctXro5eq1y/cfPWWHV8Yk2luWS8xdIolRshVTwSCW9poSO+kUlO4zDi62H36SC+3uNSiTR5rvcyvhXTnUR0BKMaoaD6RraNH+c2MH4YUdZVuzmV3M7XfJXHgRHznm2bJet3JGXGs8ZXu1IjYH3NX+qivpF825pnZxTaRljbvEh8Rty3tQsjNqjWSYMUVjvveKVTh9KW0+oH8GEbUmCQQwwcEtDoR0BB4dgEDwhkiG2BQUyiJ4o4BwsV5OaYxTGDItrFdQdPmyWa4HmgqQo2wyoRTonMGkyTb+Qj6ZOv5BP5Tn7+VcsUGoNe9nAPh1yeBWOv7zRP/8uKcdfw4jfrnz1r6MCjoleBvWcFMrgFG/J7r972m49Xp809ckh+YP8H5Jh8wRskvRP2foWvvoMKfoB39rnPO2uzDe9BY3blYX3hSfkVo3AXpmAG33sOFmARlqGFdfvOpDPl1F3t7rsH7tEw1XVKzm34w9zPvwCO18S8</latexit>µ
r⌅=<latexit sha1_base64="LFsx4CZQdFw0Nk7Cs76hK6jsVEQ=">AAADQnichVHLahRBFL3dvmL7yEQ3gpvGITLZDNVRMAiBYDYiRJKMkwRSmaa7UzMppl+prh4Si/oBf8CFKwUR8TN04Q+4yMIPUFcygpssvFPToE583Karbp17z6lTVWEe80IScmTZp06fOXtu6rxz4eKly9O1mSsbRVaKiLWjLM7EVhgULOYpa0suY7aVCxYkYcw2w/7yqL45YKLgWfpIHuZsJwl6Ke/yKJAI+bWPoqNoUmpf0TAOon6xXwaC6UXHpUWZ+IoverqjHmqXdkUQKU8rWuwLiYimkh1I40AJtqvVgwmNjuJaOzRmXdmYYK8g28jTtDQbrPyqFsYlw84Dbqxhi9ZuqzPqnbDZ4HP6jwVNBe/tyTm/VidNYsI9mXhVUocqVrPaK6CwCxlEUEICDFKQmMcQQIHfNnhAIEdsBxRiAjNu6gw0OMgtsYthR4BoH8cerrYrNMX1SLMw7Ah3ifEXyHRhlnwgr8mQvCdvyCdy/FctZTRGXg5xDsdclvvTT661vv+XleAsYe8n65+eJXRhwXjl6D03yOgU0Zg/ePx02Lq7PqtukhfkC/p/To7IWzxBOvgWvVxj68/AwQfwJq/7ZLIx3/RuNefXbteX7lVPMQXX4QY08L7vwBLch1VoQ2QtW9wSVmG/sz/bX+3huNW2Ks5V+C3s4x+kr+ap</latexit>

N
X

1 i
p J⌅
N
i=1

M
X

1
⌫
⌫
p
⇠ µ⌫ S⌅(i)
S⌅
M ⌫=1

⇠ µ⌫ : Gaussian with zero mean and variance 1(c.f. ) internal representation (2-layer): R. Monasson and R. Zecchina (1995)

H!

Replicas: machines learning in parallel

1
J⌅,i
same
random
input

2
J⌅,i

n
J⌅,i

1
S⌅
machine 1
same
randomoutput2
S⌅
machine 2nS⌅machine nglass order parametersqab,⌅ =

M
X
1

M µ=1

µ a
µ b
(S⌅
) (S⌅
)

Qab,⌅ =

N
X
1

N i=1

a
b
J⌅,i
J⌅,i

Parisi's RSB ansatz
overlap matrix

Qab = Q(x)xsha1_base64="NNxeohubzUKfMFGqJK+8u4N+6W0=">AAACZnichVFNLwNRFD0d3/VVREhsGk3FanKrolg1bCwVRVIiM+OVienMZGbaoOkfYEssrEgkxM+w8Qcs/ANiSWJj4c50EBHcl7x377n33Hfue6pt6K5HdB+RGhqbmlta26LtHZ1d3bGe3mXXKjuayGuWYTmrquIKQzdF3tM9Q6zajlBKqiFW1J1ZP79SEY6rW+aSt2eL9ZKyZepFXVM8hnK7G7FESqbA4iRnxidpIsNOiHykEght3opdYg2bsKChjBIETHjsG1Dg8iogBYLN2DqqjDns6UFeoIYoc8tcJbhCYXSH9y2OCiFqcuz3dAP2LldYzLS4Sw1xJOmOruiZbumaHunt127VoIuvZo9Ptc4V9kb3weDi67+sEp8etr9Yf6r2UMRkoFZn9XaA+HNodX5l/+R5cXohWR2hc3pi/Wd0Tzc8gVl50S5yYuEUUf4Ckqd8S38++E9neUxOpeV0bjyRnQk/oxVDGMYov3gGWcxhHnm+V+AQRziOPEhdUr80UC+VIiGnD99Mir8Dd9qK0A==</latexit><latexit0
sha1_base64="JbTPy8hhR9fKbM/TxJyClFpEMlU=">AAACZnichVFNLwNRFD0dX1VfRYTEptEQq8mtSqpWwsZSW0WCyMx4mJjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLgzHUQE9yXv3XvuPfed+55qG7rrEd1HpKbmlta2aHuso7Oruyfe27fkWmVHE0XNMixnRVVcYeimKHq6Z4gV2xFKSTXEsro75+eXK8Jxdctc9Kq2WC8p26a+pWuKx1CONuLJlEyBJUjOEGUns+yEyEcqidAWrPgl1rAJCxrKKEHAhMe+AQUur1WkQLAZW0eNMYc9PcgL1BFjbpmrBFcojO7yvs3RaoiaHPs93YC9xxUWMy3uUkcCo3RHV/RMt3RNj/T2a7da0MVXU+VTbXCFvdFzMFR4/ZdV4tPDzhfrT9UetjAVqNVZvR0g/hxag1/ZP3kuTOdHa2N0Tk+s/4zu6YYnMCsv2kVO5E8R4y8gOetb+vPBfzpLE3IqLadzk8mZ2fAzohjGCMb5xTOYwTwWUOR7BQ5xhOPIg9QtDUiDjVIpEnL68c2kxDvhWYqF</latexit><latexit

1
sha1_base64="3bzTuENv8W8Lb0MR5px024P00Uk=">AAACZnichVFNLwNRFD0dX1VfRYTEptEQq8mtSqpWwsZSW0WCyMx4mJjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLgzHUQE9yXv3XvuPfed+55qG7rrEd1HpKbmlta2aHuso7Oruyfe27fkWmVHE0XNMixnRVVcYeimKHq6Z4gV2xFKSTXEsro75+eXK8Jxdctc9Kq2WC8p26a+pWuKx1AutRFPpmQKLEFyhig7mWUnRD5SSYS2YMUvsYZNWNBQRgkCJjz2DShwea0iBYLN2DpqjDns6UFeoI4Yc8tcJbhCYXSX922OVkPU5Njv6QbsPa6wmGlxlzoSGKU7uqJnuqVreqS3X7vVgi6+miqfaoMr7I2eg6HC67+sEp8edr5Yf6r2sIWpQK3O6u0A8efQGvzK/slzYTo/Whujc3pi/Wd0Tzc8gVl50S5yIn+KGH8ByVnf0p8P/tNZmpBTaTmdm0zOzIafEcUwRjDOL57BDOaxgCLfK3CIIxxHHqRuaUAabJRKkZDTj28mJd4B41uKhg==</latexit><latexit

n

n!0

n

Probability distribution of
overlap between replicasdx(Q)P (Q) =
dQ

Extension to multi-layers

1

2

L

l

Replicated Gardner volumen

V (S0 , SL )

=

n
Y

a=1

Y
⌅

TrJa⌅

!0
@

Y

⌅\output

1

TrSa⌅ A<latexit sha1_base64="WJyPO9UnhzkN9QKvTFjHaiLgv7k=">AAADbnicjVFbaxNBFD6b9VLjpWkFEURcDJUUapi0giIUioL4oNI0Jq1k0jC7TpKhs5fuTiJ1nD/gH/DBJwUfxJ8hgi8++tCfIL4IFXzxwbMXLFq8zLIz33xzvu+cmeNGUiSKkF2rZB86fOTo1LHy8RMnT01XZmY7STiOPd72QhnGGy5LuBQBbyuhJN+IYs58V/J1d+tmer4+4XEiwuC+2ol4z2fDQAyExxRS/coHOoiZpy9Tlyvm0BBjUyt9q6apO3Bapq+JWfiJ75h5s6lp7DsTkQhMYoy+d9cs5y5U8oGqOzRisRJM9nVg9h07m7j9X1cai+FIOU/QYpkUOcqt1LBLtaYjppymqcn5hRxvp5iaXr9SJXWSDecgaBSgCsVYDSvvgMJDCMGDMfjAIQCFWAKDBL8uNIBAhFwPNHIxIpGdczBQRu0YozhGMGS3cB7irluwAe5TzyRTe5hF4h+j0oE58pG8JnvkPXlDPpHvf/TSmUdayw6ubq7lUX/66dnWt3+qfFwVjPZVf61ZwQCuZbUKrD3KmPQWXq6fPH6217q+NqcvkZfkM9b/guySt3iDYPLVe9Xka88z9wA1j7Lb+ln+AN9XI+9iVlMgWeROYBuzpO+K74nNa/zeqoOgs1hvLNUXm1eqKzeKNk7BObgINezVVViB27AKbfCsJeuB5Vpe6Yt9xj5vX8hDS1ahOQ2/DLv2A6c44+w=</latexit>µ,⌅,a<latexit sha1_base64="6OTW3KmDBK8uRZWaCCB27VLpQTc=">AAAC/HichVHNahRBEK4Z/5L1J6teArksLpEIcemJgiIEgl6CB0mybhLIZIeeTm9sdv7S3bMYm84D5AVy8KQoIsGrL5BLXsBDHkE8RsglgrWzA6JrtJrurv6qvqqvu8MsEkoTcuS4585fuHhpZLRy+crVa2PV6zeWVZpLxlssjVK5GlLFI5HwlhY64quZ5DQOI74Sdp/04ys9LpVIk+d6O+PrMd1MREcwqhEKqnuybfw4t4Hxw4iyrtrKqeTT1M7WmmdEfJXHgRGznm2bZ9bvSMqMZ42vtqRGwPqav9SFMlMQrXk6VKNthLV/bTAl7mC8FlTrpEEKqw07XunUobSFtPoBfNiAFBjkEAOHBDT6EVBQONbAAwIZYutgEJPoiSLOwUIFuTlmccygiHZx3cTTWokmeO7XVAWbYZcIp0RmDSbJF/KRHJNDsk++ktMza5miRl/LNu7hgMuzYGx3vHnyX1aMu4YXv1j/1KyhAw8LrQK1ZwXSvwUb8Huv9o6bj5YmzW3ylnxD/W/IETnAGyS97+z9Il96DRX8AO/P5x52lmca3r3GzOL9+tzj8itGYAJuwRS+9wOYg3lYgBb2/eHUnWnnrrvjvnP33U+DVNcpOTfhN3M//wQX68gW</latexit>

visible@n V n (S0 , SL )=Sn [{q̂(l)}, {Q̂(l)}] = ↵M↵=Nquenched random input/output

visible
n=0

NM
1

L
X

bond
Sent
[Q̂(l)] +

l=1

L
X
l=1

<latexit sha1_base64="W151Rme7a1IYaMmHG/mvlld9r9w=">AAADz3ichVHLbhMxFL1peJTh0RY2SGwsqlYJkMjTVipFqlTBhgWLpqEPqU5HnqnTjOp5dMYpKsaILfwAC1YgIYT4DDYs2bDoJyCWRbBBghtnShVVlDsa+9zje46vbT+VYa4o3S8NlU+dPnN2+Jxz/sLFSyOjY5dX8qSbBWI5SGSSrfk8FzKMxbIKlRRraSZ45Eux6m/f662v7oosD5P4odpLRSviW3HYDgOukPLGSuEkaXo6NutMa9bhiuyYiqwyc+swb/TzljPPuEw7fEPXXENY3o08Ledds6EfGNLc0CyLiJ/Em8azUMQKPdFAW4OWc3NA0vMoRHkaxsdEtosWYXESdyNfZIQxZ5KwTp7yQNzQ00FkSG2wCUegXzvjgXaNnjL9Re6bHTtWZM2tNgpYPeSqLOWZCrn0dAepHmMGKFtkkMuSTSywW8XGYVK0VZ3gjjXmC8XJSuVQXzWEsCzc6ijy5Mh1nhpvdJzWqQ1yHLgFGIciFpPRd8BgExIIoAsRCIhBIZbAIcdvHVygkCLXAo1chii06wIMOKjtYpXACo7sNo5bmK0XbIx5zzO36gB3kfhnqCQwQb/Q9/SAfqIf6Ff6659e2nr0etnD2e9rReqNvLja/PlfVYSzgs6R6sSeFbThtu01xN5Ty/ROEfT1u49fHjTvLE3oSfqGfsP+X9N9+hFPEO9+D942xNIrcOwDzNkgfTA7U4A59+8DrEzV3en6VGNmfOFu8RTDcA2uQwXvexYW4D4swjIEpc+lH6XfQ1BulB+Vn5af9UuHSoXmCgxE+fkfl7MPyA==</latexit>

N, M ! 1eµV (r⌅,a
)

N
X
1 i
µ
µ
µ
p
r⌅,a = S⌅,aJ⌅,a S⌅(i),aN
i=1

Replicated free-energy

F (S0 , SL )
NM

Y

L
X1

spin
Sent [q̂(l)]

l=1

P
1

e2

= Sn [{Q̂(l), q̂(l)}]ab qab (l1)Qab (l)qab (l)@ha (l) @hb (l)qab (0) = qab (L) = 1

n
Y

a=1

e

V (ha (l))
ha (l)=0

1

1st 4.2
Glass1sttransition
glass transition<latexit sha1_base64="TboprELbJ9ahhjMRilxn2XzKXFo=">AAACaHichVFNLwNBGH66vuujiwPiIhrioplViY+TcHHUVpEgze6a1qb7ZXfaoPEHJK4cnEgkxM9w8Qcc/AQcSVwcvLvdEGnwTjLzzPO+zzvPzGiuafiCsceY1NTc0trW3hHv7OruSci9fWu+U/F0ntcd0/E2NNXnpmHzvDCEyTdcj6uWZvJ1rbwU5Ner3PMNx14VBy7fttSSbRQNXRVE5ZTJvYKcZCkWxkgjUCKQRBQrjnyNLezAgY4KLHDYEIRNqPBpbEIBg0vcNmrEeYSMMM9xhDhpK1TFqUIltkxziXabEWvTPujph+p9qnBI6VCXI4xgjD2wG/bK7tkte2Ifv3arhV0CNwe0anUtdwuJ48Hc+78qi1aB3W/Vn64FipgN3Rrk3g2Z4B56XV89PHvNzWfHauPskr2Q/wv2yO7oBnb1Tb/K8Ow54uEXzAWR/nrwRrA2lVLSqXRmOrmwGH1GO4Yxigl68RksYBkryNO5JZzgFGexZ0mWBqSheqkUizT9+BHS6Ccx3Ysa</latexit>

q

spin

<latexit sha1_base64="2f1uU3s+zy0a025anBjCukW5Hv8=">AAACb3ichVHLSiNRED1pdYxRJ1EXCoI0E5RxEypG83DlA2GWvqIyKqG7vWpjv+zuBDXkB1wLLkRBQRjxM9z4Ay78BJmVODAbF1Z3mhmGYca6cG/VqTp1T92rOobu+USPMamlte1De7wj0dnV/TGZ6uld8eyqq4myZhu2u6YqnjB0S5R93TfEmuMKxVQNsaruzQb51ZpwPd22lv1DR2yayo6lb+ua4jP0db9S33BNeW66UUmlKUOhyZTJFydKxTw7BRqjfEnORqk0Ipu3U9+wgS3Y0FCFCQELPvsGFHi81pEFwWFsE3XGXPb0MC/QQIK5Va4SXKEwusf7DkfrEWpxHPT0QvYBV9jMtLlLAzKG6YFu6IXu6Zae6PWf3ephl0DNIZ9qkyucSvJ4YOnnuyyTTx+7v1n/Ve1jG8VQrc7qnRAJ5tCa/NrR6cvS5OJwfYSu6Dvrv6RHuuMJrNoP7XpBLJ4hEX5BKbDcrwf/21kZy2RzmdzCeHpqJvqMOAbxCZ/5xQuYwhfMo8z3WjjBOS5iz1K/NCTJzVIpFnH68IdJo2+CCI58</latexit>

qEA<latexit sha1_base64="2f1uU3s+zy0a025anBjCukW5Hv8=">AAACb3ichVHLSiNRED1pdYxRJ1EXCoI0E5RxEypG83DlA2GWvqIyKqG7vWpjv+zuBDXkB1wLLkRBQRjxM9z4Ay78BJmVODAbF1Z3mhmGYca6cG/VqTp1T92rOobu+USPMamlte1De7wj0dnV/TGZ6uld8eyqq4myZhu2u6YqnjB0S5R93TfEmuMKxVQNsaruzQb51ZpwPd22lv1DR2yayo6lb+ua4jP0db9S33BNeW66UUmlKUOhyZTJFydKxTw7BRqjfEnORqk0Ipu3U9+wgS3Y0FCFCQELPvsGFHi81pEFwWFsE3XGXPb0MC/QQIK5Va4SXKEwusf7DkfrEWpxHPT0QvYBV9jMtLlLAzKG6YFu6IXu6Zae6PWf3ephl0DNIZ9qkyucSvJ4YOnnuyyTTx+7v1n/Ve1jG8VQrc7qnRAJ5tCa/NrR6cvS5OJwfYSu6Dvrv6RHuuMJrNoP7XpBLJ4hEX5BKbDcrwf/21kZy2RzmdzCeHpqJvqMOAbxCZ/5xQuYwhfMo8z3WjjBOS5iz1K/NCTJzVIpFnH68IdJo2+CCI58</latexit>

qEA

bond

QEA
<latexit sha1_base64="qBB77x19pK/fXjA/+CbqC+c7boU=">AAACb3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRQXGV8cU5Sq4OtbGCygb6JkbWpqZWygY6BkbG1mYGAMZ5gaGZmbmCoZ6BmCgzAAFAfkCKxhiGFIY8hmSGUoZchlSGfIYSoDsHIZEhmIgjGYwZDBgKACKxTJUA8WKgKxMsHwqQy0DF1BvKVBVKlBFIlA0G0imA3nRUNE8IB9kZjFYdwVQRT5QZz7QlFoGBQZVg6sGKw0+G5wwWG3w0uAPTtOqwaaAXFMJpJMgelML4vm7JIK/E9SVC6RLGDIQuvC6uoQhjcEC7NpMoOsLwCIgfyRD9JdVTf8cbBWkWq1msMjgNdD9Cw1uGhwG+iCv7Evy0sDUoNkMXOAosAQBY3iAYzLCjPQMjfWMA02UHZygkcHBIM2gxKABDHFzBgcGD4YAhlCgvXkMUxjmMsxj/MgkziTHpABRysQI1SPMgAKYNAFsg45y</latexit>

QEA

bond

<latexit sha1_base64="qBB77x19pK/fXjA/+CbqC+c7boU=">AAACb3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRQXGV8cU5Sq4OtbGCygb6JkbWpqZWygY6BkbG1mYGAMZ5gaGZmbmCoZ6BmCgzAAFAfkCKxhiGFIY8hmSGUoZchlSGfIYSoDsHIZEhmIgjGYwZDBgKACKxTJUA8WKgKxMsHwqQy0DF1BvKVBVKlBFIlA0G0imA3nRUNE8IB9kZjFYdwVQRT5QZz7QlFoGBQZVg6sGKw0+G5wwWG3w0uAPTtOqwaaAXFMJpJMgelML4vm7JIK/E9SVC6RLGDIQuvC6uoQhjcEC7NpMoOsLwCIgfyRD9JdVTf8cbBWkWq1msMjgNdD9Cw1uGhwG+iCv7Evy0sDUoNkMXOAosAQBY3iAYzLCjPQMjfWMA02UHZygkcHBIM2gxKABDHFzBgcGD4YAhlCgvXkMUxjmMsxj/MgkziTHpABRysQI1SPMgAKYNAFsg45y</latexit>

spin

Figure 4: Glass order parameters close to the 1st glass transition: a) spatial profile ofcontinuoustransition
full(l)
RSB
glass
phase
the Edwards-Anderson (EA) orderparameter
for spinstoqEA
and
bonds
QEA (l) slightly
before ↵ =g2.0 (empty symbols)/after ↵ = 3.125
symbols)
the 1st glass transition.
at 1 (filled
st & (L-1)
th layer
The depth is L = 10 in this example.
b)layers
Variationof theEAliquidorderphaseparameters qEA (1) =otherremainin theqk (1) and QEA (1) = Qk (1) at the 1st layer after passing the critical point of the 1st↵ ' 2.03<latexit sha1_base64="Uj5ZNLsPCbzxQewKFMPvBgp/e3I=">AAACfnichVFNL0NBFD2er6qvYkNsGg2xUbclIVYNG0tUkag07z2jJt6X916bVCP2/oCFFUkjIuFH2PgDFv0JYkliY+H29SWC4E5m5syZe+6cmdEcQ3o+Ub1FaW1r7+iMdEW7e3r7+mMDgxueXXJ1kdNtw3a3NNUThrREzpe+IbYcV6imZohN7WCpsb9ZFq4nbWvdrzhix1SLltyTuuozVYiN5FXD2VcL1bxrxovHeU+a4jCeTtJMIZagJAUR/wlSIUggjBU7doU8dmFDRwkmBCz4jA2o8LhtIwWCw9wOqsy5jGSwL3CMKGtLnCU4Q2X2gMcir7ZD1uJ1o6YXqHU+xeDusjKOcXqka3qhB7qhJ3r/tVY1qNHwUuFZa2qFU+g/Hc6+/asyefax/6n607OPPcwHXiV7dwKmcQu9qS8fnb1kF9bGqxN0Sc/s/4LqdM83sMqvem1VrJ0jyh+Q+v7cP8FGOpmaSaZXZxOZxfArIhjFGCb5veeQwTJWkONzT1DDLe4UKBPKlDLdTFVaQs0QvoQy/wETcJJ/</latexit>

4.3
glasstransition
2nd2ndGlasstransition

1<latexit sha1_base64="TboprELbJ9ahhjMRilxn2XzKXFo=">AAACaHichVFNLwNBGH66vuujiwPiIhrioplViY+TcHHUVpEgze6a1qb7ZXfaoPEHJK4cnEgkxM9w8Qcc/AQcSVwcvLvdEGnwTjLzzPO+zzvPzGiuafiCsceY1NTc0trW3hHv7OruSci9fWu+U/F0ntcd0/E2NNXnpmHzvDCEyTdcj6uWZvJ1rbwU5Ner3PMNx14VBy7fttSSbRQNXRVE5ZTJvYKcZCkWxkgjUCKQRBQrjnyNLezAgY4KLHDYEIRNqPBpbEIBg0vcNmrEeYSMMM9xhDhpK1TFqUIltkxziXabEWvTPujph+p9qnBI6VCXI4xgjD2wG/bK7tkte2Ifv3arhV0CNwe0anUtdwuJ48Hc+78qi1aB3W/Vn64FipgN3Rrk3g2Z4B56XV89PHvNzWfHauPskr2Q/wv2yO7oBnb1Tb/K8Ow54uEXzAWR/nrwRrA2lVLSqXRmOrmwGH1GO4Yxigl68RksYBkryNO5JZzgFGexZ0mWBqSheqkUizT9+BHS6Ccx3Ysa</latexit>

q

spin

bond

continuous transition to full RSB glass phaseat 2nd& (L-2)layer transition: a) spatial profile of
Figure 5: Glass order parameters close
to the2ndthglass
the Edwards-Anderson (EA)which
orderalso
parameter
forglass
spins
qEA (l) and
bonds
QEA
(l) slightly
induce 2ndtransitions
at 1stand L-thlayer↵g (2) ' 15.38<latexit sha1_base64="64V74Zhkt2+ro2udTFp+qOwDKes=">AAAChHichVFNLwNBGH6sr6qv4iJx0GgIiTSzrYY4iHBxbEuRqDS7a9TEftndNmHTo4s/4OBEIiK98gtc/AEHP0EcSVwcvN1uIgjeycw888z7vPPMjGrrwvUYe2yRWtvaOzojXdHunt6+/tjA4LprVRyNFzRLt5xNVXG5Lkxe8ISn803b4Yqh6nxD3V9u7G9UueMKy1zzDm2+bShlU+wKTfGIKsVGi4pu7ykl3y86Rrxcq02mpoquMPhBXM4k03OlWIIlWRDxn0AOQQJhZK3YFYrYgQUNFRjgMOER1qHApbYFGQw2cdvwiXMIiWCfo4YoaSuUxSlDIXafxjKttkLWpHWjphuoNTpFp+6QMo5x9sCu2Qu7Z3X2xN5/reUHNRpeDmlWm1pul/pPhlff/lUZNHvY+1T96dnDLuYCr4K82wHTuIXW1FePTl9W5/Pj/gS7YM/k/5w9sju6gVl91S5zPH+GKH2A/P25f4L1VFJOJ1O5mcTiUvgVEYxgDJP03rNYxAqyKNC5x6jjBrdShzQtpaVMM1VqCTVD+BLSwgeefZRy</latexit>

Growth of glass phase under larger constrains

spin

bond

1

1

qEA 0.8

QEA 0.8<latexit sha1_base64="RPD8LwSR4z5gJHAWTCHdiCZKVOM=">AAACb3ichVHLSsNAFD2Nr1pfVRcKghRLRTdlWgXFVVUEl221ttiWksRRg3mRpMVa+gOuBReioCAofoYbf8CFnyCuRMGNC2/SgKiod2DmzLn33DkzI5mqYjuMPQSEtvaOzq5gd6int69/IDw4tGEbVUvmOdlQDasgiTZXFZ3nHMVRecG0uKhJKs9Le8tuPl/jlq0Y+rpTN3lZE3d0ZVuRRYeozUylUbK0yMpisxKOsjjzIvITJHwQhR9pI3yFErZgQEYVGjh0OIRViLBpFJEAg0lcGQ3iLEKKl+doIkTaKlVxqhCJ3aN5h3ZFn9Vp7/a0PfU+VRikNKhLExHE2D27Zi/sjt2wR/b+a7eG18V1U6dVamm5WRk4HF17+1el0epg91P1p2sH25j33Crk3vQY9x5yS187OH5ZW8jGGpPsgj2R/3P2wG7pBnrtVb7M8OwJQvQFie8P/hNsJOOJmXgyMxtNLfmfEcQYJjBFLz6HFFaRRo7O1XGEU5wFnoURYVyItEqFgK8ZxpcQpj8Aq2GN6g==</latexit>

0.6

0.6

0.4

0.4

0.2

0.2

0

0

0

5

10

15

2

20

4

l

6

8 10 12 14 16 18 20

l1/↵ = 0.02, 0.01, 0.005, 0.001, 0.0005, 0.00025<latexit sha1_base64="jPL88GvWQ/jap1qiqxUGT7mfcpY=">AAACknichVHLSsNAFD2Nr1ofrY+F4KZYFBdab6pFEQQfGxcufFUFlZLEaQ2mSUjSghZ/wK0LF64URMQvcKsbf8CFnyAuFdy48CYNiIp6h7lz7pl77tyZUW1Ddz2ix4hUV9/Q2BRtjrW0trXHEx2da65VdjSR0yzDcjZUxRWGboqcp3uG2LAdoZRUQ6yre3P+/npFOK5umavevi22S0rR1Au6pnhM5RPD8siWYti7yhSlKTPETvYdZQNfw2FAmWw+kQoQW/InkEOQQmiLVuISW9iBBQ1llCBgwmNsQIHLYxMyCDZz26gy5zDSg32BQ8RYW+YswRkKs3vsixxthqzJsV/TDdQan2LwdFiZRD890BW90D1d0xO9/1qrGtTwe9nnVa1phZ2PH/WsvP2rKvHqYfdT9WfPHgqYCHrVuXc7YPxbaDV95eDkZWVyub86QOf0zP2f0SPd8Q3Myqt2sSSWTxHjD5C/P/dPsJZJy6PpzNJYano2/IooetGHQX7vcUxjHovI8bnHuMEt7qRuaVKakeZqqVIk1HThi0kLHzQilCI=</latexit>“penetration depth”<latexit sha1_base64="13Jt2AvwoBcVr58dGy40xBWyYrw=">AAAChHichVHLLgRBFD3ae7wGG4mFjgkhkUmNR4iFCBtLr0GiZdLdChU13ZXumgkmljZ+wMKKRERs+QIbP2DhE8SSxMbCnZ5OBMGtVNWpU/fcOlXlKClCzdhjlVFdU1tX39CYaGpuaW1LtneshH4hcHnW9aUfrDl2yKXweFYLLfmaCriddyRfdXZny/urRR6EwveW9b7iG3l72xNbwrU1Ublkj7UnBixbqh170LRU4Cvtm5b0zApn5pIplmZRmD9BJgYpxDHvJy9hYRM+XBSQB4cHTVjCRkhtHRkwKOI2UCIuICSifY5DJEhboCxOGTaxuzRu02o9Zj1al2uGkdqlUyT1gJQm+tgDu2Iv7J5dsyf2/mutUlSj7GWfZqei5SrXdty19PavKk+zxs6n6k/PGluYiLwK8q4ipnwLt6IvHpy8LE0u9pX62Tl7Jv9n7JHd0Q284qt7scAXT5GgD8h8f+6fYGU4nRlJDy+MpqZn4q9oQDd6MUDvPY5pzGEeWTr3CNe4wa1RZwwZI8ZYJdWoijWd+BLG1AeHR5Tq</latexit>⇠(↵) / ln ↵

SciPost
MorePhysics“terraces” under larger constraints

Submission↵ = 4000<latexit sha1_base64="ghbOttJ0+kI+w09oJPANmemhYCs=">AAACbnichVHLSsNAFD2Nr1ofrQoiiFgsla7KbRUUQSi6canVPqCKJHG0oWkSkrSgxR9wLy4ERUFE/Aw3/oCLfoK4ERTcuPA2DYiKesNkzpy5586ZO4qla45L1AxIHZ1d3T3B3lBf/8BgODI0nHfMmq2KnGrqpl1UZEfomiFyrubqomjZQq4quigoleXWfqEubEczjQ133xJbVXnP0HY1VXaZKm3KulWWF2eJaDsSoyR5Ef0JUj6IwY9VM3KNTezAhIoaqhAw4DLWIcPhr4QUCBZzW2gwZzPSvH2BQ4RYW+MswRkysxX+7/Gq5LMGr1s1HU+t8ik6D5uVUcTpgW7ohe7plh7p/ddaDa9Gy8s+z0pbK6zt8NHY+tu/qirPLsqfqj89u9jFvOdVY++Wx7Ruobb19YOTl/WFbLwxTZf0xP4vqEl3fAOj/qperYnsKUL8AKnv7f4J8ulkaiaZXpuNZZb8pwhiHFNIcL/nkMEKVpHzOnaMM5wHnqVRaUKabKdKAV8zgi8hJT4AWBKNSw==</latexit>

spin

bond

Figure 8: Glass order parameter functions under stronger constraints ↵ = 4000. In thisexample L = 20 and the centeral layers at l = 8, 9 still remain in the liquid phase. a),b)

Summary: depth dependent free-energy landscapeenergySciPostPhysics landscape become simpler and ﬂatterSubmission
at deeper layersimplications of these in the last section.

5

Fluctuating boundary

Fluctuating boundary

SciPost Physics

Subm

we consider cases such that the overlap distribution of the spins on the input lay
exibit hierahircal, stucture as described by the Parisi’s matrix Eq. (20) (Fig. 10

5.1

One RSB type boundary

Here we consider the simplest case of ’1RSB’.
( structure
Figure 10: Fluctuating input layer with hierarhical overlap
r onmthei <xinputNow weanalyzethecaseoffluctuatingboundary:
spin
configuration
boundaries“1RSB" boundaryqi (0) =are allowed to fluctuate during learning following certain probabilityHere
1 distributions.
mi > xinput
14

It can be seen that alignment can be induced from the boundary.
RSB" boundaryTwo“fullmachines: those subjected to similar data are likely to be similar.... thof generalization

in the case of quenched (RS) boundary condition. We limit ourselves to ↵ such that⇠glass(↵) < L/2 so that the we have a liquid phase at the center of the system. In this“full RSB”
boundary
circumstance the boundary condition on the other side qab (L) is irrelvant.
spin↵ = 50<latexit sha1_base64="FBq/8SR3yJLinzqr1B/wPcwm3ec=">AAACbHichVG7SgNBFD1ZXzG+4qMQRBCDYhVuoqIIgmhj6SsaSUR210lcsi92N4EY/AFbCwu1UBARP8PGH7DIJ4hgE8HGwpvNgmhQ7zAzZ87cc+fMjGLrmusRVUNSS2tbe0e4M9LV3dPbF+0f2HatoqOKlGrplpNWZFfomilSnubpIm07QjYUXewohZX6/k5JOK5mmVte2RZ7hpw3tZymyh5Tu1lZtw/lxVnaj8YoTn6MNYNEAGIIYs2K3iKLA1hQUYQBARMeYx0yXG4ZJECwmdtDhTmHkebvCxwjwtoiZwnOkJkt8JjnVSZgTV7Xa7q+WuVTdO4OK8cwQU90RzV6pHt6po9fa1X8GnUvZZ6VhlbY+30nw5vv/6oMnj0cfqn+9Owhh3nfq8bebZ+p30Jt6EtHZ7XNhY2JyiRd0wv7v6IqPfANzNKberMuNs4R4Q9I/HzuZrCdjCem48n1mdjScvAVYYxgHFP83nNYwirWkOJzDZziApehV2lIGpFGG6lSKNAM4ltIk59I7IzY</latexit>↵ = 4000<latexit sha1_base64="xAqxpDAKjT5MIIgYeF0EANMJjL8=">AAACcHichVHLSsNAFD2N7/qquim48FEUcVFuVFAEQXTjsrW2FaqUJE7bYJqEJC1q8QfciwtBURAUP8ONP+DCTxB3Krhx4W0aEC3qHZg5c+49lzN3VNvQXY/oMSS1tLa1d3R2hbt7evv6IwODGdeqOJpIa5ZhOZuq4gpDN0Xa0z1DbNqOUMqqIbLq7mo9n60Kx9Utc8Pbt8V2WSmaekHXFI+p3JZi2CVlaY6I8pEYxcmP0WYgByCGIBJW5Bpb2IEFDRWUIWDCY2xAgcsrBxkEm7lt1JhzGOl+XuAQYdZWuEpwhcLsLu9FvuUC1uR7vafrq/e4wmKlxV0OMYoJeqAbeqV7uqUn+vi1W83vUnezz6fa0Ao7338UTb3/qyrz6aH0pfrTtYcCFny3Oru3fab+Dq2hrx6cvKYW1ydqk3RJz+z/gh7pjl9gVt+0q6RYP0WYv0D+OfBmkJmJy7PxmeRcbHkl+IxODGMcUzzxeSxjDQmk/Ykd4wznoRcpKo1IY41SKRRohvAtpOlPiACNzg==</latexit>bondTeacher student setting
a statistical inference problem (SIP)

1) Training
randomtraining
data

Randomly quenchedteacher
Jij

student
Jij2) TestOut put of teacher

Student is forced toreproduceteacherʼs output

Randomly quenchedrandomtest
data

teacher
Jij
CompareNow quenchedstudentJijteacher+student machine (inference)
input

outputrandom teacher machine
overlapr, R<latexit sha1_base64="gAZkjdb1BwUuoMmDyRStNaIBVEw=">AAACaXichVG7SgNBFD1Z3/GRRBvRJhgUCwmTKChWQRtLjSYKGsLuOjFLNjvL7iQYgz9gYRvBSkFQ/Awbf8DCTxDLCDYW3t0siAb1DsycOfeeO2dmNNs0XMnYc0jp6e3rHxgcCg+PjI5ForHxvCtqjs5zujCFs6epLjcNi+ekIU2+ZztcrWom39Uq615+t84d1xDWjmzYvFBVjyyjZOiq9ChnIZ4tRhMsyfyId4NUABIIYlNEb3GAQwjoqKEKDguSsAkVLo19pMBgE1dAkziHkOHnOU4RJm2NqjhVqMRWaD6i3X7AWrT3erq++pgqBCkFdTlFHLPsid2xNntk9+yFffzarel38dw0aNU6Wm4XI2eT2+//qqq0SpS/VH+6lihhxXdrkHvbZ7x76B19/aTV3l7Nzjbn2DV7Jf9X7Jk90A2s+pt+s8WzlwjTF6R+Png3yKeTqcVkemspkVkLPmMQ05jBPL34MjLYwCZydG4Z52jhIvSqxJRJZapTqoQCzQS+hZL4BMTmizU=</latexit>

student machine
Review: Zdeborova-Krzakala (2017)
1+s replicar, qRS solution

Bayes optimal, Nishimori condition

R, Q

1

<latexit sha1_base64="3HEYnsHWRMFn2phEp/1T0/6t39M=">AAACaXichVFNLwNBGH66vqo+2nIRLo0NcZBmtiTEqeHiqKVIENldg4n9srttVNM/4OBaiROJhPgZLv6Ag58gPVbi4uDd7SaC4J1k5pnnfZ93npnRHEN4PmPPMamjs6u7J96b6OsfGEym0kPrnl12dV7SbcN2NzXV44aweMkXvsE3HZerpmbwDe1oKchvVLjrCdta86sO3zHVA0vsC131A8qdzhzvpmSWZWFkfgIlAjKiWLFTt9jGHmzoKMMEhwWfsAEVHo0tKGBwiNtBjTiXkAjzHHUkSFumKk4VKrFHNB/QbitiLdoHPb1QfUIVNilt6lJHBhPsid2xFntk9+yFvf/arRZ2CdxUadXaWu7sJs9GVt/+VZm0+jj8VP3p2sc+5kO3gtw7IRPcQ2/rK6eN1upCcaI2ya5Zk/xfsWf2QDewKq/6TYEXL5GgL1C+P/hPsJ7LKjPZXGFWzi9GnxHHGMYxRS8+hzyWsYISnXuIczRwEWtKaWlEGm2XSrFIM4wvIckfAzOLVA==</latexit><latexit sha1_base64="owAZR8mkXjRZkkQggK8+U2zvTMw=">AAACaXichVG7SgNBFD1Z3/GRqE3QJhgUCwkTFRQr0cbSRPMADbK7jmbIZnfZnQQ1+AMWthGsFATFz7DxByzyCZIygo2FdzcLokG9AzNnzr3nzpkZzTaEKxlrhpSe3r7+gcGh8PDI6FgkOj6Rc62qo/OsbhmWU9BUlxvC5FkppMELtsPVimbwvFbe9PL5GndcYZm78tTmxYp6bIojoavSozIL8fRBNMGSzI94N0gFIIEgtq3oPfZxCAs6qqiAw4QkbECFS2MPKTDYxBVRJ84hJPw8xznCpK1SFacKldgyzce02wtYk/ZeT9dXn1CFRUqLupwjjln2wh5Ymz2zR/bKPn7tVve7eG5OadU6Wm4fRC5iO+//qiq0SpS+VH+6ljjCqu9WkHvbZ7x76B197azR3lnLzNbn2C1rkf8b1mRPdAOz9qbfpXnmGmH6gtTPB+8GucVkaim5mF5OrG8EnzGIacxgnl58BevYwjaydG4Jl2jgKtRSxpWYMtUpVUKBZhLfQkl8AoJEixQ=</latexit>

0.8
0.6
0.4

↵ = 0.02↵ = 0.01<latexit sha1_base64="eTerCyt7ZDpu/wP9SPjlIKC8w8w=">AAACcHichVFNSwJRFD1OX2ZfVhuhRR9iRAt5Y0ERBFGblmppgUrMTE8dmi9mRsnEP9A+WgRFQVD0M9r0B1r0E6JdBm1adB0HoqS6D94977x7LufdJ1ua6riMPQWEru6e3r5gf2hgcGh4JDw6lnXMiq3wjGJqpr0rSw7XVINnXNXV+K5lc0mXNb4jH2y07neq3HZU09h2axYv6FLJUIuqIrlE5fKSZpWlVRZn4l44SsmLqU4g+iAKP5Jm+AZ57MOEggp0cBhwCWuQ4NDKQQSDRVwBdeJsQqp3z9FAiLQVquJUIRF7QHuJTjmfNejc6ul46kOqMElpUpcGphBjj+yWNdkDu2PP7OPXbnWvS8tNjbLc1nJrb+Q4svX+r0qn7KL8pfrTtYsilj23Krm3PKb1DqWtrx6dNrdW0rH6LLtiL+T/kj2xe3qBUX1TrlM8fYYQfYH4c+CdIJuIiwvxRGoxurbuf0YQE5jBHE18CWvYRBIZb2InOMdF4FWICJPCdLtUCPiacXwLYf4TfeaNyQ==</latexit><latexit sha1_base64="m2pOZmsFsUYbMrWxUlOFapurQtg=">AAACcHichVHLSsNAFD2Nr1pfVTcFFz6KIi7KtAqKIIhuXLbVPqAWSeK0huZFkhZr6Q+4FxeCoiAofoYbf8CFnyDurODGhTdpQLSod2DumTP3XM7ckUxVsR3GngJCV3dPb1+wPzQwODQ8Eh4dy9pG1ZJ5RjZUw8pLos1VRecZR3FUnjctLmqSynNSZdO9z9W4ZSuGvuPUTV7UxLKulBRZdIgq7IqqeSCusRhL7IWjlLyY6gRxH0ThR9II32AX+zAgowoNHDocwipE2LQKiIPBJK6IBnEWIcW752giRNoqVXGqEImt0F6mU8FndTq7PW1PfUgVBikN6tLEFGbZI7tlLfbA7tgz+/i1W8Pr4rqpU5baWm7ujRxHtt//VWmUHRx8qf507aCEFc+tQu5Nj3HfIbf1taPT1vZqerYxx67YC/m/ZE/snl6g197k6xRPnyFEXxD/OfBOkE3E4ouxRGopur7hf0YQE5jBPE18GevYQhIZb2InOMdF4FWICJPCdLtUCPiacXwLYeETf+iNyg==</latexit>

0.2
0

0

5

10

15

20

l

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
2

4

6

8 10 12 14 16 18 20lSymmetry breaking ﬁeld: remanent bias in the liquid phaseO(ln(N )/N )<latexit sha1_base64="fa2RXETstgtUqCDhusVSqSei5LY=">AAACm3ichVHBShtRFD2Otmq0NepGKEIwWCKU8cZEYlwFsylF1MRGhSSEmfEZByczw8wkqEN+wB8Q6spCocWfKHTTuu/CTyguFdx00TuTRBFpex+8d9+559x33nuqbeiuR3TVJ/UPPHs+ODQcGRl98XIsOj6x5VpNRxMlzTIsZ0dVXGHopih5umeIHdsRSkM1xLZ6kA/q2y3huLplvveObFFtKHVT39M1xWOoFk37lbBJ2amrVT8pUxhvSF7MLCVpkZMO0l5PVAwzsTY3vzbXrkXjPWaM5AxRNp2N9ZixXimObmxY0c+oYBcWNDTRgIAJj3MDClweZSRBsBmrwmfM4UwP6wJtRFjbZJZghsLoAc913pW7qMn7oKcbqg+ZYbHS4i5txDBLP+kL3dB3uqBf9Puv3fywS+DmiFe1oxV2bexkavPuv6oGrx72H1T/dO1hD0uhW53d2yES3EPr6FvHpzeby8VZ/zV9pGv2f05X9I1vYLZutU8FUTxDhL+A5GwQqfsHf5psLcjJlJwqpOO5le5nDOEVZpDgF88gh7fYQInP/YCv+IFLaVrKS++k1Q5V6utqJvEopNIfg96Zwg==</latexit>

Simulations of learningrandom inputs/outputs
S0,1

S2,1

S1,1

<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit>

S3,1<latexit sha1_base64="(null)">(null)</latexit>

S0,2

S1,2

<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

S0,3Monte Carlo simulation<latexit sha1_base64="(null)">(null)</latexit>

Hamiltonian
M
XX

H=

⌅ µ=1

<latexit sha1_base64="vssSqFWBU3tPq4vKkCsQnjc58XU=">AAAlkXic3VrNbxtFFB++wXz0S0hIXCxCUCuVyCkgClKkJo2ThqaNGydp2rqNdp31h7K2N7u2a8eEP4B/gAMnkDggzvwFXPgHOCCucEAci8SFA++9mfXO164tBKXUT/bOvvm9N+/NvHk7M2s38JtRt1D48Yknn3r6mWefe/6F3IsvvfzKiZOnTu9EnV5Y9barHb8T7rpO5PnNtrfdbXZ9bzcIPafl+t5N9+Ay1t/se2HU7LS3usPAu9ty6u1mrVl1usDaO7lxZeHtStRr7Y0qru9UD6LDnhN6x3nBa/UW5o/vja4d7+Qrvlfrng3vIfNYg1fCZr3RPZfL5fZOzhTmCvTJjwvzemGGiU+pc+r0q6zC9lmHVVmPtZjH2qwLZZ85LAK6w+ZZgQXAu8tGwAuh1KR6jx2zHMj2AOUBwgHuAfzW4e6O4LbhHnVGJF2FVnz4hiCZZ7OFHwpfFx4Uvi98U/i18GeqrhHpQFuGcHW5rBfsnfj0tfIfE6VacO2yRiKVIRHBnUOehfD1J/jXZTV2kfxqgp8BcdDjKm+rf/TZg/KHm7OjtwpfFn4DX78o/Fj4Drxt93+vfnXD2/w8w5LY6hBqa/BFjdP3ThqyDvoc4DZoBAfsvDYiaXIuUCsT0QK+T5ERgrb7GT03nbbpvUcPWtIop+HQtgN2pFgW83y4utQ3IfQixkGD+gnj9jxxuVd4F0ApAjzKdmC0cezr1JMYOdUMCyKQaYk2Qphx6UjePzk2m6GrQRbhLHUpVqOxRCBQ2N6QonkEPESirYhC3KNKf8e2PNAyK7IVtsauw/c2fDeohHz8XaH7LUIiPTzbHl6/YVzXKCa7UGpRRuhCjCxQGWPqA/A8yXH7hMJYiShzBZDpH6fYyBtUhli4xpYgFtYpKnbYItukko3+Tdsedr9hbPiU8Sow4gv069OTwKHcGte5VOfSnVpTp5o6PW1aWt0+1e1THkI5uTWP6jC6MGv6FJ+y7BHVH1labAhJnX9A/APK0IHWmk91PlnpgkWqZItq8Vklc9vEbWvcAXEHYLPMDYgbAFduNSIu+lc3+qZLdeiDqr9G/D49D/gTWW2nYdQntVWqqxI3p0hFwr5I0zckfi91FJCbtGf3ZJUQq+MIkFteprrlcQTIclvCF7zy0Uzq1qlmPWW8dql2V/OlRNyS5ntZeFC22r4iZPR+LokeKxk9dov420qPqXHtST2WHt8hoUJ6Xqtx0RCREfeKXItriwW6tiFS8Dkut90RkdyhtXVdjAXOvxrJ4BMfV6VzUJoDvce01ojrUQLrsa4+rs8pOkpAiKmIp0Ue7lUtRSAVUdQQ14BUxDVC5CXMDpCK2RHWJJjLQCrmstbSCpCKWDFaugKkYq5oWm4CqYibhi1bQCpmS9NyHUhFXDdsWQJSMUualjUgFbFm2LIJpGI2NS3rQCpi3bDlIyAV85GB2QBSMRuGNctAKmZZs2YRSEUsaohVIBWxathSBlIxZcOWq0Aq5qrW0jaQitg2WroBpGJuGJhdIBWza1hzC0jF3NKsuQ2kIm5b5uSmmNeYTVo02h5pOa9FTktBrdFegqNUfQ0gjnThPk+7J9W7RWnMOGbRwBwCyYhDw/8lKd45Zsnaj7sKZtfAlKSsxDElA4MrAhkRGNb0gWRE39AxAJIRAwMxpF1ighgaiCMgGXFk2OEQyRhHiwqXSEa4GqKRMoZqHllTEGYe6QHJiJ4l9nB3P6Jrm3gh7ZnbFIGy33iaEGvrQN/KyJHQrEogokZ43IMM4HckeLqdHWGDrle3NaS9TV94hE9nPspqtuK6ltmeNFdQoxm/EWGjKZAuPc2TdiviTCKnWRjRnKmMTy34icjI2nZgRZpxHUF7NmRoQR5SL1foDMMnfro1pj3TyAZW2XAq2dAq27NiexZk34rsW5ANK7JhQQ6syIEFObQihxZk3YqsW5HT9Fvd2m9NK9bMAgM60zJbQb6Z0W06lwzcjhVnrvA64xlh5gwerzaJeoaE2Yud8QyxS5jzpDOefXYJ2xxctXq8akEWrciiJfdiPsMTmpBObeNzxxk2D98LipYq7TIQ5YEG+6kGJ+4B7iZckBiArgpZ0aFIf5Pq90FTJE4Oh9JpYp7aPhb5EM+T0DLMCyrmAmDeJG+yLEnzD/fLaAtvayT0TdI2jV/xc+bx8Sjd4jSv/t8ePtreHk8xe7lsMoe5FlylV8Q7q/jp1AfJnrAYTwrwdKMBevKwc79IPdUFyTy1F1AW4DXvkQ0B2Rb3ynlpRP7rM8ms08rJmHhs1iFj4p68CLuhIsP9Il4vw94e74qEWIR6fC+Ad3jNU3lTSG5II/3P2Pbf9Zscd12IM75ejSMmeY+lrsB92ofw06WaOEV16K2Wb6zvQ4FNtHJegtb3rPyUQp7R8ruJ+MxClimLPX66TLzjV/d1XGYH7qtQMk8FOhrC9lwvj2cwSiLvvvSGEfc+KOfQavIOzKW7GT3HM1I+o7f2lByZ3aZHqyB8G3+X1gmHMMs/YWdFhqmN89A54XWaHldoist18n1E3B6tI3iGivdp6fYkejzBtemYbEs7xZq2iFhH2l9nW9O22KNrybLnPq39ealJcdMgb7Lbvk8IWSpZQWe3t0WrTnXvuyWtRNPkWtRL/PxHlZZrJkVxlU7obGd2k6K/RucAVSkecS4FNJccyjzh+J8f8vo0kuLUPD8satpHY2tklKqR78aXNZS+Q+Ko/YmeuXR64Sl+xX2b/FNiJOESb7L08hOJSVpjlKozpGi26+WZLNGbnOEkGrLsatK74rKIIn6HOeljkU/vQRmz08dsgZ/d752cMf5LZBZ2LszNvzN34ca7M5eWxP+MnmevszcgW82z99kleDqX2DbY8y37if3Mfpk9M/vB7KVZgX3yCSFzhimf2at/AYvumGc=</latexit>

Gap

S0,N

S1,N

µ
V r⌅

soft-core potential

N
X
1 i µ
µ
µ
p
r⌅ =
J⌅ S⌅(i) S⌅
N
i=1

e

V (h)

=e

✏h2

✓(h)

<latexit sha1_base64="Pm1bLs+NvJhfEezLaXpbhuTJH3k=">AAACunichVFNSxtBGH5cq9X4Fe1F6GUxROzBMBsLllIhpJce/UoUXA2z68QM2S93JwG77B/oH/DgqYVSSn9Cj734Bzx49KgeI/TSg282C9JK23eYmXeeeZ53npmxAkdGirHLIW34ycjo07Hx3MTk1PRMfnauHvmd0BY123f8cNfikXCkJ2pKKkfsBqHgruWIHav9tr+/0xVhJH1vW50EYt/lR55sSpsrghr5mjiIl01LKK7Xl1ovEn3tATBFEEnH9/TWQVxOElO1CCWSbh4fd/ihnrLWzGbI7dhI4nYjNkNXrybbSSNfYCWWhv44MbKkgCzW/fwXmDiEDxsduBDwoCh3wBFR24MBhoCwfcSEhZTJdF8gQY60HWIJYnBC2zQe0WovQz1a92tGqdqmUxzqISl1FNkF+8p67Jx9Y9fs119rxWmNvpcTmq2BVgSNmQ/zWz//q3JpVmg9qP7pWaGJV6lXSd6DFOnfwh7ou+9Pe1uvN4vxIvvEbsn/R3bJftANvO6d/XlDbJ4hRx9g/Pncj5N6uWSslMobLwuVavYVY3iOBSzRe6+igndYR43O/Y4r3OBWe6NZmtTaA6o2lGme4bfQ1D0pJKq0</latexit>

1
=
kB T

<latexit sha1_base64="I3kOm1j0xPHvbkwIKL9Rjj68l4U=">AAAC8nichVHNThRBEK4Z/MH1hwUvEi8bNhi8bHrQBGNCQvRCPBBgXSBh2ElP04udnT+6ezZip1/AxLMJnoAYY/QtvPgCHngE43FJuHiwdnYSI6BWp7urv6rvq+ruMIuE0oQcO+7IpctXro5eq1y/cfPWWHV8Yk2luWS8xdIolRshVTwSCW9poSO+kUlO4zDi62H36SC+3uNSiTR5rvcyvhXTnUR0BKMaoaD6RraNH+c2MH4YUdZVuzmV3M7XfJXHgRHznm2bJet3JGXGs8ZXu1IjYH3NX+qivpF825pnZxTaRljbvEh8Rty3tQsjNqjWSYMUVjvveKVTh9KW0+oH8GEbUmCQQwwcEtDoR0BB4dgEDwhkiG2BQUyiJ4o4BwsV5OaYxTGDItrFdQdPmyWa4HmgqQo2wyoRTonMGkyTb+Qj6ZOv5BP5Tn7+VcsUGoNe9nAPh1yeBWOv7zRP/8uKcdfw4jfrnz1r6MCjoleBvWcFMrgFG/J7r972m49Xp809ckh+YP8H5Jh8wRskvRP2foWvvoMKfoB39rnPO2uzDe9BY3blYX3hSfkVo3AXpmAG33sOFmARlqGFdfvOpDPl1F3t7rsH7tEw1XVKzm34w9zPvwCO18S8</latexit>

Dynamical variables
with random boundariesµ
iJ⌅ , S⌅(i = 1, 2, . . . , N )(µ = 1, 2, . . . , M )S0,i , SL,i(i = 1, 2, . . . , N )(⌅ = 1, . . . , LN )

Relaxation of autocorrelation functions

Cbond (t, ⌅) =<latexit sha1_base64="c/nVoVvOOj8RyKmvCdzJ4IimyBQ=">AAAlunic3VpLbyPHEW45dmzTj107MGDAF8Kygl1AEEabBHEMC7AelKy1dpcrSlrtLtfCDDV8QEPOaIbkkqLpH+A/kENOCZCD4Z+RS/6AD774HuToAL74kKrqHk6/ZkgEydoxC+T0VH9VXdVdXdPdQy8KOknfcb5Zeu4Xz7/wyxdfern0yquvvX7t+htvniThIG74x40wCONTz038oNPzj/udfuCfRrHvdr3Af+BdbGP9g6EfJ52wd9QfR/6TrtvqdZqdhtsH1tn10fbZpB53y17YO5/e6K/WvcBtXCSXAzf2b27Um7HbmKxPJ3en5Xoy6J5NOhvr00/hth64vVbgl2+DuCQCdZ3pDeemld2/Wa7HJFYqlc6uLztrDn3Ks8K6Xlhm4lMN33jzLVZn5yxkDTZgXeazHutDOWAuS4Aes3XmsAh4T9gEeDGUOlTvsykrgewAUD4gXOBewG8L7h4Lbg/uUWdC0g1oJYBvDJJltuJ87XzpfOf83fnK+YfzQ66uCelAW8Zw9bisH51d++Lt2vdzpbpw7bN2JlUgkcCdS57F8A3m+NdnTfY++dUBPyPioMcN3tbw6o/f1T44XJn82vmL80/w9c/ON87fwNve8F+Nv973D/9UYElqdQy1TfiixsV7Jw/ZAn0ucNs0giO2qo1InpwH1C1EdIEfUGTEoO1pQc8tpm1x79GDrjTKeTi07YJdKZalvACuHvVNDL2IcdCmfsK4XSUu9wrvIiglgEfZEEYbx75FPYmR0yiwIAGZrmgjhhmXj+T9U2IrBbraZBHOUo9iNZlJRAKF7Y0pmifAQyTaiijE/VTpP7GtDLTDKmyX7bO78H0E33tUQj7+7tL9ESGRnp1tz67fMK6bFJN9KHUpI/QhRjaojDH1B/A8y3HnhMJYSShzRZDpf06xUTaoBrFwh21BLBxQVJywTXZIJRv9L2171v2GsRFQxqvDiG/Qb0BPApdya1rnUZ1Hd2pNi2pa9LTpanXnVHdOeQjl5NZ8qsPowqwZUHzKsldUf2VpsS0kdf4F8S8oQ0daawHVBWSlBxapkl2qxWeVzO0Rt6dxR8Qdgc0yNyJuBFy51YS46F/L6Js+1aEPqv4m8Yf0POBPZLWdtlGf1TaorkHckiKVCPsSTd+Y+IPcUUBu1p7dkz1C7M0iQG55h+p2ZhEgyx0JX/DKRzOrO6Cag5zxOqXaU82XKnGrmu814UHNavuukNH7uSp6rGr02EPiHys9psa1L/VYfnzHhIrpea3GRVtERtorci2uLTbo2oNIwee43HYoIjmktXVLjAXOvybJ4BMfV6VrUFoDvVNaa6T1KIH1WNea1ZcUHVUgxNTF06IM96qWCpCKqGiIO0Aq4g4hyhLmBEjFnAhrMsw2kIrZ1lraBVIRu0ZLHwOpmI81LQ+AVMQDw5YjIBVzpGm5C6Qi7hq2bAGpmC1Nyz6Qitg3bDkEUjGHmpYDIBVxYNhyG0jF3DYw94BUzD3Dmh0gFbOjWbMJpCI2NcQekIrYM2ypAamYmmHLJ0Aq5hOtpWMgFXFstHQfSMXcNzCnQCrm1LDmIZCKeahZ8whIRTyyzMlDMa8xm3RptH3SsqpFTldB7dNegqNUfW0gjvTgvky7J9W7TWnMOGbTwFwCyYhLw/8tKd45Zsvaj6cK5tTAVKWsxDFVA4MrAhkRGdYMgWTE0NAxApIRIwMxpl1ihhgbiCsgGXFl2OESyRhXiwqPSEZ4GqKdM4ZqHtlXEGYeGQDJiIEl9nB3P6Frj3gx7Zl7FIGy33iakGoLoW9l5ERoViUQ0SQ87kFG8DsRPN3OUNig69VtjWlvMxQe4dOZj7KarbiuHXYmzRXUaMZvQthkAaRHT/Os3bo4kyhpFiY0Z+qzUwt+IjKxth1ZkWZcJ9CeDRlbkJfUy3U6wwiIn2+Nac8ispFVNl5INrbKDqzYgQU5tCKHFmTbimxbkCMrcmRBjq3IsQXZsiJbVuQi/day9lvHijWzwIjOtMxWkG9mdJvOLQN3YsWZK7xwNiPMnMHj1SbRKpAwezGczRC7hDlPwtnss0vY5uCe1eM9C7JiRVYsuRfzGZ7QxHRqm547LrN1+N5StDRol4EoHzTYTzU4cQ9wN+GBxAh01cmKkCL9Pao/B02JODkcS6eJZWp7KvIhniehZZgXVMwtwLxH3hRZkucf7pfRFt7WROibp20Rv9LnzM/Ho3yL87z6//bwp+3tdIHZy2WzOcy14Cq9Lt5ZpU+nIUgOhMV4UoCnG23QU4ad+/vUU32QLFN7EWUBXvM7siEi29JeWZVG5Mc+kyw6rZyPScfmADIm7skrsBuqMNwv4nUb9vZ4VyHEJtTjewG8w2uZyodC8p400v8d2368fpPjrg9xxteracRk77HUFXhA+xB+utQUp6guvdUKjPV9LLCZVs7L0PqelZ9SyDNafjeRnlnIMjWxx8+XSXf86r6Oy5zAfQNK5qlAqCFsz/XabAajJPKeSm8Yce+Dci6tJh/DXHpS0HM8I5ULeutMyZHFbfq0CsK38U9onXAJs/xzdkNkmOYsD90UXufp8YSmtNwi3yfEHdA6gmeodJ+Wb0+mxxdcm475tvRyrOmJiHWl/XWxNT2LPbqWInue0tqflzoUN23yprjtp4SQpbIVdHF7R7TqVPe+R9JKNE+uS73Ez39UablmXhQ36ITOdmY3L/qbdA7QkOIR51JEc8mlzBPP/vkhr08TKU7N88OKpn0ys0ZGqRr5bnxHQ+k7JI46n+uZR6cXvuJX2rfZPyUmEi7zpkgvP5GYpzVFqTpjima7Xp7JMr3ZGU6mociuDr0rroko4neYkz4T+fRTKGN2+oxt8LP7s+vLxn+JzMLJrbX136zduv/b5Y+2xP+MXmLvsHchW62z37OP4OlcZcdgz7dLzy+9tvT6yocr3kpn5YJDn1sSMr9iymel/2+LVqXS</latexit>

N
X
1

N i=1

M

i
i
hJ⌅
(0)J⌅
(t)i

Cbond (t, l)

1 X µµCspin (t, ⌅) =hS⌅ (0)S⌅
(t)i
M µ=1
<latexit sha1_base64="CbseTssj5vdYGyZHi0M2r6lTo5Y=">AAAlwHic3VrNbiPHEW45ieMwib22EyBALoRlBbuAIIw2CeIEEGD9ULJs7S5XlLTaXa6FGWr4A81wRjMklxRNP4BfwAefEiCHII+RS14gB99zCXJ0gFx8cFV1D6f/ZkgEydqxCuTMVH1VXdVdXdPdlBcHvXTgOJ+vvPSd737v5e+/8oPKD3/041dfu/X6G2dpNExa/mkrCqLk3HNTP+j1/dNBbxD453Hiu6EX+I+8q12UPxr5SdqL+ieDSew/C91Ov9futdwBsC5ufbx7MW0mYTWNe/3Z7cF60wvc1lV6PXQT/85Ws524renmbHpvVm2mwxCw4XBrc/YhMJqB2+8EfrUBTEkJZICZ3XbuFAgGd6rNhFQrlcrFrVVnw6G/6vxmU79ZZeKvHr3+xk9Zk12yiLXYkIXMZ302gPuAuSwFeso2mcNi4D1jU+AlcNcjuc9mrAK6Q0D5gHCBewXfHXh6Krh9eEabKWm3oJUAPgloVtma8zfnT84Xzl+dPzv/cL4stDUlG+jLBK4e1/Xji9c++Vnj3wu1QrgOWDfXKtFI4cmlyBL4BAviG7A2e4fi6kGcMXEw4hZva3Tz6ReN3x2vTX/h/MH5J8T6e+dz5y8QbX/0r9YfH/rHn5V4knmdgLQNH7S4fO8UITtgzwVul0ZwzNa1ESnS84DCUkQI/IAyIwFrz0t6bjlry0ePEYTSKBfh0LcrdqN4lvECuHrUNwn0IuZBl/oJ83aduDwqfIrhLgU86kYw2jj2HepJzJxWiQcp6ISijQRmXDGS90+FrZXY6pJHOEs9ytV0rhELFLY3oWyeAg+R6CuiEPdNpf/EtyrQHquxfXbI7sPnCXwe0B3y8Xufnk8IifTifHtx/YZ53aacHMBdSBVhADmyRfeYU7+FyPMad0kozJWUKlcMlf7blBtVgxqQC/fYDuTCEWXFGdtmx3Rno/+lby+63zA3Aqp4TRjxLfoO6E3gUm3NZB7JPHpSJR2SdOhtE2qyS5JdUh1CPbk1n2SYXVg1A8pPWfeG5DeWFrtCU+dfEf+KKnSstRaQLCAvPfBI1QxJiu8qmdsnbl/jjok7Bp9lbkzcGLhyqylxMb6O0TcDkmEMqv028Uf0PuBvZLWdriHPpS2StYhbUbRS4V+q2ZsQf1g4CsjN27NHckCIg3kGyC3vkWxvngGy3omIBa98NHPZEUmOCsbrnKTnWix14ta12BsigobV932ho/dzXfRY3eixx8Q/VXpMzWtf6rHi/E4IldD7Ws2LrsiMrFdkKa4ttujah0zB97jcdiQyOaK1dUeMBc6/NungGx9XpRtwtwF2Z7TWyOSogXKUdebyimKjDoSYpnhbVOFZtVIDUhE1DXEPSEXcI0RVwpwBqZgz4U2O2QVSMbtaS/tAKmLfaOk9IBXznmblEZCKeGT4cgKkYk40K/eBVMR9w5cdIBWzo1k5BFIRh4Yvx0Aq5lizcgSkIo4MX94HUjHvG5gHQCrmgeHNHpCK2dO82QZSEdsa4gBIRRwYvjSAVEzD8OUDIBXzgdbSKZCKODVaegikYh4amHMgFXNuePMYSMU81rx5AqQinljm5LGY11hNQhptn6ysa5kTKqhD2ktwlGqvC8SRHjxXafekRrctjRnHbBuYayAZcW3EvyPlO8fsWPvxXMGcG5i6VJU4pm5gcEUgI2LDmxGQjBgZNsZAMmJsICa0S8wREwNxAyQjbgw/XCIZ42pZ4RHJCE9DdAvGUK0jhwrCrCNDIBkxtOQe7u6ndO0TL6E9c58yUI4bTxMyaxH0rYycCsuqBiLahMc9yBi+p4Kn+xkJH3S7uq8J7W1GIiJ8O/NRVqsVt7XHLqS5ghbN/E0Jmy6B9OhtnrfbFGcSFc3DlOZMc35qwU9Epta2YyvSzOsU2rMhEwvymnq5SWcYAfGLvTH9WUY3tuomS+kmVt2hFTu0IEdW5MiC7FqRXQtybEWOLciJFTmxIDtWZMeKXKbfOtZ+61mxZhUY05mW2QryzYpus7lj4M6sOHOFF81nhFkzeL7aNDolGmYvRvMZYtcw50k0n312DdscPLBGfGBB1qzImqX2Yj3DE5qETm2zc8dVtgmfu4qVFu0yEOWDBfupBiceAe4mPNAYg60meRFRpr9N8kuwlIqTw4l0mliltmeiHuJ5EnqGdUHF3AXM2xRNmSdF8eF+GX3hbU2FvUXWlokre898eyIq9rgoqv/vCL/Z0c6WmL1cN5/D3Aqu0pviN6vs7TQCzaHwGE8K8HSjC3aqsHN/h3pqAJpVai+mKsAlvyYfYvIt65V1aUS+7jPJstPKxZhsbI6gYuKevAa7oRrD/SJed2Fvj081QmyDHH8XwCe8Vun+WGg+kEb6v+Pb19dvct4NIM/4ejXLmPx3LHUFHtA+hJ8utcUpqku/agXG+j4R2Nwq5+Vofc/KTynkGS3/NpGdWcg6DbHHL9bJdvzqvo7rnMFzC+7MU4FIQ9je6435DEZN5D2XfmHEvQ/qubSafApz6VlJz/GKVC3prQulRpa36dMqCH+Nf0brhGuY5R+z26LCtOd16I6IusiOJyxl9x2KfUrcIa0jeIXK9mnF/uR2fMG12VjsS7/Am77IWFfaX5d707f4o1sp8+c5rf35XY/ypkvRlLf9nBCyVr6CLm/vhFad6t73RFqJFumF1Ev8/EfVliWLsrhFJ3S2M7tF2d+mc4CWlI84l2KaSy5VnmT+nx/y+jSV8tQ8P6xp1qdzb2SUapHvxvc0lL5D4qjLhZF5dHrhK3FlfZv/p8RUwuXRlNnlJxKLrGYo1WZC2Wy3yytZbjc/w8ktlPnVo9+KGyKL+BPWpI9EPf0Q7rE6fcS2+Nn9xa1V43+JzJuzuxubv9y4+/BXq+/uiP8zeoX9nL0F1WqT/Ya9C2/nOjsFf/6+Ull5c+Unaztr3bVo7ZpDX1oROm8y5W/t5isZZKi0</latexit>

Cspin (t, l)<latexit sha1_base64="fOqUfVnJWnBzBdHVnHlg0xYVdW0=">AAAlVnic3VpfbxtFEN8CpcX8aVpAQuLFIg1qpShyCoiCVKn546Rp09aNkzRtXSKfc/6jnO3rne3accMH4AvwwAsg8VD4GLzwBZDoR0A8FokXhJiZ3fPtvztbCNpSj+zbm/3N7Mzu7Nzunh3fa4SdXO7RkRdefOnoy8eOv5J59bXX3zgxdfLUdtjuBhV3q9L22sGOUw5dr9FytzqNjufu+IFbbjqee9PZX8L6mz03CBvt1mZn4Lt3m+Vaq1FtVModYO1OvbW0OywFzWzoN1qHZzqz3tlMJrM7NZ2by9EnOyrM64VpJj6F9slTb7MS22NtVmFd1mQua7EOlD1WZiHQHTbPcswH3l02BF4ApQbVu+yQZUC2CygXEGXg7sNvDe7uCG4L7lFnSNIVaMWDbwCSWTaT+zn3MPc491Puh9yvuT8TdQ1JB9oygKvDZV1/98QX7xT/GCvVhGuH1WOpFIkQ7srkWQBfb4x/HVZl58mvBvjpEwc9rvC2egdfPi5+ujEzfD/3be438PWb3KPcj+Btq/d75bsb7sZXKZZEVgdQW4Uvapy8d5KQNdBXBm6dRrDPZrURSZJzgJqpiCbwPYqMALTdT+m5ybRN7j160JRGOQmHtu2zA8WyiOfB1aG+CaAXMQ7q1E8Yt7PE5V7hnQ+lEPAo24bRxrGvUU9i5FRSLAhBpinaCGDGJSN5/2TYTIquOlmEs9ShWA1HEr5AYXsDiuYh8BCJtiIKcc8q/RPbskDLLM9W2Bq7Bt/b8L1OJeTj7wrdbxIS6cnZ9uT6DeO6SjHZgVKTMkIHYuQClTGmPgHP4xy3RyiMlZAylw+Z/nmKjaxBRYiFq2wRYmGdomKbLbANKtnov7TtSfcbxoZHGa8EI36Bfj16EpQpt0Z1DtU5dKfW1KimRk+bpla3R3V7lIdQTm7NpTqMLsyaHsWnLHtA9QeWFutCUufvE3+fMrSvteZRnUdWOmCRKtmkWnxWydwWcVsat0/cPtgsc33i+sCVWw2Ji/7VjL7pUB36oOqvEr9HzwP+RFbbqRv1cW2F6irEzShSobAv1PQNiN9NHAXkxu3ZPVklxOooAuSWl6lueRQBstym8AWvfDTjunWqWU8Yrx2q3dF8KRC3oPleFB4UrbavCBm9nwuixwpGj90i/pbSY2pcu1KPJcd3QKiAntdqXNRFZES9Itfi2uICXVsQKfgcl9tui0hu09q6JsYC51+VZPCJj6vSOSjNgd5DWmtE9SiB9VhXG9VnFB0FIMSUxNMiC/eqljyQishriKtAKuIqIbISZhtIxWwLa2LMEpCKWdJaWgFSEStGS5eAVMwlTctNIBVx07BlE0jFbGpargGpiGuGLYtAKmZR07IGpCLWDFs2gFTMhqZlHUhFrBu2XAZSMZcNzHUgFXPdsGYZSMUsa9YsAKmIBQ2xCqQiVg1bikAqpmjYcgVIxVzRWtoCUhFbRks3gFTMDQOzA6RidgxrbgGpmFuaNbeBVMRty5zcEPMas0mTRtslLbNa5DQV1BrtJThK1VcH4kgH7rO0e1K9W5DGjGMWDMw9IBlxz/B/UYp3jlm09uOOgtkxMAUpK3FMwcDgikBG+IY1PSAZ0TN09IFkRN9ADGiXGCMGBuIASEYcGHaUiWRMWYsKh0hGOBqinjCGah5ZUxBmHukCyYiuJfZwdz+ka4t4Ae2ZWxSBst94mhBpa0Pfysih0KxKIKJKeNyD9OF3KHi6nW1hg65XtzWgvU1PeIRPZz7KarbiupbZrjRXUKMZvyFhwwmQDj3N43ZL4kwio1kY0pwpjU4t+InI0Nq2b0WacR1CezZkYEHeo14u0RmGR/xka0x7JpH1rbLBRLKBVbZrxXYtyJ4V2bMg61Zk3YLsW5F9C3JgRQ4syJoVWbMiJ+m3mrXfGlasmQX6dKZltoJ8M6PbdC4auG0rzlzhtUczwswZPF5tErUUCbMX26MZYpcw50l7NPvsErY5uGr1eNWCzFuReUvuxXyGJzQBndpG547TbB6+5xQtFdplIMoFDfZTDU7cA9xNOCDRB10lsqJNkX6a6vdAUyhODgfSaWKW2j4U+RDPk9AyzAsq5hxgTpM3aZYk+Yf7ZbSFtzUU+sZpm8Sv6Dnz/HiUbHGSV/9vD59tbw8nmL1cNp7DXAuu0kvinVX0dOqBZFdYjCcFeLpRBz1Z2Lmfp57qgGSW2vMpC/Caj8gGn2yLemVWGpGnfSaZdlo5HhONzTpkTNyT52E3lGe4X8TrEuzt8S5PiAWox/cCeIfXLJU3hOR1aaT/HdueXr/JcdeBOOPr1Shi4vdY6grco30IP12qilPUMr3V8oz1fSCwsVbOi9H6npWfUsgzWn43EZ1ZyDJFscdPlol2/Oq+jstsw30FSuapQFtD2J7rxdEMRknk3ZfeMOLeB+XKtJq8A3PpbkrP8YyUTemtXSVHprfp0ioI38bfpXXCPZjln7MzIsNUR3norPA6SY8jNEXlGvk+JG6X1hE8Q0X7tGR7Yj2u4Np0jLellWBNS0RsWdpfp1vTstija0mz5z6t/XmpQXFTJ2/S275PCFkqXkGnt7dJq05177sprUST5JrUS/z8R5WWa8ZFcYVO6GxnduOiv0rnABUpHnEu+TSXypR5gtE/P+T1aSjFqXl+mNe0D0fWyChVI9+NL2sofYfEUXtjPXPo9MJV/Ir6Nv6nxFDCxd6k6eUnEuO0RihVZ0DRbNfLM1msNz7DiTWk2dWgd8VFEUX8DnPSA5FPP4MyZqcH7AI/u9+dmjb+S2QWts/NzX8wd+7Gh9MXF8X/jI6zd9l7kK3m2cfsIjydC2wL7DlgX7OH7PvTv5z+a+bozDEOfeGIkHmTKZ+Zqb8B58eAeQ==</latexit><latexit sha1_base64="rpwICZ3lqiwREAFFHzawHaEApys=">AAAlVnic3VpbbxtFFJ4CpcVcGm4SEi8WaRCVosgpIApSpebipKFp68ZJmrYukddZX5S1d7tru3bc8AP4AzzwAkg8AD+DF/4AEv0JiEeQeEGIc87Meue2awtBKfjI3tkz3zlzzsyZszOzdgKvFXULhYcnnnjyqZNPnzr9TO7Z555/4czMiy/tRn4vrLk7Nd/zwz2nGrleq+PudFtdz90LQrfadjz3pnO4gvU3+24YtfzOdncYuHfb1UanVW/Vql1g7c+8srI/qoTtvON3Do7f6s5753K53P7MbGGhQJ/8uLCoF2aZ+JT8F196lVXYAfNZjfVYm7msw7pQ9liVRUB32CIrsAB4d9kIeCGUWlTvsmOWA9keoFxAVIF7CL8NuLsjuB24R50RSdegFQ++IUjm2Vzhh8LXhV8K3xe+LfxU+D1V14h0oC1DuDpc1g32z3zyWvm3iVJtuHZZM5HKkIjgrkqehfD1JvjXZXV2gfxqgZ8BcdDjGm+rf/TpL+UPtuZGbxa+LPwMvn5ReFj4Drzt9H+tfXXD3fosw5LY6hBq6/BFjdP3ThqyAfqqwG3SCA7YvDYiaXIOUDsT0Qa+R5ERgrb7GT03nbbpvUcP2tIop+HQtkN2pFgW8zy4OtQ3IfQixkGT+gnjdp643Cu8C6AUAR5lfRhtHPsG9SRGTi3Dgghk2qKNEGZcOpL3T47NZehqkkU4Sx2K1WgsEQgUtjekaB4BD5FoK6IQ97jSX7EtD7TKimyNbbBr8L0N3+tUQj7+rtH9NiGRHp1tj67fMK7rFJNdKLUpI3QhRi5SGWPqffA8yXEHhMJYiShzBZDp/0+xkTeoDLFwlS1DLGxSVOyyJbZFJRv9k7Y96n7D2PAo41VgxC/Sr0dPgirl1rjOoTqH7tSaBtU06GnT1uoOqO6A8hDKya25VIfRhVnTo/iUZY+o/sjSYlNI6vxD4h9Shg601jyq88hKByxSJdtUi88qmdshbkfjDog7AJtlbkDcALhyqxFx0b+G0TddqkMfVP114vfpecCfyGo7TaM+qa1RXY24OUUqEvZFmr4h8Xupo4DcpD27J+uEWB9HgNzyKtWtjiNAltsWvuCVj2ZSt0k1mynjtUe1e5ovJeKWNN/LwoOy1fY1IaP3c0n0WMnosVvE31F6TI1rV+qx9PgOCRXS81qNi6aIjLhX5FpcW1ykawciBZ/jctu+iGSf1tYNMRY4/+okg098XJUuQGkB9B7TWiOuRwmsx7rGuD6n6CgBIaYinhZ5uFe1FIFURFFDXAVSEVcJkZcwu0AqZldYk2BWgFTMitbSGpCKWDNaugykYi5rWm4CqYibhi3bQCpmW9NyDUhFXDNsWQZSMcualg0gFbFh2LIFpGK2NC2bQCpi07DlQyAV86GBuQ6kYq4b1qwCqZhVzZolIBWxpCHWgVTEumFLGUjFlA1brgCpmCtaSztAKmLHaOkGkIq5YWD2gFTMnmHNLSAVc0uz5jaQirhtmZNbYl5jNmnTaLukZV6LnLaC2qC9BEep+ppAHOnAfZ52T6p3S9KYccySgbkHJCPuGf4vS/HOMcvWftxTMHsGpiRlJY4pGRhcEciIwLCmDyQj+oaOAZCMGBiIIe0SE8TQQBwByYgjw44qkYypalHhEMkIR0M0U8ZQzSMbCsLMIz0gGdGzxB7u7kd07RAvpD1zhyJQ9htPE2JtPvStjBwJzaoEIuqExz3IAH5Hgqfb6QsbdL26rSHtbfrCI3w681FWsxXXtcr2pbmCGs34jQgbTYF06GmetFsRZxI5zcKI5kxlfGrBT0RG1rYDK9KM6wjasyFDC/Ie9XKFzjA84qdbY9ozjWxglQ2nkg2tsj0rtmdB9q3IvgXZtCKbFuTAihxYkEMrcmhBNqzIhhU5Tb81rP3WsmLNLDCgMy2zFeSbGd2mc9nA7Vpx5grPH88IM2fweLVJNDIkzF70xzPELmHOE388++wStjm4bvV43YIsWpFFS+7FfIYnNCGd2sbnjrNsEb7nFS012mUgygUN9lMNTtwD3E04IDEAXRWywqdIP0v1B6ApEieHQ+k0MU9tH4t8iOdJaBnmBRVzHjBnyZssS9L8w/0y2sLbGgl9k7RN41f8nPn/eJRucZpX/20PH29vj6eYvVw2mcNcC67SK+KdVfx06oNkT1iMJwV4utEEPXnYuV+gnuqCZJ7aCygL8Jp3yYaAbIt7ZV4akX/7TDLrtHIyJh6bTciYuCcvwm6oyHC/iNcV2NvjXZEQS1CP7wXwDq95Km8JyevSSP89tv17/SbHXRfijK9X44hJ3mOpK3CP9iH8dKkuTlGr9FbLM9b3ocAmWjkvQet7Vn5KIc9o+d1EfGYhy5TFHj9dJt7xq/s6LrML9zUomacCvoawPdfL4xmMksi7L71hxL0PylVpNXkH5tLdjJ7jGSmf0Vv7So7MbtOlVRC+jb9L64R7MMs/Zm+JDFMf56Fzwus0PY7QFJcb5PuIuD1aR/AMFe/T0u1J9LiCa9Mx2ZZOijUdEbFVaX+dbU3HYo+uJcue+7T256UWxU2TvMlu+z4hZKlkBZ3d3jatOtW977a0Ek2Ta1Mv8fMfVVqumRTFNTqhs53ZTYr+Op0D1KR4xLkU0FyqUuYJx//8kNenkRSn5vlhUdM+Glsjo1SNfDe+qqH0HRJHHUz0zKHTC1fxK+7b5J8SIwmXeJOll59ITNIao1SdIUWzXS/PZIne5Awn0ZBlV4veFZdFFPE7zEkPRD79CMqYnR6wi/zsfn9m1vgvkVnYPb+w+PbC+RvvzF5aFv8zOs1eZ29Atlpk77FL8HQusR2w54h9zr5m35z98ewfcyfnTnHoEyeEzMtM+czN/AmWwIBi</latexit>

1

1

0.8

0.8

0.6

0.6

0.4

0.2

1
2
3
4
5
6
7
8
9

0.4

l=1
2
3
4
5
6
7
8
9
10

0.2

0

0

0×100 1×104 2×104 3×104 4×104 5×104 6×104 7×104 8×104 9×104 1×105

0×100 1×104 2×104 3×104 4×104 5×104 6×104 7×104 8×104 9×104 1×105

t
N = 20, M = 200(↵ = 10)t
L = 10

T = 0.015

240 samples

teacher-student setting
Greedy Monte Carlosimulation on
Binary perceptronbondTraining
spin

Test

spin

teacher-student overlaps

Summary of this work
Solution space of over-parametrized DNN<latexit sha1_base64="FNsqtBcwkWx/yebBj4t5E/qt9Z0=">AAACfHichVHLShxBFD3TmkTHJE4iguCmcWIwhIx3ZnyNK9GNmwQfGRUcGbrb0hT2dDXdNYM6+AP+gAtXCoKJn+HGH8jCTwguFbIJIbcfJIgkuQVVp86959apKtt3ZaiJrjNGR+ejx0+6urM9T5897829eLkSqmbgiKqjXBWs2VYoXOmJqpbaFWt+IKyG7YpVe2cuyq+2RBBK5X3Ue77YaFjbntySjqWZquf6a7vSrPmB8rUya65njrwf/fCmnstToTJWqVDJTMD4eApKk2axQHHkkcaCyp2jhk0oOGiiAQEPmrELCyGPdRRB8JnbQJu5gJGM8wIHyLK2yVWCKyxmd3je5t16ynq8j3qGsXqXKxQrFXc5gIlh+kqf6Zau6IK+0Y+/dmvHXSI3e7zaiVb49d7DgeXv/1U1eNX49Ef1T9caW5iK3Up278dMdA8n0bf2j26Xp5eG26/plG7Y/wld0yXfwGvdOWeLYukY2eQLoij/fvCHYKVUKJYL5cWx/Mxs+hldGMQQRvjFJzGDeSygyue2cYIvuMj8NF4Zb413SamRSTV9uBfGxC/QQ5KQ</latexit>⇠ / ln(M/N )

solid
input

N

solid
liquid“encoding”

symmetry breaking
ﬁeld O(log(N )/N )output“decoding”<latexit sha1_base64="oM8Gsp8WsKFYDS584EbxWWte52U=">AAACcXichVHLSiNBFD1pHXXijIm6mcFNMDhEhHjzQjMrmdm48h0VjEh3W8bGTlfT3Qk6wR/wA3ThQkcQFD/DjT/gwk8QlxFmMwtvPxiRQb0FVafOvefWqSrNNg3XI7qLKR2dH7q6ez7Gez997ksk+weWXdlwdFHRpSmdVU11hWlYouIZnilWbUeodc0UK9rOTz+/0hSOa0hryduzxXpdrVnGlqGrHlPV2UzVlLXMzOj4zOhGMk3ZcrFcpnwqBKVSBPITqVyWgkgjijmZvEAVm5DQ0UAdAhY8xiZUuDzWkAPBZm4dLeYcRkaQF9hHnLUNrhJcoTK7w3ONd2sRa/He7+kG6l2ukKyU3GUfKYzQLV1Sm27oiu7p76vdWkEX380er1qoFfZG4uDL4p93VXVePWw/q9507WELk4Fbg93bAePfQw/1zV9H7cXvCyOtb3RGD+z/N93RNd/Aaj7q5/Ni4Rjx8Av8KPx78P/Bcj6bK2QL88X01I/oM3owhGFk+MUnMIVpzKHC59o4xAlOY23lq5JShsNSJRZpBvEilLEnjtiOcQ==</latexit>L

1. Liquid phase helps equilibration
2. Crystal phase + symmetry breaking ﬁeld
enables generalization
3. Spacial evolution of the hierarchal free-energy landscape:
DNN naturally has the power of renormalization: classiﬁcation,
feature detection

Outlook
Numerical simulations to test theoretical predctions
Various statistical inference problems
input

outputstudent machine

Complex systems with heterogeneity
gene regulatory network,…
functionality vs robustness in biology
allostericity
ultra-stable glass, rheology

Theories in

d=1+1

Thank you!

Like “Landau to Ginzburg-Landau” but more microscopic