Deep Learning and Physics Seminar Nov 11th, 2021

深層ニューラルネットワークにおけるレプリカ対称性の破れとその空間構造
Hajime Yoshino
Cybermedia Center,Osaka University

Hajime Yoshino, SciPostPhys. Core 2, 005 (2020).
「最近の研究から - 深層ニューラルネットワークの解剖ー統計力学によるアプローチ」日本物理学会誌76巻9号(2021年9月号)

“Deep Learning”

1978

statistical mechanics of disordered systems

without quencheddisorderspin glass
theoryand FARbeyond

1987

1991

1999

2001

2020

2008

in progresshttps://www.nobelprize.org/uploads/2021/10/sciback̲fy̲en̲21.pdf“for groundbreaking contributions to
our understanding of complex physical systems”

Perceptron

S1

J1<latexit sha1_base64="EyF8loPMOp/1dKuHfHz6QKGw6+I=">AAACaHichVE9SwNBEH05v+NX1ELFJhgiVmEuiF+VaCNWaowKMYS7c6NLLnfH3SWgwT9gY6lipSAi/gwb/4CFP0EtI9hYOLlERII6y+7Ovp03+3ZGd0zp+URPIaWlta29o7Mr3N3T29cfGRjc9OySa4i0YZu2u61rnjClJdK+9E2x7bhCK+qm2NILS7X7rbJwPWlbG/6BI7JFbc+SeWloPkPplVxFPcpFYpSYI3VuWo02O2qCAouhYat25AY72IUNAyUUIWDBZ9+EBo9HBioIDmNZVBhz2ZPBvcARwswtcZTgCI3RAq97fMo0UIvPtZxewDb4FZOny8wo4vRIt1SlB7qjZ/r4NVclyFHTcsC7XucKJ9d/PJJ6/5dV5N3H/jfrT80+8pgNtErW7gRI7RdGnV8+PK2m5tfjlQm6olfWf0lPdM8/sMpvxvWaWL9AOGhAvcrRZuerAZvJhDqdSK5NxRYWG63oxBjGMcn1nsEClrGKNL8rcYIznIdelIgyrIzWQ5VQgzOEH6aMfwLbNIvM</latexit><latexit sha1_base64="QoVpjNLS1LOP88I7//d20UgzJjY=">AAACaHichVFNLwNBGH66vuujiwPi0mgqTs1sSYiTcHGktSpBmt01mHS7u9ndNmHjD7g4Ik4kIuJnuPgDDn4CjpW4OHh3u4nQ4J3MzDPPvM87z8zojik8n7GnhNTW3tHZ1d2T7O3rH0jJg0Prnl1zDa4atmm7G7rmcVNYXPWFb/INx+VaVTd5Sa8shfulOnc9YVtr/oHDt6vaniV2haH5RKnFcqAcleUMy7Eo0q1AiUEGcazY8g22sAMbBmqogsOCT9iEBo/aJhQwOMRtIyDOJSSifY4jJElboyxOGRqxFRr3aLUZsxatw5pepDboFJO6S8o0suyR3bIGe2B37Jl9/ForiGqEXg5o1pta7pRTx6PF939VVZp97H+p/vTsYxdzkVdB3p2ICW9hNPX1w9NGcb6QDSbZFXsl/5fsid3TDaz6m3G9ygsXSNIHKD+fuxWs53PKdC6/OpNZWIy/ohvjmMAUvfcsFrCMFah0rsAJznCeeJFkaUQaa6ZKiVgzjG8hTXwCVvKLhw==</latexit>

Perceptron（McCulloch-Pitts model)

J2<latexit sha1_base64="Vw3GYVa6OmzOaqt91wgAmrRZvQI=">AAACaHichVG7SgNBFD1Z3/EVtVCxCQkRq3A3iK9KtBErX3lADGF3HXVws7vsbgIa/AEbSxUrBRHxM2z8AQs/QS0VbCy82UREgnqHmblz5p47Z+7VHVN6PtFjSGlpbWvv6OwKd/f09vVHBgYznl12DZE2bNN2c7rmCVNaIu1L3xQ5xxVaSTdFVt9brN1nK8L1pG1t+PuOKJS0HUtuS0PzGUovF6upw2IkTslZUmen1GizoyYpsDgatmJHrrGJLdgwUEYJAhZ89k1o8HjkoYLgMFZAlTGXPRncCxwizNwyRwmO0Bjd43WHT/kGavG5ltML2Aa/YvJ0mRlFgh7ohl7pnm7piT5+zVUNctS07POu17nCKfYfjay//8sq8e5j95v1p2Yf25gJtErW7gRI7RdGnV85OHldn1tLVMfpkl5Y/wU90h3/wKq8GVerYu0c4aAB9SpHm52vBmRSSXUqmVqdjM8vNFrRiTHEMMH1nsY8lrCCNL8rcYxTnIWelYgyrIzWQ5VQgzOEH6bEPgHdNYvN</latexit>

S2

S0<latexit sha1_base64="msQsOKxDYussk++CeQfiXTU+cko=">AAACaHichVFNLwNBGH66vqo+ujggLqIhTs1sSYhTw8WxVUWCNLtrWpPuV3a3TWrTP+DiiDiRiIif4eIPOPgJOJK4OHi73UQQvJOZeeaZ93nnmRnNMYTnM/YQkzo6u7p74r2Jvv6BwaQ8NLzh2TVX50XdNmx3S1M9bgiLF33hG3zLcblqagbf1Korrf3NOnc9YVvrfsPhu6ZasURZ6KpPVLFQCjLNkpxiaRbG5E+gRCCFKHK2fIUd7MGGjhpMcFjwCRtQ4VHbhgIGh7hdBMS5hES4z9FEgrQ1yuKUoRJbpbFCq+2ItWjdqumFap1OMai7pJzENLtn1+yF3bEb9sjef60VhDVaXho0a20td0rJw7HC278qk2Yf+5+qPz37KGMx9CrIuxMyrVvobX394PilsLQ2HcywC/ZM/s/ZA7ulG1j1V/0yz9fOkKAPUL4/90+wkUkrc+lMfj6VXY6+Io4JTGGW3nsBWawihyKdK3CEE5zGniRZGpXG26lSLNKM4EtIUx9Y84uI</latexit><latexit sha1_base64="CpYN3TX7zie2aju2flZiK+fYnXI=">AAACaHichVFNLwNBGH66vuujxQFxaTQVp+YtEuLUcHFEVSXVNLtrWhPb3c3utglN/4CLI+JEIiJ+hos/4OAn4Eji4uDtdhOhwTuZmWeeeZ93npnRbEO6HtFjSOno7Oru6e0L9w8MDkWiwyNbrlV1dJHVLcNytjXVFYY0RdaTniG2bUeoFc0QOW1/pbmfqwnHlZa56R3YolBRy6YsSV31mMpminVqFKNxSpIfsXaQCkAcQaxZ0WvsYBcWdFRRgYAJj7EBFS63PFIg2MwVUGfOYST9fYEGwqytcpbgDJXZfR7LvMoHrMnrZk3XV+t8isHdYWUMCXqgG3qle7qlJ/r4tVbdr9H0csCz1tIKuxg5Gs+8/6uq8Oxh70v1p2cPJSz6XiV7t32meQu9pa8dnrxmljYS9Wm6pBf2f0GPdMc3MGtv+tW62DhHmD8g9fO528HWbDI1l5xdn4+nl4Ov6MUkpjDD772ANFaxhiyfK3GMU5yFnpWoMqZMtFKVUKAZxbdQpj4BVPGLhg==</latexit><latexit sha1_base64="bjaTBwxC/18IqaaiEiOmjg75OaU=">AAACwXichVHNThRBEC5GVFx/WORiwmXiBoKXTQ0xSkhICFyMBwIsCyQ0TmbG3t0O80d37yp2+gV8AQ+eNDHG+BZ48QU88AYYT4qJFw/UzE5ilKg1me6vvqqvqro7zGOhNOLxiHNh9OKly2NXalevXb8xXp+4uaWyvox4O8riTO6EgeKxSHlbCx3znVzyIAljvh3urxTx7QGXSmTppj7M+V4SdFPREVGgifLrrOUbtO6iYTJxVTe1Lot5R8+yjgwi41nD1IHUZtVapvqJb8SiZx+R6zLNn+qyv3nSE5pb85Ci1raKlUnR7ek7fr2BTSzNPQ+8CjSgsrWs/hYYPIYMIuhDAhxS0IRjCEDRtwseIOTE7YEhThISZZyDhRpp+5TFKSMgdp/WLnm7FZuSX9RUpTqiLjH9kpQuTOMnfIen+BHf42f8+ddapqxRzHJIezjU8twff36r9eO/qoR2Db1fqn/OrKED8+WsgmbPS6Y4RTTUD569OG0tbEybGXyNX2j+V3iMH+gE6eB79Gadb7yEGj2A9+d1nwdbc03vXnNu/W5jabl6ijGYgtswS/d9H5bgAaxBm/oewQl8hW/OiiOc3JHDVGek0kzCb+aYM90FsKE=</latexit>

ExampleS0 = sgn

JN<latexit sha1_base64="bBlneqlUawVrp8hZOvs6vKFE+Gs=">AAACaHichVFNLwNRFD0dX1VfxQKxaTQVq+ZOI6VWwkYsxFdpUtLMjIcX05nJzLQJjT9gY4lYkYiIn2HjD1j4CViS2Fi4nVZEGtyX9959591z33n36o4pPZ/oMaQ0Nbe0toXbIx2dXd090d6+Nc8uuYbIGrZpuzld84QpLZH1pW+KnOMKraibYl3fm63er5eF60nbWvX3HbFZ1HYsuS0NzWcoO1+oLBwWonFKZkjNpNVYo6MmKbA46rZoR6+xgS3YMFBCEQIWfPZNaPB45KGC4DC2iQpjLnsyuBc4RIS5JY4SHKExusfrDp/yddTiczWnF7ANfsXk6TIzhgQ90A290j3d0hN9/JqrEuSoatnnXa9xhVPoORpcef+XVeTdx+4360/NPrYxGWiVrN0JkOovjBq/fHDyujK1nKiM0iW9sP4LeqQ7/oFVfjOulsTyOSJBA2pVjjU6Xw1YSyXVdDK1NB6fnqm3IoxhjGCM6z2BacxhEVl+V+IYpzgLPStRZUAZqoUqoTqnHz9MGfkEFWCL6Q==</latexit>SN# of possible machines<latexit sha1_base64="ll1XK4P4L3OcfSAFC2bl6pR3Ka8=">AAACbHichVHLSsNAFD2N7/pofSyEIhSL4qrcaFErCKIbl61aK/giiaMG0yQkaVFLf8CdK8GuFATFz3DjD7jwE0Rwo+DGhbdJQETUOzBz5tx7Lmfmqrahux7RY0Rqam5pbWvviHZ2dffE4r19q65VdjRR0CzDctZUxRWGboqCp3uGWLMdoZRUQxTVg4VGvlgRjqtb5op3ZIvNkrJn6ru6pnhMFce3qhO12enteIrSWZKzk3LyJ5DT5EcKYeSs+DU2sAMLGsooQcCEx9iAApfXOmQQbOY2UWXOYaT7eYEaoqwtc5XgCoXZA973+LYesibfGz1dX33IFRYrLe5SQxIj9EA39Er3dEtP9PFrt6rfpeHmiE810Ap7O3YyuPz+r6rEp4f9L9Wfrj3sYtp3q7N722ca79ACfeX47HV5ZmmkOkqX9Mz+L+iR7vgFZuVNu8qLpTqiwQj8SAZgKhOC7NcIVsfT8mQ6k8+k5ubDYbQjgWGM8Y9PYQ6LyKHguzvFOeqRF2lASkhDQakUCTX9+BbS6CdTgo1W</latexit>

3

2 =8

machine1+++

machine2+++

activation function

DATA1

+

-

N
X
1
p
J i Si
N i=1

!

+

+
+

+

+
+

-

…

-

……

DATA3

DATA2

-

+

-

+
+

-

+

+
+

S1

Perceptron（McCulloch-Pitts model)

J1<latexit sha1_base64="EyF8loPMOp/1dKuHfHz6QKGw6+I=">AAACaHichVE9SwNBEH05v+NX1ELFJhgiVmEuiF+VaCNWaowKMYS7c6NLLnfH3SWgwT9gY6lipSAi/gwb/4CFP0EtI9hYOLlERII6y+7Ovp03+3ZGd0zp+URPIaWlta29o7Mr3N3T29cfGRjc9OySa4i0YZu2u61rnjClJdK+9E2x7bhCK+qm2NILS7X7rbJwPWlbG/6BI7JFbc+SeWloPkPplVxFPcpFYpSYI3VuWo02O2qCAouhYat25AY72IUNAyUUIWDBZ9+EBo9HBioIDmNZVBhz2ZPBvcARwswtcZTgCI3RAq97fMo0UIvPtZxewDb4FZOny8wo4vRIt1SlB7qjZ/r4NVclyFHTcsC7XucKJ9d/PJJ6/5dV5N3H/jfrT80+8pgNtErW7gRI7RdGnV8+PK2m5tfjlQm6olfWf0lPdM8/sMpvxvWaWL9AOGhAvcrRZuerAZvJhDqdSK5NxRYWG63oxBjGMcn1nsEClrGKNL8rcYIznIdelIgyrIzWQ5VQgzOEH6aMfwLbNIvM</latexit><latexit sha1_base64="QoVpjNLS1LOP88I7//d20UgzJjY=">AAACaHichVFNLwNBGH66vuujiwPi0mgqTs1sSYiTcHGktSpBmt01mHS7u9ndNmHjD7g4Ik4kIuJnuPgDDn4CjpW4OHh3u4nQ4J3MzDPPvM87z8zojik8n7GnhNTW3tHZ1d2T7O3rH0jJg0Prnl1zDa4atmm7G7rmcVNYXPWFb/INx+VaVTd5Sa8shfulOnc9YVtr/oHDt6vaniV2haH5RKnFcqAcleUMy7Eo0q1AiUEGcazY8g22sAMbBmqogsOCT9iEBo/aJhQwOMRtIyDOJSSifY4jJElboyxOGRqxFRr3aLUZsxatw5pepDboFJO6S8o0suyR3bIGe2B37Jl9/ForiGqEXg5o1pta7pRTx6PF939VVZp97H+p/vTsYxdzkVdB3p2ICW9hNPX1w9NGcb6QDSbZFXsl/5fsid3TDaz6m3G9ygsXSNIHKD+fuxWs53PKdC6/OpNZWIy/ohvjmMAUvfcsFrCMFah0rsAJznCeeJFkaUQaa6ZKiVgzjG8hTXwCVvKLhw==</latexit>

J2<latexit sha1_base64="Vw3GYVa6OmzOaqt91wgAmrRZvQI=">AAACaHichVG7SgNBFD1Z3/EVtVCxCQkRq3A3iK9KtBErX3lADGF3HXVws7vsbgIa/AEbSxUrBRHxM2z8AQs/QS0VbCy82UREgnqHmblz5p47Z+7VHVN6PtFjSGlpbWvv6OwKd/f09vVHBgYznl12DZE2bNN2c7rmCVNaIu1L3xQ5xxVaSTdFVt9brN1nK8L1pG1t+PuOKJS0HUtuS0PzGUovF6upw2IkTslZUmen1GizoyYpsDgatmJHrrGJLdgwUEYJAhZ89k1o8HjkoYLgMFZAlTGXPRncCxwizNwyRwmO0Bjd43WHT/kGavG5ltML2Aa/YvJ0mRlFgh7ohl7pnm7piT5+zVUNctS07POu17nCKfYfjay//8sq8e5j95v1p2Yf25gJtErW7gRI7RdGnV85OHldn1tLVMfpkl5Y/wU90h3/wKq8GVerYu0c4aAB9SpHm52vBmRSSXUqmVqdjM8vNFrRiTHEMMH1nsY8lrCCNL8rcYxTnIWelYgyrIzWQ5VQgzOEH6bEPgHdNYvN</latexit>

S2

S0<latexit sha1_base64="msQsOKxDYussk++CeQfiXTU+cko=">AAACaHichVFNLwNBGH66vqo+ujggLqIhTs1sSYhTw8WxVUWCNLtrWpPuV3a3TWrTP+DiiDiRiIif4eIPOPgJOJK4OHi73UQQvJOZeeaZ93nnmRnNMYTnM/YQkzo6u7p74r2Jvv6BwaQ8NLzh2TVX50XdNmx3S1M9bgiLF33hG3zLcblqagbf1Korrf3NOnc9YVvrfsPhu6ZasURZ6KpPVLFQCjLNkpxiaRbG5E+gRCCFKHK2fIUd7MGGjhpMcFjwCRtQ4VHbhgIGh7hdBMS5hES4z9FEgrQ1yuKUoRJbpbFCq+2ItWjdqumFap1OMai7pJzENLtn1+yF3bEb9sjef60VhDVaXho0a20td0rJw7HC278qk2Yf+5+qPz37KGMx9CrIuxMyrVvobX394PilsLQ2HcywC/ZM/s/ZA7ulG1j1V/0yz9fOkKAPUL4/90+wkUkrc+lMfj6VXY6+Io4JTGGW3nsBWawihyKdK3CEE5zGniRZGpXG26lSLNKM4EtIUx9Y84uI</latexit><latexit sha1_base64="CpYN3TX7zie2aju2flZiK+fYnXI=">AAACaHichVFNLwNBGH66vuujxQFxaTQVp+YtEuLUcHFEVSXVNLtrWhPb3c3utglN/4CLI+JEIiJ+hos/4OAn4Eji4uDtdhOhwTuZmWeeeZ93npnRbEO6HtFjSOno7Oru6e0L9w8MDkWiwyNbrlV1dJHVLcNytjXVFYY0RdaTniG2bUeoFc0QOW1/pbmfqwnHlZa56R3YolBRy6YsSV31mMpminVqFKNxSpIfsXaQCkAcQaxZ0WvsYBcWdFRRgYAJj7EBFS63PFIg2MwVUGfOYST9fYEGwqytcpbgDJXZfR7LvMoHrMnrZk3XV+t8isHdYWUMCXqgG3qle7qlJ/r4tVbdr9H0csCz1tIKuxg5Gs+8/6uq8Oxh70v1p2cPJSz6XiV7t32meQu9pa8dnrxmljYS9Wm6pBf2f0GPdMc3MGtv+tW62DhHmD8g9fO528HWbDI1l5xdn4+nl4Ov6MUkpjDD772ANFaxhiyfK3GMU5yFnpWoMqZMtFKVUKAZxbdQpj4BVPGLhg==</latexit>

NX
1
p
J i SiN i=1<latexit sha1_base64="bjaTBwxC/18IqaaiEiOmjg75OaU=">AAACwXichVHNThRBEC5GVFx/WORiwmXiBoKXTQ0xSkhICFyMBwIsCyQ0TmbG3t0O80d37yp2+gV8AQ+eNDHG+BZ48QU88AYYT4qJFw/UzE5ilKg1me6vvqqvqro7zGOhNOLxiHNh9OKly2NXalevXb8xXp+4uaWyvox4O8riTO6EgeKxSHlbCx3znVzyIAljvh3urxTx7QGXSmTppj7M+V4SdFPREVGgifLrrOUbtO6iYTJxVTe1Lot5R8+yjgwi41nD1IHUZtVapvqJb8SiZx+R6zLNn+qyv3nSE5pb85Ci1raKlUnR7ek7fr2BTSzNPQ+8CjSgsrWs/hYYPIYMIuhDAhxS0IRjCEDRtwseIOTE7YEhThISZZyDhRpp+5TFKSMgdp/WLnm7FZuSX9RUpTqiLjH9kpQuTOMnfIen+BHf42f8+ddapqxRzHJIezjU8twff36r9eO/qoR2Db1fqn/OrKED8+WsgmbPS6Y4RTTUD569OG0tbEybGXyNX2j+V3iMH+gE6eB79Gadb7yEGj2A9+d1nwdbc03vXnNu/W5jabl6ijGYgtswS/d9H5bgAaxBm/oewQl8hW/OiiOc3JHDVGek0kzCb+aYM90FsKE=</latexit>

S0 = sgn

JN<latexit sha1_base64="bBlneqlUawVrp8hZOvs6vKFE+Gs=">AAACaHichVFNLwNRFD0dX1VfxQKxaTQVq+ZOI6VWwkYsxFdpUtLMjIcX05nJzLQJjT9gY4lYkYiIn2HjD1j4CViS2Fi4nVZEGtyX9959591z33n36o4pPZ/oMaQ0Nbe0toXbIx2dXd090d6+Nc8uuYbIGrZpuzld84QpLZH1pW+KnOMKraibYl3fm63er5eF60nbWvX3HbFZ1HYsuS0NzWcoO1+oLBwWonFKZkjNpNVYo6MmKbA46rZoR6+xgS3YMFBCEQIWfPZNaPB45KGC4DC2iQpjLnsyuBc4RIS5JY4SHKExusfrDp/yddTiczWnF7ANfsXk6TIzhgQ90A290j3d0hN9/JqrEuSoatnnXa9xhVPoORpcef+XVeTdx+4360/NPrYxGWiVrN0JkOovjBq/fHDyujK1nKiM0iW9sP4LeqQ7/oFVfjOulsTyOSJBA2pVjjU6Xw1YSyXVdDK1NB6fnqm3IoxhjGCM6z2BacxhEVl+V+IYpzgLPStRZUAZqoUqoTqnHz9MGfkEFWCL6Q==</latexit>SN

activation function

Statistical mechanics of

Ji which meet random constraints<latexit sha1_base64="SrPmZMPamWJWKJsDjcavCQ41Tc8=">AAACaHichVG7SgNBFD1ZXzG+ohYqNiFBsQp3ReKjCtqIla88QCXsrhMd3Owuu5uAhvyAjaWKlYKI+Bk2/oCFn6CWCjYW3mwiIqLeYWbunLnnzpl7dceUnk/0EFJaWtvaO8Kdka7unt6+aP9A1rPLriEyhm3abl7XPGFKS2R86Zsi77hCK+mmyOl7C/X7XEW4nrStdX/fEVslbceSRWloPkOZpUJV1grRBCVnSZ1NqbGfjpqkwBJo2rIdvcImtmHDQBklCFjw2TehweOxARUEh7EtVBlz2ZPBvUANEeaWOUpwhMboHq87fNpoohaf6zm9gG3wKyZPl5kxjNE9XdML3dENPdL7r7mqQY66ln3e9QZXOIW+w+G1t39ZJd597H6x/tTso4iZQKtk7U6A1H9hNPiVg+OXtbnVseo4XdAz6z+nB7rlH1iVV+NyRayeIRI0oFHl2E/nswHZyaSaSk6uTCXS881WhDGKOCa43tNIYxHLyPC7Ekc4wWnoSYkqQ8pII1QJNTmD+GZK/ANLe4wE</latexit>µSi = ±1
Elisabeth Gardner(1957-1988)

M↵=
N<latexit sha1_base64="iTzPUROAZVVVVQ4eS3/QQJQkGBI=">AAADAHichZFLaxRBEMdrxlccH1n1IngZXKLx4FKziIoQCOYigpKHuwlksstMb2/SZl729K7Gpi85+gU85KQgKh7Eq1cvfgEP+QghxwhePFgzOygaYnro6ep/1a+qujvMIpErxG3LPnL02PETYyedU6fPnB2vnTvfztOBZLzF0iiVS2GQ80gkvKWEivhSJnkQhxFfDNdnCv/ikMtcpMkjtZHxlThYTURfsECR1K1ttaccXyTKdfxMpr2ufjzlmY5+aPy+DJjuub7iz1RZRz9dE4obfZ+CjDHaz59IpZt+JoxxeEdfHyEHAR3dJIjm71J+PCiLPajwkKvAHU7KTuEx10y3VscGlsPdb3iVUYdqzKa1t+BDD1JgMIAYOCSgyI4ggJy+ZfAAISNtBTRpkixR+jkYcIgdUBSniIDUdfqv0m65UhPaFznzkmZUJaIpiXRhAr/he9zDr/gBd/Dngbl0maPoZYPWcMTyrDv+4uLCj0OpmFYFa3+o//asoA+3y14F9Z6VSnEKNuKHz1/uLdyZn9BX8DXuUv+vcBu/0AmS4Xf2Zo7Pb4FDD+D9e937jXaz4d1sNOdu1KfvVk8xBpfgMkzSfd+CabgHs9ACZlnWVQstz96039kf7U+jUNuqmAvw17A//wJWyscI</latexit>

Gardner volume“Hardcore” constraint<latexit sha1_base64="buqQMfIAvnJWoco3d5kEck62VG8=">AAACeXichVHLSsNAFD2Nr1ofrXUjuKkWRVyUqQqKIBTduFHaaqtgpUziVEPTJCRpUUN/wB9wIQgKYsXPcOMPuPATxGUFF7rwNg2IinoHZs6ce8+dMzOyqam2w9hjQOro7OruCfaG+voHBsORoWjeNqqWInKKoRnWtsxtoam6yDmqo4lt0xK8ImtiSy6vtPJbNWHZqqFvOkem2K3wfV0tqQp3iCpGogWumQd8qVCyuOKu1d31ejESZwnmRewnSPogDj/SRqSBAvZgQEEVFQjocAhr4LBp7CAJBpO4XbjEWYRULy9QR4i0VaoSVMGJLdO8T7sdn9Vp3+ppe+pDqjBIaVCXOmKYYA/shjXZPbtlT+z9126u16Xl5ohWua0VZjF8MrLx+q+qQquDg0/Vn64dlLDguVXJvekxrXsobX3t+LS5sZidcCfZJXsm/xfskd3RDfTai3KVEdkzhOgLkt8f/CfIzySSs4mZzFw8tex/RhCjGMcUvfg8UlhFGjk69xDnuEYj8CaNSVPSdLtUCviaYXwJafYDwveSBA==</latexit><latexit sha1_base64="ZCSBj4jq9FzMi63OIRT/i22PHQ8=">AAACfnichVHLSsNAFD3Gd31V3ShuikWpC+u0iBZBKLpxqdY+wNaSxKkNTZOQTAtaint/wIUrhSIi6Ee48Qdc9BPEZQU3LrxNA6Ki3pCZc8/cc+fMjGLpmiMYa3ZJ3T29ff0Dg76h4ZHRMf/4RMoxK7bKk6qpm3ZGkR2uawZPCk3oPGPZXC4rOk8rpc32errKbUczjT1xbPFcWT4ytIKmyoKovH+aH9QWswoXcqAaKi7U17OiSAnBvD/IwsyNwE8Q8UAQXmyb/mtkcQgTKioog8OAIKxDhkPfPiJgsIjLoUacTUhz1znq8JG2QlWcKmRiSzQeUbbvsQbl7Z6Oq1ZpF51+m5QBzLEndsNa7JHdsmf2/muvmtuj7eWYZqWj5VZ+7Gwq8favqkyzQPFT9adngQJirleNvFsu0z6F2tFXT85bibXdudo8u2Iv5P+SNdkDncCovqqNHb57AR89QOT7df8EqWg4shKO7iwH4xveUwxgBrMI0X2vIo4tbCNJ+56igTvcS5DmpUVpqVMqdXmaSXwJKfYBhbeSQA==</latexit>

M ! 1 with fixed ↵

!

e

v(h)

= ✓(h)V =pattern

Z Y
N

dJj
p e
2⇡
j=1

µ = 1, 2, . . . , M
Jj 2
2

M
Y

e

v(r µ )

µ=1

N
X
1
µ
µ
µ
p J i Si
“Gap” r = S0N
i=1<latexit sha1_base64="sp3nu5R6W0dkB8Ucw1A1NFzbtU4=">AAACw3ichVHLahRBFD1pX3GMZtSN4GZwSHDjUB1ERQgERRAXkmScJJBOmupOTVJMv6yuHo1F/YA/4MKVgoj4GbrwB1zkE9TlBNy48PYDRIN6i6o699Q9996qCrJI5pqxgynn2PETJ09Nn26dmTl7brZ9/sJanhYqFIMwjVK1EfBcRDIRAy11JDYyJXgcRGI9GN0tz9fHQuUyTR7p/UxsxXw3kUMZck2U395W28aLC7vY6fuG2drx8iL2jVx0yX9ovaHioXGt8fLHShNhPS2e6qq2ebIntbDmAYVb26/lJb7mjXiWcb/dZT1WWecocBvQRWPLafstPOwgRYgCMQQSaMIROHIam3DBkBG3BUOcIiSrcwGLFmkLihIUwYkd0bpL3mbDJuSXOfNKHVKViKYiZQdz7DN7xybsE3vPvrAff81lqhxlL/u0B7VWZP7s80v97/9VxbRr7P1S/bNnjSFuVb1K6j2rmPIWYa0fP3sx6d9enTPz7DX7Rv2/YgfsA90gGR+Gb1bE6ku06APcP5/7KFhb6Lk3egsr17tLd5qvmMZlXMFVeu+bWMJ9LGNAdT/iKyY4dO45I0c5ug51phrNRfxmjv0J8RqyzA==</latexit>Gardnerʼs volume: design space of a perceptron
solution1

storage capacity

solution2…
Q = 0 liquid<latexit sha1_base64="rdrpDef9g5f70h08W3ezT46Yyr4=">AAACo3ichVFLLwNRFP6M97vYSGwaTcWqOSNSKpEIG4mNliKhkZlx1cS8zEwrNP6AhS2JFYmF+Bki8Qcs/ASxJLGxcOZ2RERwbu695373fOepe5YZhESPDUpjU3NLa1t7R2dXd09voq9/JXArviGKhmu5/pquBcIyHVEMzdASa54vNFu3xKq+Oxf9r1aFH5iusxweeKJka2XH3DYNLYygfHKaNhMpyuRIzWXV5E9FzZCUFGJZdBN32MAWXBiowIaAg5B1CxoCXutQQfAYK6HGmM+aKf8FjtDB3ApbCbbQGN3ls8yv9Rh1+B35DCTb4CgWb5+ZSaTpga7phe7php7o/VdfNekjyuWAb73OFd5m7/Hg0tu/LJvvEDtfrD9zDrGNSZmrybl7EomqMOr86uHpy9JUIV0boUt65vwv6JFuuQKn+mpc5UXhXHp3mLMvq7VlfIf7W2Nc56hHsWbFsQPscZSor9xPObz6hJI/lc/hrYxl1GxmLD+empmNx9iGIQxjlGc1gRnMYxFF9r6DE5ziTEkrC0pBWa6bKg0xZwDfRCl9AJArl6k=</latexit>

Q > 0 solid

Q=1<latexit sha1_base64="CnKOYK2g0mHAbn/XHn9vk3d3o6E=">AAACpHichVFLLyRRFP7UzHjO0MZGYlPRIVadUyJeCxE2FhZaKyQYqaq5zY16qaruien4AzaWiBWJhczPEIk/YOEniCWJjYVTt0tkIjg3995zv3u+87RDV8YJ0U2D9uXrt8am5pbWtu8/2jtynT8X46ASOcJ0AjeIlm0rFq70hZnIxBXLYSQsz3bFkr01nf4vVUUUy8BfSHZCseZZG74sS8dKGDKL+oRO67k8FcbIGBs29LeKUSAleWQyF+QusYrfCOCgAg8CPhLWXViIea3AACFkbA01xiLWpPoX2EUrcytsJdjCYnSLzw1+rWSoz+/UZ6zYDkdxeUfM1NFH13RO93RF/+iWnt71VVM+0lx2+LbrXBGud+x1lx4/ZXl8J9h8ZX2Yc4IyRlWuknMPFZJW4dT51b8H96Xx+b5aP53SHed/Qjd0wRX41QfnrCjmj5V3nzl/VLWeiu9zf2uM2xx1N9PcLHaMbY6S9pX7qYZXn5D+VnkZ3uJgwRguDBaH8pNT2Rib0YNeDPCsRjCJGczBZO8S+zjEkdavzWolzaybag0Zpwv/ifbrGQBol9Q=</latexit><latexit sha1_base64="fT62gCfAlqqGBdyNwAeMlBbYblc=">AAACkHichVE9S8NQFD3G7++qi+BSLBWnciOirSBWXcTJtvYDVCSJrzWYJiFJC7X4BwRXOzgpOIh/wc3FP+DgTxDHCi4O3qQVEVFvSN59595zc947qm3orkf01CF1dnX39Pb1DwwODY+MhsbGc65VcTSR1SzDcgqq4gpDN0XW0z1DFGxHKGXVEHn1aN2v56vCcXXL3PZqttgrKyVTL+qa4jGUSS3L+6EIxRIkJxbk8M9EjlEQEbRjywrdYRcHsKChgjIETHicG1Dg8rMDGQSbsT3UGXM404O6wAkGmFvhLsEdCqNH/C3xbqeNmrz3Z7oBW+O/GPw6zAwjSo90Q016oFt6pvdfZ9WDGb6WGq9qiyvs/dHTyczbv6wyrx4Ov1h/avZQRDzQqrN2O0D8U2gtfvW40cwspaP1GbqiF9Z/SU90zycwq6/adUqkL/7QY7OKGlf823P9uwusavkR/pl8WpWbi8kLsbnUfCS51jatD1OYxiw7s4gkNrCFLM8s4QznaEjjUlxakVZbrVJHmzOBbyFtfgCbQ5MH</latexit>

M↵=N<latexit sha1_base64="GQxCJLfEypinHDhWRdRp7mUbzrI=">AAACeXichVHLSsNQED2N7/po1Y3gploUcVEmKr5AEN24UXxVBSvlJt5qME1CkhY19Af8AReCoCAqfoYbf8CFnyAuFVzowmlaERF1Ltw7c+6cuWfuaI5peD7RQ0Spqa2rb2hsija3tLbF4u0da55dcHWZ1m3Tdjc04UnTsGTaN3xTbjiuFHnNlOva3mz5fr0oXc+wrVX/wJFbebFjGTlDFz5D2XhHRpjOrpjK5FyhB/OlYKGUjScpNUHqxKia+OmoKQotiaot2vErZLANGzoKyEPCgs++CQGP1yZUEBzGthAw5rJnhPcSJUSZW+AsyRmC0T3edzjarKIWx+WaXsje5wybmTZXKSGBPrqna3qmO7qhR3r/tVoQVimrOeBTq3Clk40dda28/svK8+lj94v1p2ofOYyHag1W74RIuQ+9wi8eHj+vTC73Bf10Tk+s/4we6JY7sIov+sWSXD5BNBxB5Z8TP53PEawNpdTR1NDSSHJ6pjqMRnSjFwP842OYxhwWkeZ393GKS1xF3pQeZUAZrKQqkSqnE99MGf4AWgiSUg==</latexit><latexit sha1_base64="yQBe8wqt39GutSWqDFLqIABYIAk=">AAAC03ichVFNTxRBEH0MoIgoK15MvGzcQDhtajYGxYSE6MVwMCywQMLCpGfsxQ7z5czsGuzMxXgy8YyJJ004EP6DF2PgD3jgJxiPmHjxYE3vEGKIWpPpfvW6X1V1lRv7Ks2ITgaswaHhS5dHroxeHbt2fbxyY2I1jbqJJ1te5EfJuitS6atQtjKV+XI9TqQIXF+uuTuPivO1nkxSFYUr2W4sNwOxHaqO8kTGlFNpNB0t3Hyu3UmEp+1cP8nbaTdwtJqz8y32FhgyEGfAzZ1KjeqzZM/O2NWLwK6TsRpKW4wqX9DGU0Tw0EUAiRAZYx8CKX8bsEGImduEZi5hpMy5RI5R1nb5luQbgtkdXrfZ2yjZkP0iZmrUHmfx+U9YWcUkfaUDOqVjOqRv9OuvsbSJUdSyy7vb18rYGX9za/nnf1UB7xmenav+WXOGDu6bWhXXHhumeIXX1/de7p0uP1ia1FP0kb5z/R/ohD7zC8LeD2+/KZfem+gha16Y1wYmf8j91cy7nDUvkV/mTvGcsxR95X6a4fUnVL0Izoa32qjbM/VG825t/mE5xhHcxh1M86zuYR6PsYgWR3+HTzjCsdWytPXKet2/ag2Umpv4w6y3vwHAc6zd</latexit>

overlap between machine a and blearning with the same data

Qab =

N
X
1

N i=1

a b
Ji Ji

Gardnerʼs volume: design space of a perceptron
solution1

storage capacity

solution2…
Q = 0 liquid<latexit sha1_base64="rdrpDef9g5f70h08W3ezT46Yyr4=">AAACo3ichVFLLwNRFP6M97vYSGwaTcWqOSNSKpEIG4mNliKhkZlx1cS8zEwrNP6AhS2JFYmF+Bki8Qcs/ASxJLGxcOZ2RERwbu695373fOepe5YZhESPDUpjU3NLa1t7R2dXd09voq9/JXArviGKhmu5/pquBcIyHVEMzdASa54vNFu3xKq+Oxf9r1aFH5iusxweeKJka2XH3DYNLYygfHKaNhMpyuRIzWXV5E9FzZCUFGJZdBN32MAWXBiowIaAg5B1CxoCXutQQfAYK6HGmM+aKf8FjtDB3ApbCbbQGN3ls8yv9Rh1+B35DCTb4CgWb5+ZSaTpga7phe7php7o/VdfNekjyuWAb73OFd5m7/Hg0tu/LJvvEDtfrD9zDrGNSZmrybl7EomqMOr86uHpy9JUIV0boUt65vwv6JFuuQKn+mpc5UXhXHp3mLMvq7VlfIf7W2Nc56hHsWbFsQPscZSor9xPObz6hJI/lc/hrYxl1GxmLD+empmNx9iGIQxjlGc1gRnMYxFF9r6DE5ziTEkrC0pBWa6bKg0xZwDfRCl9AJArl6k=</latexit>

Franz-Parisi (2016)

<0<latexit sha1_base64="QKTgd2hL7KzSKsssDJcEfPcCs0c=">AAACbnichVHLSsNAFD2N7/qqCiKIGCyVrspNkfrAhejGpa9qoS2SxKmGpklI0oIWf8C9uBAUBRHxM9z4Ay78BHEjKLhx4W1aESnqHWbmzpl77py5V3NMw/OJHkNSS2tbe0dnV7i7p7evPzIwuOnZZVcXad02bTejqZ4wDUukfcM3RcZxhVrSTLGlFZdq91sV4XqGbW34+47Il9RdyygYuuozlM0VVcdRZXlepu1IlBKzpMymFLnZURIUWBQNW7Ej18hhBzZ0lFGCgAWffRMqPB5ZKCA4jOVRZcxlzwjuBQ4RZm6ZowRHqIwWed3lU7aBWnyu5fQCts6vmDxdZsqI0QPd0Cvd0y090cevuapBjpqWfd61Olc42/1HI+vv/7JKvPvY+2b9qdlHATOBVoO1OwFS+4Ve51cOTl7X59Zi1Um6pGfWf0GPdMc/sCpv+tWqWDtFOGhAvcpys/PVgM1kQkklkqtT0YXFRis6MYoJxLne01jAMlaQDip2jDOch16kYWlMGq+HSqEGZwg/TIp/ApICjWs=</latexit>

Q > 0 solid

Q=1<latexit sha1_base64="CnKOYK2g0mHAbn/XHn9vk3d3o6E=">AAACpHichVFLLyRRFP7UzHjO0MZGYlPRIVadUyJeCxE2FhZaKyQYqaq5zY16qaruien4AzaWiBWJhczPEIk/YOEniCWJjYVTt0tkIjg3995zv3u+87RDV8YJ0U2D9uXrt8am5pbWtu8/2jtynT8X46ASOcJ0AjeIlm0rFq70hZnIxBXLYSQsz3bFkr01nf4vVUUUy8BfSHZCseZZG74sS8dKGDKL+oRO67k8FcbIGBs29LeKUSAleWQyF+QusYrfCOCgAg8CPhLWXViIea3AACFkbA01xiLWpPoX2EUrcytsJdjCYnSLzw1+rWSoz+/UZ6zYDkdxeUfM1NFH13RO93RF/+iWnt71VVM+0lx2+LbrXBGud+x1lx4/ZXl8J9h8ZX2Yc4IyRlWuknMPFZJW4dT51b8H96Xx+b5aP53SHed/Qjd0wRX41QfnrCjmj5V3nzl/VLWeiu9zf2uM2xx1N9PcLHaMbY6S9pX7qYZXn5D+VnkZ3uJgwRguDBaH8pNT2Rib0YNeDPCsRjCJGczBZO8S+zjEkdavzWolzaybag0Zpwv/ifbrGQBol9Q=</latexit><latexit sha1_base64="fT62gCfAlqqGBdyNwAeMlBbYblc=">AAACkHichVE9S8NQFD3G7++qi+BSLBWnciOirSBWXcTJtvYDVCSJrzWYJiFJC7X4BwRXOzgpOIh/wc3FP+DgTxDHCi4O3qQVEVFvSN59595zc947qm3orkf01CF1dnX39Pb1DwwODY+MhsbGc65VcTSR1SzDcgqq4gpDN0XW0z1DFGxHKGXVEHn1aN2v56vCcXXL3PZqttgrKyVTL+qa4jGUSS3L+6EIxRIkJxbk8M9EjlEQEbRjywrdYRcHsKChgjIETHicG1Dg8rMDGQSbsT3UGXM404O6wAkGmFvhLsEdCqNH/C3xbqeNmrz3Z7oBW+O/GPw6zAwjSo90Q016oFt6pvdfZ9WDGb6WGq9qiyvs/dHTyczbv6wyrx4Ov1h/avZQRDzQqrN2O0D8U2gtfvW40cwspaP1GbqiF9Z/SU90zycwq6/adUqkL/7QY7OKGlf823P9uwusavkR/pl8WpWbi8kLsbnUfCS51jatD1OYxiw7s4gkNrCFLM8s4QznaEjjUlxakVZbrVJHmzOBbyFtfgCbQ5MH</latexit>

clusteringglass

jamming(SAT/UNSAT tradition )Replica Symmetry Breaking (RSB)<latexit sha1_base64="yQBe8wqt39GutSWqDFLqIABYIAk=">AAAC03ichVFNTxRBEH0MoIgoK15MvGzcQDhtajYGxYSE6MVwMCywQMLCpGfsxQ7z5czsGuzMxXgy8YyJJ004EP6DF2PgD3jgJxiPmHjxYE3vEGKIWpPpfvW6X1V1lRv7Ks2ITgaswaHhS5dHroxeHbt2fbxyY2I1jbqJJ1te5EfJuitS6atQtjKV+XI9TqQIXF+uuTuPivO1nkxSFYUr2W4sNwOxHaqO8kTGlFNpNB0t3Hyu3UmEp+1cP8nbaTdwtJqz8y32FhgyEGfAzZ1KjeqzZM/O2NWLwK6TsRpKW4wqX9DGU0Tw0EUAiRAZYx8CKX8bsEGImduEZi5hpMy5RI5R1nb5luQbgtkdXrfZ2yjZkP0iZmrUHmfx+U9YWcUkfaUDOqVjOqRv9OuvsbSJUdSyy7vb18rYGX9za/nnf1UB7xmenav+WXOGDu6bWhXXHhumeIXX1/de7p0uP1ia1FP0kb5z/R/ohD7zC8LeD2+/KZfem+gha16Y1wYmf8j91cy7nDUvkV/mTvGcsxR95X6a4fUnVL0Izoa32qjbM/VG825t/mE5xhHcxh1M86zuYR6PsYgWR3+HTzjCsdWytPXKet2/ag2Umpv4w6y3vwHAc6zd</latexit>

overlap between machine a and blearning with the same data

Qab =

N
X
1

N i=1

M
↵=
N
<latexit sha1_base64="GQxCJLfEypinHDhWRdRp7mUbzrI=">AAACeXichVHLSsNQED2N7/po1Y3gploUcVEmKr5AEN24UXxVBSvlJt5qME1CkhY19Af8AReCoCAqfoYbf8CFnyAuFVzowmlaERF1Ltw7c+6cuWfuaI5peD7RQ0Spqa2rb2hsija3tLbF4u0da55dcHWZ1m3Tdjc04UnTsGTaN3xTbjiuFHnNlOva3mz5fr0oXc+wrVX/wJFbebFjGTlDFz5D2XhHRpjOrpjK5FyhB/OlYKGUjScpNUHqxKia+OmoKQotiaot2vErZLANGzoKyEPCgs++CQGP1yZUEBzGthAw5rJnhPcSJUSZW+AsyRmC0T3edzjarKIWx+WaXsje5wybmTZXKSGBPrqna3qmO7qhR3r/tVoQVimrOeBTq3Clk40dda28/svK8+lj94v1p2ofOYyHag1W74RIuQ+9wi8eHj+vTC73Bf10Tk+s/4we6JY7sIov+sWSXD5BNBxB5Z8TP53PEawNpdTR1NDSSHJ6pjqMRnSjFwP842OYxhwWkeZ393GKS1xF3pQeZUAZrKQqkSqnE99MGf4AWgiSUg==</latexit>

a b
Ji Ji

replica=machines learning in parallelwith the same training data
1
2
3
4
5

Replica Symmetry Breaking =Replica “Permutation” Symmetry Breaking

replica=machines learning in parallelwith the same training data
1
2
3
4
5

Hierarchical Replica (permutation) symmetry breaking and ultrametricity
complexfree-energy landscape<latexit sha1_base64="ITIrUg+zwFMwhAD+SWv9W+6jMqo=">AAACbHichVG7SgNRED1ZXzE+Eh+FIEIwKFZhVnzFSrSxjI8YQUV21xtdstldd2+CGvIDdlaCqRQExc+w8Qcs/AQRbBRsLJxsIiKizoU7M2fmDGfu1V3L9CXRQ0hpam5pbQu3Rzo6u7qjsZ7eNd8peobIGI7leOu65gvLtEVGmtIS664ntIJuiayeX6jVsyXh+aZjr8pDV2wVtF3bzJmGJhnKbu5psrxf2Y4lKJkiNTWlxn8GapICS6BhaSd2hU3swIGBIgoQsCE5tqDB57MBFQSXsS2UGfM4MoO6QAUR5ha5S3CHxmie713ONhqozXltph+wD7jDYabDUyqIY4Tu6Zpe6I5u6JHef51WDqbU1Byy1+tc4W5HjwdW3v5lFdhL7H2x/lQtkcNMoNZk9W6A1PYw6vzS0enLyuzySHmULuiJ9Z/TA93yBnbp1bhcEstVRPgLPt85/nuwNp5UJ5O0NJGYm298RhiDGMYYv/g05rCINDKBuhOcoRp6VvqVQWWo3qqEGpw+fDNl9AO7XI2T</latexit>

overlap matrix

ultrametricityq̂Overlap distribution<latexit sha1_base64="kcF3SCKr8QjHDrEYpCFwjzclvT0=">AAACoXichVFNS+NQFD3G0akfo7WzEdwUS8XZlJuhqBUE0Y3u2jr9gColSV81mCYhSYs19A/MH3AhCArDKP4Fd7OZPzALf4LMUmE2s/DmA0REvSF55517z81576q2obse0e2QNPxhZPRjYmx8YvLT1HRyJlV1ra6jiYpmGZZTVxVXGLopKp7uGaJuO0LpqIaoqYebQb7WE46rW+Y3r2+LvY6yb+ptXVM8pprJVHGx9GVtt+0omt86Gvit0qCZzFCuQHJhSU6/BHKOwsggjqKVvMEuWrCgoYsOBEx4jA0ocPlpQAbBZm4PPnMOIz3MCwwwztouVwmuUJg95O8+7xoxa/I+6OmGao3/YvDrsDKNLP2hK7qn33RNd/T/1V5+2CPw0udVjbTCbk5/n935966qw6uHgyfVm549tLESetXZux0ywSm0SN87PrnfWS1n/QW6oL/s/5xu6RefwOw9aD9Konz6hh+bXfQ5E9yeG9xdNKow0hFYzseg8DSq6tecvJTLl/KZ9Y14aAnMYR6LPJllrGMLRVS45xHO8BOXUkbalopSOSqVhmLNZzwLqfEIl+aaTA==</latexit>

distance

1<latexit sha1_base64="DX86leatul5UFSW1Q2u7LsfJOc0=">AAACjnichVE9S8NQFD2NX7V+VV0EF7FUnMqLiIogFl06ttWqUIsk8bU+TJOQvBZq8Q/oqjg4KTiIf8HNxT/g0J8gjgouDt6kAdGi3pC888695+a8d3XHFJ5krBVRurp7evui/bGBwaHhkfjo2JZn11yDFwzbtN0dXfO4KSxekEKafMdxuVbVTb6tH677+e06dz1hW5uy4fBSVatYoiwMTRKVU/fiCZZiQUx1AjUECYSRteP32MU+bBiooQoOC5KwCQ0ePUWoYHCIK6FJnEtIBHmOY8RIW6MqThUasYf0rdCuGLIW7f2eXqA26C8mvS4pp5BkT+yWvbJHdsee2cevvZpBD99Lg1a9reXO3sjJxMb7v6oqrRIHX6o/PUuUsRR4FeTdCRj/FEZbXz+6eN1YziebM+yavZD/K9ZiD3QCq/5m3OR4/vIPPw65aFDGvz3PvzsalfpzMJ1gay6lLqTmcvOJ9Fo4tCgmMY1Zmswi0sggiwL15DjFGc6VuLKgrCir7VIlEmrG8S2UzCd5rpIa</latexit>

0 Q(x)<latexit sha1_base64="VKwTklyRFSWcSDHjsrPLraBQugE=">AAACjnichVHLSsNQED3G9zvqRnAjloqrMimiIoiiG5dW7QOqSBJva2heJGmhFn9At4oLVwouxF9w58YfcOEniMsKblw4SQOiok5I7rln5kzOvaO5puEHRE9tUntHZ1d3T29f/8Dg0LA8Mprznaqni6zumI5X0FRfmIYtsoERmKLgekK1NFPktcpamM/XhOcbjr0d1F2xa6ll2ygZuhowlaE9OUEpimLyJ1BikEAcG458hx3sw4GOKiwI2AgYm1Dh81OEAoLL3C4azHmMjCgvcIQ+1la5SnCFymyFv2XeFWPW5n3Y04/UOv/F5Ndj5SSS9Eg31KQHuqVnev+1VyPqEXqp86q1tMLdGz4e33r7V2XxGuDgU/Wn5wAlLEReDfbuRkx4Cr2lrx2eN7cWN5ONabqiF/Z/SU90zyewa6/6dUZsXvzhx2UXdc6Et+eHd8ejUr4P5ifIpVPKXCqdmU2srMZD68EEpjDDk5nHCtaxgSz3FDjBKc4kWZqTlqTlVqnUFmvG8CWk9Q93hJIZ</latexit><latexit sha1_base64="Qo0ZJ0HaJgDnBR3E3PRAdJbiLpc=">AAACpnichVHNLgNRFD4df1U/LRYkNo2G1KY5Iw1qJWysRFGVIDUzbmvS6cxk5rZpTfoCXsDCCrEQvIGdjRew8AhiSWJj4cydSUQEZzJzv/ud85357j2qbeguR3yKSB2dXd090d5YX//AYDwxNLzlWnVHYwXNMixnW1VcZugmK3CdG2zbdphSUw1WVKvLfr7YYI6rW+Ymb9lsr6ZUTL2sawonqpQYzad3OWty0clTDUWrtr1me7qUSGEmh3JuVk7+BHIGRaQgjDUrcQe7cAAWaFCHGjAwgRM2QAGXnh2QAcEmbg884hxCusgzaEOMtHWqYlShEFulb4V2OyFr0t7v6Qq1Rn8x6HVImYRJfMQrfMUHvMZn/Pi1lyd6+F5atKqBltml+PHYxvu/qhqtHA6/VH965lCGeeFVJ++2YPxTaIG+cXTyurGwPulN4Tm+kP8zfMJ7OoHZeNMu82z99A8/NrloUca/Pde/u2BUIpIBmMuGIPc1qq2ZjDybyeazqcWlcGhRGIcJSNNk5mARVmANCsLhBdzArZSWVqWCVAxKpUioGYFvIe1/AjKrnR0=</latexit>x<latexit sha1_base64="PDEynTrDTzj0DHd/M5fJCZKLhgY=">AAACjnichVE9S8NQFD2NX/WzURfBpVgUp/JSpIogFl06ttWqoCJJfOqjaRKS12It/gFdFQcnBQfxL7i5+Acc/AniqODi4E0aEBX1huSdd+49N+e9a7iW8CVjjzGlrb2jsyve3dPb1z+QUAeHVnyn5pm8bDqW460Zus8tYfOyFNLia67H9aph8VWjshjkV+vc84VjL8uGyzer+q4tdoSpS6KK+1tqiqVZGMmfQItAClEUHPUWG9iGAxM1VMFhQxK2oMOnZx0aGFziNtEkziMkwjzHIXpIW6MqThU6sRX67tJuPWJt2gc9/VBt0l8sej1SJjHOHtg1e2H37IY9sfdfezXDHoGXBq1GS8vdrcTRyNLbv6oqrRJ7n6o/PUvsYCb0Ksi7GzLBKcyWvn5w9rI0WxpvTrBL9kz+L9gju6MT2PVX86rIS+d/+HHJRYMywe35wd3RqLTvg/kJVjJpLZvOFKdSuYVoaHGMYgyTNJlp5JBHAWXqyXGME5wqqpJV5pT5VqkSizTD+BJK/gMTY5Jh</latexit>

Replica symmetry breaking
G. Parisi (1979)
ﬁrst found in the SK model(mean-ﬁeld spin glass)

a

b

c

Similarity<latexit sha1_base64="yQBe8wqt39GutSWqDFLqIABYIAk=">AAAC03ichVFNTxRBEH0MoIgoK15MvGzcQDhtajYGxYSE6MVwMCywQMLCpGfsxQ7z5czsGuzMxXgy8YyJJ004EP6DF2PgD3jgJxiPmHjxYE3vEGKIWpPpfvW6X1V1lRv7Ks2ITgaswaHhS5dHroxeHbt2fbxyY2I1jbqJJ1te5EfJuitS6atQtjKV+XI9TqQIXF+uuTuPivO1nkxSFYUr2W4sNwOxHaqO8kTGlFNpNB0t3Hyu3UmEp+1cP8nbaTdwtJqz8y32FhgyEGfAzZ1KjeqzZM/O2NWLwK6TsRpKW4wqX9DGU0Tw0EUAiRAZYx8CKX8bsEGImduEZi5hpMy5RI5R1nb5luQbgtkdXrfZ2yjZkP0iZmrUHmfx+U9YWcUkfaUDOqVjOqRv9OuvsbSJUdSyy7vb18rYGX9za/nnf1UB7xmenav+WXOGDu6bWhXXHhumeIXX1/de7p0uP1ia1FP0kb5z/R/ohD7zC8LeD2+/KZfem+gha16Y1wYmf8j91cy7nDUvkV/mTvGcsxR95X6a4fUnVL0Izoa32qjbM/VG825t/mE5xhHcxh1M86zuYR6PsYgWR3+HTzjCsdWytPXKet2/ag2Umpv4w6y3vwHAc6zd</latexit>

Qab =<latexit sha1_base64="Wi3xBIK3P0DO6Axk/5txD2MjBGM=">AAACh3ichVFNSxtRFD2Z1qrR1thuCm4GQyQBSe+UEE2h4Memy0SNBlTCzPQlfThfzEwCNrgv/QNddFVBRArdtns3/oEu/AnFZQQ3LrzzAUGk7R1m7nnn3XPfeXMNz5JBSHSZUR49HnsyPjGZnZp++mwmN/t8O3B7vimapmu5fsvQA2FJRzRDGVqi5flCtw1L7BgH69H+Tl/4gXSdrfDQE/u23nVkR5p6yFQ7N98o6otG6a062PNt1ZbOkVqMKLO0qDaKBudSO5enco20WlVTHwKtTHHkkUbdzZ1iD+/hwkQPNgQchIwt6Aj42YUGgsfcPgbM+YxkvC9whCxre1wluEJn9oC/XV7tpqzD66hnEKtNPsXi12eligL9pjMa0gV9pz90+9deg7hH5OWQs5Fohdee+fxy8+a/KptziA8j1T89h+hgOfYq2bsXM9EtzETf//hluPlmozBYoGO6Yv/f6JLO+QZO/9o8aYiNr8gmA4hDTcBSJQW10QC2X5e1arnSqORX1tJRTGAO8yjy/17CCt6hjiaf+wk/8BO/lEnllVJVlpNSJZNqXuBeKKt3iF+USg==</latexit>

Q(a, b) = min(Q(a, c), Q(b, c))<latexit sha1_base64="eo5FZMdVTbgrvYek16WrtyU2QDw=">AAACZHichVFNS8NAEH2N37VqtQiCIMWieCoTKWo9Fb14tNZWQYskca3BNAlJWtDiH9Cr4sGTgoj4M7z4Bzz0DwjisYIXD06TgIios2z27dt5s28zqm3orkfUjEgdnV3dPb190f7YwOBQfHik5Fo1RxNFzTIsZ1NVXGHopih6umeITdsRSlU1xIZ6sNw+36gLx9Utc907tEW5qlRMfU/XFI+pfH4nnqJ0luTsnJz8CeQ0+ZFCGKtW/Bbb2IUFDTVUIWDCY2xAgctjCzIINnNlNJhzGOn+ucAxoqytcZbgDIXZA/5WeLcVsibv2zVdX63xLQZPh5VJTNET3VGLHumeXujj11oNv0bbyyGvaqAV9s7QyVjh/V9VlVcP+1+qPz172MOC71Vn77bPtF+hBfr60UWrsLg21Zima3pl/1fUpAd+gVl/027yYu0S0aABfiQDMJ8JQfarAaXZtDyXzuQzqdxS2IpejGMSM/y/55HDClZR5HsFTnGG88izFJMS0miQKkVCTQLfQpr4BH//irg=</latexit>

Q
N
X
1

N i=1

a b
Ji Ji

dx
P (Q) =
dQ

Multi-layer Neural Network
Design weights to satisfy boundary conditions

S0,1

S2,1

S1,1

<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit>

S3,1<latexit sha1_base64="(null)">(null)</latexit>

S0,2

S1,2

<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

NS0,3<latexit sha1_base64="(null)">(null)</latexit>

S0,N

S1,N

input

outputhiddenL
1
2
M
Si,l = (Si,l , Si,l , . . . , Si,l )<latexit sha1_base64="IQ1ZI7jiQ/Wy8+UMKWaYvXyVZfA=">AAACvnichVFNS9xAGH5Mq9WtH9v2IvQSulgUlmWyFloKwmIvvRTUdV3B6JJkZ9fBJBOTyaIN+QP+gR56aqHQ0t/QUy/9Az149iS2Jwu99NB3s0FRUd+BmWee933eeWbGDlwRKcYOh7Q7d4dH7o2OFe6PT0xOFR88XItkHDq84UhXhuu2FXFX+LyhhHL5ehByy7Nd3rR3XvXzzR4PIyH9VbUf8E3P6vqiIxxLEdUqNhPT7uj1tJWIspsuzNYHYCsx0vIZrqZl021LFZ1Tb9I5XdfN3d3YautnrOnF6YIZeLrRKpZYhWWhXwVGDkrIY0kWP8NEGxIOYnjg8KEIu7AQ0diAAYaAuE0kxIWERJbnSFEgbUxVnCosYndo7tJuI2d92vd7Rpl6jyokKSV1SaFjhv1kX9gp+8G+smP279puSdal72afVnug5UFr6mC6/vdWlUerwva56kbXCh28yNwKch9kTP8ezkDfe/vutP5yZSZ5yj6yE/L/gR2y73QDv/fH+bTMV96jQF9gXH7wq2CtWjHmK9XlZ6XaYv4Zo3iMJ5ilF3+OGl5jCQ069xuO8Au/tZrW0TxNDkq1oVzzCBdC2/sPJVqrwQ==</latexit>µSi,l = ±1↵ = M/N

1
J⌅

µ
S⌅(1)

S0,1

S2,1

S1,1

<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit>

S3,1<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅(2)

2
J⌅

S0,2

<latexit sha1_base64="(null)">(null)</latexit>

µ
S⌅

N
J⌅

S0,3
<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="PzUIaUxmlcfnXo6NBtE8K0QXcG4=">AAACeXichVHNLgNRFP46/uun9bOQ2KAhYtHcthJiJWzEQqoUCTWZGReTzp+ZaRMmfQEvYGGDRKQ8ho0XsOgjiCWJBQtnpiOIlHNz7z33O+c797v3yJamOi5jtYjQ1NzS2tbeEe3s6u6JxXv71h2zZCs8r5iaaW/KksM11eB5V3U1vmnZXNJljW/IxQU/vlHmtqOaxpp7ZPGCLu0b6p6qSC5BYnxgSfS2ZU1Sis5hSbJ5ZcdbrojxRCrJAhtmv5zPUAKhZc34NbaxCxMKStDBYcAlX4MEh8YWUmCwCCvAI8wmTw3iHBVEiVuiLE4ZEqFFWvfptBWiBp39mk7AVugWjaZNzGGMsQdWZc/snt2yR/besJYX1PC1HNEu17ncEmMng6uv/7J02l0cfLH+1OxiDzOBVpW0WwHiv0Kp88vHp8+rs7kxb5xdsifSf8Fq7I5eYJRflKsVnjtD9HsDGjvr6WQqk0yvTCXm5sNWtGMIo5ig/57GHBaRRZ7uPcY5qriJvAkjwoQwWU8VIiGnHz9MyHwAAzCSow==</latexit>µ
S⌅(N )

S1,2<latexit sha1_base64="(null)">(null)</latexit>µ
S⌅ = sgn<latexit sha1_base64="8TJBS5cQywa2sByQTfTnXTF/7rU=">AAACfnichVFNLwNBGH66vqq+igtxEQ3hoKYl0Tg1XJykVEuiNLtryqb7ZT+a1KZx9wccnEgaEQk/wsUfcPATxLESFwdvt5sIgncyM8888z7vPDMjmapiO4w9hYS29o7OrnB3pKe3r38gOjiUtw3XknlONlTD2pZEm6uKznOO4qh827S4qEkq35LKK839rQq3bMXQN52qyXc18UBXSoosOkQVo6PZPa+gubWiV5BUUS7bR65o8em1mVoxGmNx5sf4T5AIQAxBZIzoFQrYhwEZLjRw6HAIqxBhU9tBAgwmcbvwiLMIKf4+Rw0R0rqUxSlDJLZM4wGtdgJWp3Wzpu2rZTpFpW6RchyT7JFdswZ7YDfsmb3/WsvzazS9VGmWWlpuFgdOR7Jv/6o0mh0cfqr+9OyghJTvVSHvps80byG39JXjs0Z2aWPSm2KX7IX8X7Andk830Cuvcn2db5wjQh+Q+P7cP0E+GU/Mx5PrC7H0cvAVYYxhAtP03otIYxUZ5OjcE9RxizsBwpQwK8y1UoVQoBnGlxBSH/TUk20=</latexit><latexit sha1_base64="8fA3cAJF8y34tZihZo9XmxWhHFk=">AAAC8nichVFLaxRBEK4ZX3F9ZNWL4mXIEtlclp5EUIRA0It4kCTrJoF0dpjp9G6anVe6exZj039A8CzoKYqI6L/wkj+QQ36CeFzBiwdrZwfExEc13V31dX1VX3dHeSyUJuTIcU+dPnP23NT52oWLly5P169cXVNZIRnvsCzO5EYUKh6LlHe00DHfyCUPkyjm69Hgwfh8fcilEln6RO/lfCsJ+6noCRZqhIL6i3bX0KSwgaFRHLKB2i1Cye2ioTLxVD+1Ho15TzdpT4bM+NZQtSu1eWwtVUUSGLHo2y6GHtX8qS71GMm3rXl0rGLXCGv/2Kwp5iyVor+j54J6g7RIad5Jx6+cBlS2nNXfA4VtyIBBAQlwSEGjH0MICscm+EAgR2wLDGISPVGec7BQQ26BWRwzQkQHuPYx2qzQFONxTVWyGXaJcUpkejBLDskHMiIH5CP5Qn78tZYpa4y17OEeTbg8D6afX29//y8rwV3Dzi/WPzVr6MHdUqtA7XmJjG/BJvzhs5ej9r3VWXOLvCFfUf8+OSKf8Qbp8Bt7t8JXX0MNP8A//twnnbX5lr/Qml+53Vi6X33FFNyEGWjie9+BJXgIy9DBviPnhjPjNFztvnL33beTVNepONfgN3M//QSdqsPi</latexit>

0

N
X

1
i µ
p
J⌅ S⌅(i)
N i=1

N
X
1
µ
SL,i = sgn @ p
JL,i,j sgn
N j=1

!

S0,N

S1,N

input

N
X
1
p
JL 1,j,k · · · sgn
N k=1

N
X

1
µ
p
J1,l,m S0,m
N m=1

e.g.

N X
M ⇣
X
µ
E=
SL,i
i=1 µ=1(2) try to minimize the loss function
via back-propagation!!1
A

desired output

Usual strategy of learning(1) define “loss function"

outputhidden(S⇤ )µL,i⌘2e.g. SDG (stochastic gradient descent)Too much long-ranged, highly convoluted, non-linear interaction! …hard to analyzeGardner volume in deep perceptron networkµ
S⌅(1)<latexit sha1_base64="y3/5uQTl0AefBrBN7OHPZIr+t80=">AAACfnichVHLLgRBFD3aa4zXYENsJiaEhXFbZDxWwsYSY5AYJt2t0Jl+6cckdCb2fsDCikREJHyEjR+w8AliSWJj4U7PiIjgVqrq1ql7bp26V3UM3fOJHuuk+obGpuZYS7y1rb2jM9HVverZgauJnGYbtruuKp4wdEvkfN03xLrjCsVUDbGmFucr92sl4Xq6ba34B47YNJVdS9/RNcVnqJDoy26FeTMoF8K8aiha0dsPFFcMyyPlQiJF6WmSpzNy8qcjpymyFGq2aCcukcc2bGgIYELAgs++AQUejw3IIDiMbSJkzGVPj+4FyogzN+AowREKo0Ved/m0UUMtPldyehFb41cMni4zkxikB7qiF7qna3qi919zhVGOipYD3tUqVziFzuPe7Nu/LJN3H3tfrD81+9jBVKRVZ+1OhFR+oVX5pcOTl+zM8mA4ROf0zPrP6JHu+AdW6VW7WBLLp4hHDahWOfnT+WzA6nhazqTHlyZSs3O1VsTQjwEMc70nMYsFLCLH7x7hAje4lSANSaPSWDVUqqtxevDNpKkPUQ+Tng==</latexit>

1J⌅<latexit sha1_base64="5M2VOZ6Jor/BMqssU9qvzawOdlY=">AAACeXichVHNTttAEP5ioEAoTfg5IHEJRCDUQzQOEBJOCC5VT0kgCVJCI9ssYMWxje1EAisvwAtw4EIroYr2MXrpC/SQR0AcQeIAh46doKpCtLPa3dlv55v9dka1Dd31iLoRaWBw6M3wyGh07O34u1h8YrLsWi1HEyXNMixnV1VcYeimKHm6Z4hd2xFKUzVERW1sBfeVtnBc3TJ3vBNb7DWVQ1M/0DXFY6gen/5Y92uqoWgN97ilOKLzyZc79XiSUrlshlbSCUoRZdOUYWeV5JycS8iMBJZE3/JW/Ctq2IcFDS00IWDCY9+AApdHFTIINmN78Blz2NPDe4EOosxtcZTgCIXRBq+HfKr2UZPPQU43ZGv8isHTYWYCC/SLrumOftJ3uqGnV3P5YY5Aywnvao8r7HrsbGb74b+sJu8ejv6w/qnZwwGyoVadtdshEvxC6/Hbp+d32+vFBX+RvtAt6/9MXfrBPzDb99pVQRQvEA0b0Kty4qXz3IByOiVnUunCSnJjs9+KEcxiHktc7zVs4APyKPG7p7jENb5FHqU5aUl63wuVIn3OFP4yafk3TheSyw==</latexit>

S0,1

S2,1

S1,1

<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit>

S3,1<latexit sha1_base64="(null)">(null)</latexit>

2J⌅<latexit sha1_base64="mzaNe/Lhoic+b4HbgdvjR0/kYsc=">AAACeXichVHNTttAEP7iFhrSFsLPoVIvgSgIcYjGbkgTTqi9IE4QCETiJ7LNAlYc29hOpMTKC/ACHHopSBUCHoMLL8Ahj4B6BIlDe+jYCUIItZ3V7s5+O9/stzOaYxqeT9SNSa9eDwy+iQ8l3r57PzySHB1b9+yGq4uybpu2W9FUT5iGJcq+4Zui4rhCrWum2NBqX8P7jaZwPcO21vyWI7br6r5l7Bm66jNUTU4sVYMtzVT1mnfYUF3R2QmUTjWZpmyxkKeckqIsUUGhPDtzJBflYkpmJLQ0+rZsJ8+whV3Y0NFAHQIWfPZNqPB4bEIGwWFsGwFjLntGdC/QQYK5DY4SHKEyWuN1n0+bfdTic5jTi9g6v2LydJmZQoZu6Jzu6Jou6ZZ+/zVXEOUItbR413pc4VRHjj6sPvyXVefdx8ET65+afeyhEGk1WLsTIeEv9B6/2T6+W50vZYJpOqWfrP+EunTFP7Ca9/qPFVH6hkTUgF6VUy+dxwasK1k5n1VWcumFL/1WxPERU5jhen/GAhaxjDK/28Z3nOMi9kualGak2V6oFOtzxvHMpE9/AFAYksw=</latexit>µ
S⌅(2)<latexit sha1_base64="c8zQVbhO4Jb0y1OQjRAbgi23O3g=">AAACfnichVHLLgRBFD3ae7wGG2IzMSEsjNsT8VoJG0uMQWKYdLcaOtMv/ZiEzsTeD1hYkYiIhI+w8QMWPkEsSWws3OkZERHcSlXdOnXPrVP3qo6hez7RY51U39DY1NzSGmtr7+jsinf3rHl24Goiq9mG7W6oiicM3RJZX/cNseG4QjFVQ6yrxYXK/XpJuJ5uW6v+gSO2TGXX0gu6pvgM5eP9me0wZwblfJhTDUUrevuB4oqR9Gg5H09SaobkmUk58dORUxRZEjVbsuOXyGEHNjQEMCFgwWffgAKPxyZkEBzGthAy5rKnR/cCZcSYG3CU4AiF0SKvu3zarKEWnys5vYit8SsGT5eZCQzRA13RC93TNT3R+6+5wihHRcsB72qVK5x813Ff5u1flsm7j70v1p+afRQwHWnVWbsTIZVfaFV+6fDkJTO7MhQO0zk9s/4zeqQ7/oFVetUulsXKKWJRA6pVTvx0Phuwlk7Jk6n08kRybr7WihYMYBAjXO8pzGERS8jyu0e4wA1uJUjD0pg0Xg2V6mqcXnwzafoDUxGTnw==</latexit>

S0,2

µ
S⌅

S1,2

<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="N3pgLctjw+/Q3ixYK/R0jPl4BzY=">AAACe3ichVHLLgRBFD3T3uMxg0QkNhMTIiKT2+K9EjaWGIPEMOluhc70Sz8moc0P+AELKxIR4S9s/ICFTxBLEhsSd3pGRAS3UlW3Tt1z69S9qmPonk/0EJPq6hsam5pb4q1t7R2JZGfXqmcHriZymm3Y7rqqeMLQLZHzdd8Q644rFFM1xJpanK/cr5WE6+m2teIfOGLTVHYtfUfXFJ+hQrInWwjzqqFoRW8/UFxR3grzZlAuJNOUmSZ5ekJO/XTkDEWWRs0W7eQl8tiGDQ0BTAhY8Nk3oMDjsQEZBIexTYSMuezp0b1AGXHmBhwlOEJhtMjrLp82aqjF50pOL2Jr/IrB02VmCgN0T1f0THd0TY/09muuMMpR0XLAu1rlCqeQOO7Nvv7LMnn3sffF+lOzjx1MRVp11u5ESOUXWpVfOjx5zs4sD4SDdE5PrP+MHuiWf2CVXrSLJbF8injUgGqVUz+dzwasjmbkiczo0lh6dq7Wimb0oR9DXO9JzGIBi8jxu0c4xzVuYu9SWhqWRqqhUqzG6cY3k8Y/AOaek/0=</latexit>

N
J⌅

S0,3
<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="mAwaG5t9Ln5IJahlAurqNSsC4Bg=">AAACeXichVHNTttAEP5iKD9pgfBzQOKSEgUhDtE4hJBwQvSCOKAEGoLET2SbJVhxbGM7kcDKC/ACHLhAJYTSPgYXXoBDHqHqEaQe2kPHTqqqqgqz2t3Zb+eb/XZGtQ3d9Yg6Eamv/83A4NBw9O27kdGx2PjEjms1HE2UNMuwnF1VcYWhm6Lk6Z4hdm1HKHXVEGW19iG4LzeF4+qW+dE7s8VBXama+rGuKR5DldjURsXfVw1Fq7mnDcURrUN/s1WJJSiVz2Upk45TiiiXpiw7SyTn5XxcZiSwBHpWsGJ32McRLGhooA4BEx77BhS4PPYgg2AzdgCfMYc9PbwXaCHK3AZHCY5QGK3xWuXTXg81+RzkdEO2xq8YPB1mxpGkR2rTEz3QF/pKP/+byw9zBFrOeFe7XGFXxi6mt7+/yqrz7uHkD+tFzR6OkQu16qzdDpHgF1qX3zy/fNpe2Ur6c/SJvrH+G+rQPf/AbD5rt0WxdYVo2IBuleP/Or8bsJNOydlUuphJrK71WjGEGcxinuu9jFWso4ASv3uOa7TxOfJDei/NSwvdUCnS40ziL5MWfwGINJLo</latexit><latexit sha1_base64="0NsuDkD+SRBOKqtfC197SiHhO7s=">AAAC9nichVFLaxRBEK4ZX3F9ZKMXQYTBJXFzWXpCiEYIBL2IB0mybhJIZ4eesXfT7LzS3bMkNv0HvHnyIAgKIYgX/4MX/4CH/ISQY4RcPFg7uyIaHzVMd9XX9VV93RXmsVCakAPHPXP23PkLYxcrly5fuTpenbi2qrJCRrwVZXEm10OmeCxS3tJCx3w9l5wlYczXwt7Dwflan0slsvSp3s35ZsK6qeiIiGmEgurLZtvQpLCBoWHMop7aLpjkdsFQmXiqm1qPxryj67QjWWR8a6jalto8sZaqIgmMWPBtG0OPar6jSz0GW/BUM2se/1a1bYS1f2xYF9OWStHd0tNBtUYa88Sfn/O9047fIKXVYGRLWXUfKDyDDCIoIAEOKWj0Y2Cg8NsAHwjkiG2CQUyiJ8pzDhYqyC0wi2MGQ7SHaxejjRGaYjyoqUp2hF1i/CUyPZgkX8h7ckw+kw/kkHz7ay1T1hho2cU9HHJ5Hoy/uNE8+S8rwV3D1k/WPzVr6MC9UqtA7XmJDG4RDfn956+Om/dXJs0UeUeOUP9bckA+4Q3S/tdob5mvvIZKOYDhK3unnR8DWJ1p+HONmeXZ2uKD0SjG4Cbchjq+911YhEewBC3se+LccqacO+6O+8bdc/eHqa4z4lyHX8z9+B34nsX6</latexit>

µ
S⌅
= sgn

µ
S⌅(N )<latexit sha1_base64="wQtXVD3ucQmwH1t5oksdFNZ42OY=">AAACfnichVHLLgRBFD3ae7wGG2IzMSEsjNsiXithYyUYg8Qw6W41dKZf+jEJnYm9H7CwIhERCR9h4wcsfIJYkthYuNMzIiK4laq6deqeW6fuVR1D93yixxqptq6+obGpOdbS2tbeEe/sWvPswNVERrMN291QFU8YuiUyvu4bYsNxhWKqhlhXC/Pl+/WicD3dtlb9A0dsmcquped1TfEZysV709th1gxKuTCrGopW8PYDxRVDi8OlXDxJqWmSpyfkxE9HTlFkSVRtyY5fIosd2NAQwISABZ99Awo8HpuQQXAY20LImMueHt0LlBBjbsBRgiMURgu87vJps4pafC7n9CK2xq8YPF1mJjBAD3RFL3RP1/RE77/mCqMcZS0HvKsVrnByHcc96bd/WSbvPva+WH9q9pHHVKRVZ+1OhJR/oVX4xcOTl/TMykA4SOf0zPrP6JHu+AdW8VW7WBYrp4hFDahUOfHT+WzA2lhKnkiNLY8nZ+eqrWhCH/oxxPWexCwWsIQMv3uEC9zgVoI0KI1Io5VQqabK6cY3k6Y+AItJk7s=</latexit>

N
X

1
i µ
p
J⌅
S⌅(i)
N i=1

!

S0,N

S1,N

input

outputhiddenGardner volume generalized for a multi-layer network(c.f.. ) Single percepron: E. Gardner (1987)

trace over hidden variables<latexit sha1_base64="dlbXGJKzLhma8gof35Tub2xoxw0=">AAAD3XichVFNT9RAGH53FxWrfOnFxMtEAtka2ExXg2KyCdELMUCAlY+EYTdtmYWBftHOrsHJHL0Y45WDJ02MMf4ML/4BD1y8G4+YePHgu+02iqBOM523zzzP0+edcSJPJJLSo0Kx1Hfu/IX+i8alywODQ8MjV1aTsB27fMUNvTBed+yEeyLgK1JIj69HMbd9x+Nrzt6D7v5ah8eJCINH8iDim769HYiWcG2JUHOk8HicrJYVc1qkrsvUnMjLOdOs8YZaIPNEMdf2utgZNM80tVFjHm/JMovicKupvJqlG2pu0tI9QKTAgmZJ22+qekMxv62RNyF0jUU+QWIstnekaWQ+hIlAEoP05MzxbHcv2W/bMc8td3PLVmy7auvhH6yG2tVasWQ/lqrKIqE1MbCZyYxePotuNlQVNTjzMKnA4dIms5oxY5ywnSSyXX5T3XJ9PVszsn6wmTTMvM6B36075Tjv9wRuNodHaWWaWtNTFjldWBWajlHojcVw+C0w2IIQXGiDDxwCkFh7YEOCzwZYQCFCbBMUYjFWIt3noMFAbRtZHBk2onv43savjR4a4HfXM0nVLv7FwxmjksAY/UTf0WP6kb6nX+iPv3qp1KOb5QBXJ9PyqDn07Fr9+39VPq4Sdn6p/plZQgvuplkFZo9SpNuFm+k7Tw6P6/eWx9Q4fU2/Yv5X9Ih+wA6Czjf3zRJffglGegHZKZPTRX4Bq9WKNVWpLt0enbnfu4p+uA43oIznfQdmYBYWYQXcwudiX3GgOFhqlp6WnpdeZNRioae5CidG6fAnnmkT2A==</latexit>

V (S(0), S(L)) = e
Hamiltonian with“short-ranged” interactions

N M S(S(0),S(l))

H=

M X
X

µ=1 ⌅

0

=@

L
N
Y1 Y

µ
l=1 i=1 Sl,i
=±1

µ
v(r⌅
)

“Hardcore” constraint<latexit sha1_base64="1kOUsL+mXnLV+PXT9v+GEY+Oo0w=">AAACfnichVFNLwNRFD0dX1VfxYbYNBrCQt1ppEoiETaWvopEaWbGoxPTmcnMaxMasfcHLKxIGhEJP8LGH7DwE8SSxMbC7bQiIrgv7717z7vnvvPe1V3L9CXRY0hpaGxqbgm3RtraOzq7ot09a75T9AyRMRzL8TZ0zReWaYuMNKUlNlxPaAXdEuv6/nz1fL0kPN907FV54IqtgrZnm7umoUmGctF+sV0ey+pCarHSSH70aCYr8xywm4vGKTFF6lRKjf101AQFFkfdFp3oJbLYgQMDRRQgYEOyb0GDz2MTKgguY1soM+axZwbnAkeIMLfIWYIzNEb3ed3jaLOO2hxXa/oB2+BbLJ4eM2MYoge6ohe6p2t6ovdfa5WDGlUtB7zrNa5wc10nfStv/7IKvEvkv1h/apbYRTrQarJ2N0CqrzBq/NLh6cvK9PJQeZgu6Jn1n9Mj3fEL7NKrUVkSy2eIBA2o/XLsp/PZgLVkQk0lkksT8dm5eivCGMAgRvi/JzGLBSwiw/ceo4Ib3CpQhpUxZbyWqoTqnF58MyX9ARswkos=</latexit>

e

v(h)

X

= ✓(h)

10
A@

“gap”

Z YY
N

⌅ j=1

j
dJ⌅

p

2⇡

(J

e

j 2
)
⌅
2

1

Ae

H

N
X
1 i µ
µ
µ
p J⌅ S⌅(i) S⌅
r⌅ =
N
i=1
<latexit sha1_base64="pX04t51jI+95sRRPfnFJgV6GbsA=">AAAC+3ichVHNThRBEK4Z/MH1hwUvJl4mbDAYddNDDIgJCdGL8WCAdYGEYSc9bS90dv7o7lnFTr+AD6AHT2qMUbj6BF58AQ88gvGIiRcO1s6uMQpKTaa7+qv6vqruivJYKE3InuMOnTh56vTwmcrZc+cvjFRHx5ZVVkjGmyyLM7kaUcVjkfKmFjrmq7nkNIlivhJ17vbiK10ulcjSh3o75+sJ3UhFWzCqEQqrz2XLBElhQxNEMWUdtVVQye2cF6giCY2Y823LPLBBW1JmfGsCtSU1AjbQ/Iku65vHm0Jza+7/pdEywtrGUfKT4qr1jozYG0GH5jkNqzVSnyX+7LTvHXb8OimtBgNbyKrvIIBHkAGDAhLgkIJGPwYKCr818IFAjtg6GMQkeqKMc7BQQW6BWRwzKKIdXDfwtDZAUzz3NFXJZlglxl8i04MJ8oW8J/vkM9khX8nBP7VMqdHrZRv3qM/leTjy7FLjx7GsBHcNm79Z/+1ZQxtulb0K7D0vkd4tWJ/fffpiv3F7acJcIa/JN+z/Fdkjn/AGafc7e7vIl15CpRxA/5W9w86vASxP1f3p+tTizdr8ncEohuEyjMMkvvcMzMM9WIAm1j1wxp1rznXXum/cD+5uP9V1BpyL8Ie5H38CkzLI4A==</latexit>We study two scenarios of machine learningwith artificially generated training data
randominput

randomoutput<latexit sha1_base64="kZCI/UwBgv6UVlm3dMhkzoyGe5I=">AAACaXichVFNLwNRFD0d3/XVshE2jYZYNXe0UbUSNmLlqyVBmpnx8JjOTGamTWj8ASs7wYpERPwMG3/Awk8QSxIbC3emE2KB+/LmnXfePfedN1d3TOn5RE8xpam5pbWtvSPe2dXd05tI9pU8u+oaomjYpu2u6ZonTGmJoi99U6w5rtAquilW9f3Z4Hy1JlxP2taKf+CIzYq2Y8ltaWg+U6X5cl3uHZUTacoUgphINUC2wIByKmWzKTVDYaQRxYKduMEGtmDDQBUVCFjwGZvQ4PFYhwqCw9wm6sy5jGR4LnCEOGurnCU4Q2N2n787vFuPWIv3QU0vVBt8i8nTZWUKI/RIt/RKD3RHz/Txa616WCPwcsCr3tAKp9x7PLD8/q+qwquP3W/Vn559bGMy9CrZuxMywSuMhr52ePq6PLU0Uh+lK3ph/5f0RPf8Aqv2ZlwviqULxL8bUIgakM9FoKB+NaA0nlEnMupiLj09E7WiHUMYxhj/7zymMYcFFPnePZzgDOexFyWpDCiDjVQlFmn68SOU9CfFEY0q</latexit>

Jij

Randominput

outputrandom teacher
teacher
Jij

student

student
Jij

scenario (１）scenario（2）Scenario (1) Random inputs/random outputs- a constraint satisfaction problem
2. A Potts anti-ferromagnet on random graphs
It is immediate to realize that the q-coloring problem is equivalent to the question of determining
if the ground-state energy of a Potts anti-ferromagnet on a random graph is zero or not [8].
Consider indeed a graph G = (V, E) defined by its vertices V = {1, . . . , N } and edges (i, j) ∈ E
which connect pairs of vertices i, j ∈ V; and the Hamiltonian- glass/jamming physics
H({s}) =!

δ(si , sj ) .

(1)(i,j)∈E

With this choice there is no energy contribution for neighbors with different colors,
but a positive contribution otherwise.
The ground state energy (the energy at zerotemperature) is thus zero if and only if the graph is q-colorable. This transforms thecoloring problem into a well-defined statistical physics model. Usually, two types of randomgraphs are considered: in the c−regular ensemble all points are connected to exactly
c neighbors, while in the Erdős-Rényi case the connectivity has a Poisson distribution.

“Random Constraint Satisfaction Problem”(ランダム制約充足問題）

synaptic weights

Jij

3. Cavity method: Warnings, Beliefs and Surveys
Over the last few years, a number of studies haveinvestigated CSPs following the adaptation of the so-calledcavity method [2] to random graphs [4, 9]. It is a powerfulheuristic tool —whose exactness is widely accepted but
has still to be rigorously demonstrated— equivalent to thereplica method of disordered systems [2]. Its main idea
lies in the fact that a large random graph is locally treelike, and that an iterative procedure known in physics as
the Bethe-Peirls method can solve exactly any model ona tree (such models are often qualified as “mean field” Figure 1. Example: a proper 3in physics). Interestingly, it was realized [10] that an coloring of a small graph.
equivalent formalism has been developed independently incomputer science [11], where it is called Belief Propagation(BP, which conveniently enough, may also stands for
Bethe-Peirls). Defining ψci→j (c = 1, .., q) as the probability that the spin i has color c in absenceof the spin j (the “belief” that the spin j has on the properties of the spin i), BP reads
i j

Example:Graph coloring

randominput

machinerandomoutput

Q: How may diﬀerent ways the machine
can be designed to satisfy theimposed random inputs/outputs ?

Antiferromagnetic Potts modelH=ψci→j =

1"

Z0i→j k∈N (i)\{j}#

1 − ψck→i

$

X
i,j

q ,q

(2)

Connection to glass physics

where Z0i→j is a normalization constant and the notation k ∈ N (i) \ {j} means the set ofneighbors of i except j. From a fixed point of these equations, the complete beliefs in presenceof all spins can be also computed. They give, for each vertex, the probability of each color fromwhich other quantities, as for instance the number of solutions, can be computed. A simpler
formalism, called Warning Propagation (WP), restricts itself to frozen variables (i.e. to variables
for which only one color can satisfy the constraints). However, WP does not allow to computethe number of solutions, only their existence, but is definitely simpler to handle.
It was soon realized, however, that these methods developed for trees could not be
used straightforwardly on all random graphs because of a non-trivial phenomenon called
clustering [9, 12] (for which rigorous results are now available, see [13]). Indeed, while for

Clustering transition = glass transition
SAT/UNSAT transition = jamming transition

Replicated Gardner volume<latexit sha1_base64="1dUrqnP4O481fHpLXmiEIRwheDU=">AAADqnichZHNbtNAEMfHDdBiPprCBYmLRdQolUK0yaFQpEoVcECIQ5s0aVHcRGtnk1ix1+56HRSsfQFegAMnkBBCfQwuXDlwKDwB4lgkLhwY2yGlVMBa3p2dmd9/Z2etwHVCScihNpc7c/bc/MJ5/cLFS5cX80tXWqEfCZs1bd/1xa5FQ+Y6nDWlI122GwhGPctlO9boXhLfGTMROj7flpOA7Xl0wJ2+Y1OJru6Sdt/k7Intex7lvdi0QoWTS+1RuB9RwZRuWmzg8JjtcyoEnSi91Ym5Ml3WlyVM7RsN1Y2JKs/sR8oUzmAoV/TietEMhN/rxnS9qhJMz7ipNznNiE3hGdsCwVThIeZRlcVmQhllHGOmlVTo0nCY8X4kg0gq9YdY43cxw5jJTXW8qIyRMlU668Q38aKSGuOS6J7oAIY7SapaUYbJfR55FhOGbjJs16wp3XyBVNZIdW21apw2qhWSjgJMx6affwMm9MAHGyLwgAEHibYLFEL82lAFAgH69iBGn0DLSeMMFOjIRpjFMIOid4TzAHftqZfjPtEMU9rGU1z8BZIGLJOP5C05Iu/JAflCfvxVK041klomuFoZy4Lu4rNrje//pTxcJQyPqX/WLKEPt9NaHaw9SD3JLeyMHz99ftS4U1+Oi+QV+Yr1vySH5B3egI+/2a+3WP0F6OkDZF02Thu/HqBVq1RXK7WtWmHj7vQpFuA63IAS9vsWbMAD2IQm2NqB9kH7pH3OlXP13ONcO0ud06bMVTgxcr2f/PMHUw==</latexit>n

V (S0 , SL )

=

n
Y

a=1

Y
⌅

TrJa⌅

!0
@

replicated machines
Order parameters<latexit sha1_base64="GTxtaHLJa7YedtM7CfyEvvH72cc=">AAADNXichVHNahRBEK4Z/+L6k425CF4Gl8iuyNITQjSBQNCLCAnZrJsEMtmhe9Ibm8zfTs+sxKFfwLPgwZOCB8kD+AAi+AIecvAmgnjwEEEQD9b0jqis0R6mu+rr+r6qrmKxL2RKyIFhHjt+4uSpsdOVM2fPnR+vTlxYk1GWeLzjRX6UbDAquS9C3klF6vONOOE0YD5fZ7u3ivv1AU+kiMK76V7MtwK6E4qe8GiKkFt91Hdzyq45TKoFp5dQL7dVvqQcmQVu7gTZgq266Nfb6GFMt8BUo5vTUYgpy+n3M7pdaf1Fc7nUFFpxWd3R7LpoqELsd48pt1ojzTliz83a1qhhN4leNSjXSlR9DQ5sQwQeZBAAhxBStH2gIPHbBBsIxIhtQY5YgpbQ9xwUVJCbYRTHCIroLu476G2WaIh+oSk128MsPv4JMi2YIm/JC3JI3pB98pF8P1Ir1xpFLXt4siGXx+74w4vtr/9lBXimcO8X6581p9CDG7pWgbXHGile4Q35gwePD9vzq1P5FfKMfML6n5ID8gpfEA6+eM9bfPWJVg+Rc1+/NtD5Q+xvjjjDrKq0/DK3hD5mKfqK/dTDG07IGjV+Dm9tumnPNqdbM7XFm+UYx+ASXIY6zuo6LMJtWIEOqn8zLKNhXDVfmu/M9+aHYahplJxJ+GOZn38A1eXSiA==</latexit>qab,⌅ =

replicas: machines learning in parallel with the same dataY⌅\output

1

TrSa⌅ AYeµv(r⌅,a )µ,⌅,aa = 1, 2, . . . , n<latexit sha1_base64="CR7z0kBBXxEO63X7ypvkC8hVLl8=">AAACm3ichVHLSsNQED3GV31X3QgiFEvFRSmTIvUBguhGxIWvaqE+SNKrBtMkJLeFWvwBf0DBlYIL8RfcufEHXPgJ4lLBjQsnaUWkqBOSO/fMnMm59+iuZfqS6KlJaW5pbWuPdHR2dff09kX7BzZ9p+QZIms4luPldM0XlmmLrDSlJXKuJ7Sibokt/WghqG+Vheebjr0hK67YKWoHtrlvGppkaFebVZPp5LZVcKSftPeicUpNkzqdUWONiZqiMOKox4oTvcM2CnBgoIQiBGxIzi1o8PnJQwXBZWwHVcY8zsywLnCCTuaWuEtwh8boEX8PeJevozbvg5l+yDb4Lxa/HjNjSNAj3dArPdAtPdPHr7Oq4YxAS4VXvcYV7l7f6dD6+7+sIq8Sh9+sPzVL7GMq1GqydjdEglMYNX75+Ox1fWYtUR2jK3ph/Zf0RPd8Arv8ZlyvirWLP/S4rKLCleD2/ODuQqtqfsQaky+rNtMpNZNKr07E5+brpkUwjFGMszOTmMMiVpDlmR7OcYkrZURZUJaU5Vqr0lTnDOJHKNlPWkaXKw==</latexit>

M
X
1

M µ=1

µ a
µ b
(S⌅ ) (S⌅ )Qab,⌅ =

N
X
1

N i=1

<latexit sha1_base64="T8Fidg6OWBc1KEM16r4hefhxLZo=">AAADj3icjVHLahRBFL2d9hHHRya6Edw0DpEJxKEmSGKEyKAgCiqZTGYSmJo01W3NTJF+pbtmJClrpSt/wIUrBRfiZ4jgD7jIJ4jgJoIbF95+aAjBRzVdderWPefcqutEnkgkIXvGhHns+ImTk6dKp8+cPTdVnj7fScJR7PK2G3phvOGwhHsi4G0ppMc3opgz3/H4urN1Oz1fH/M4EWGwJnci3vPZIBB94TKJIbv8lfZj5qqr1OGSWTTE3FRK3akq6vStlrYV0XO/8X09qzcVjX1rLBKBJlqrhw/0cq5CPd6XNYtGLJaCebYK9IFiZxO3/6tKYzEYSusJSiyTwqN0SDe3aqW4S5WiQyatpq56s3M53k4x1T0rlyqU7HKF1JZIfWmhbh0F9RrJRgWKsRKWPwCFRxCCCyPwgUMAErEHDBL8ulAHAhHGeqAwFiMS2TkHDSXkjjCLYwbD6BbOA9x1i2iA+1Qzydguunj4x8i0YIZ8Im/JPvlI3pHP5McftVSmkdayg6uTc3lkTz2/2Pr+T5aPq4ThAeuvNUvow/WsVoG1R1kkvYWb88e7L/ZbN1Zn1BXymnzB+l+RPfIebxCMv7lvmnz1ZaYeIOdxdls/8w/wfRXGHXTVBfIK7wS20SV9V3zPrHl5h6yj4FfzOvO1+kJtvnmt0rhVtHESLsFlqGKvFqEBd2EF2uAaa8au8dR4Zk6bi+ZNs5GnThgF5wIcGua9n2mr8TA=</latexit>

Replicated free-energy

F (S0 , SL )NM<latexit sha1_base64="xXZO8rqL/ueQttC5ZXgPnZQHfrE=">AAAD+XichVHLbhMxFL1peJTh0RY2SGwsqlYJkMgTUKFIlSrYsGDRNKStVKcjz8RpRvU8OuNEKsY/wA+wYFUkJB5LtuzYsGTDop+AWBbBBgnuTBJVUUXxaOzrc+85PvZ1Y+mnitKDwkTx1OkzZyfPWecvXLw0NT1zeS2Neoknml4ko2TD5amQfiiayldSbMSJ4IErxbq78zDLr/dFkvpR+ETtxaIV8O3Q7/geVwg5MwV/njQcHZpNpjXrckV2TUmWmbk12tcH+5a1xLiMu3xLV2xDWNoLHC2XbLOlHxvS2NIsCYgbhW3j5KEIFWqigM4FWtbNMUqmMSSlsR8eI+UuWoSFUdgLXJEQxqx5wrppzD1xQ9/2AkMq4yYsgXqdhHvaNrpmBknumt18LsmKXa4Pw/IIK7OYJ8rn0tFdhDLEjEF5kUEsidpYkB8VGotJ0VFVgidWmCsUJ/3SiF82hLDE3+4q8uxIdYkaZ3qWVhepvbhgk+OBXaX5mIXhWImmPwKDNkTgQQ8CEBCCwlgChxS/TbCBQoxYCzRiCUZ+nhdgwEJuD6sEVnBEd3Dext3mEA1xn2mmOdvDUyT+CTIJzNGv9C09pJ/pe/qN/v6nls41Mi97uLoDroidqedXG7/+ywpwVdA9Yp3oWUEH7uVeffQe50h2C2/A7z99cdi4vzqn5+kr+h3979MD+glvEPZ/eK/rYvXlCX5idLGHmez10uzt8lYN+kGOB6NWrdWq9kK1Vr8zu/xg2LRJuAbXoYSduQvL8AhWoAle4UvhZ+HPBBR1cb/4pvhuUDpRGHKuwNgofvgLFvQZBw==</latexit>

Sn [{q̂(l)}, {Q̂(l)}] = ↵

N
X
1 i
µ
µ
µ
p J⌅,a S⌅(i),ar⌅,a = S⌅,aN
i=1<latexit sha1_base64="Q+kIGIpS6mrlAdct66lnao1pWXE=">AAAC/HichVHNahRBEK4Z/+L6k9VcBC+DSyRCXHqC5A8CQS/iQZKsmwQy2aGn7U2anb9096zGpn0AXyAHT4oiErz6Al58AQ95BPEYwYuCtbMrIazRGqa7+qv6qr7uivJYKE3IgeOeOn3m7LmR85ULFy9dHq1eubqqskIy3mRZnMn1iCoei5Q3tdAxX88lp0kU87Woc68XX+tyqUSWPtK7Od9M6FYq2oJRjVBY3ZMtEySFDU0QxZR11E5BJZ+kdsFrnBAJVJGERiz4tmUe2qAtKTO+NYHakRoBG2j+VJfKzJNtobk1D4ZqtIyw9q8NJsQtjHthtUbqc8Sfm/a9Ycevk9JqMLClrPoOAngMGTAoIAEOKWj0Y6Cg8NsAHwjkiG2CQUyiJ8o4BwsV5BaYxTGDItrBdQtPGwM0xXOvpirZDLvE+EtkejBOvpD35JB8JvvkK/l5Yi1T1uhp2cU96nN5Ho6+uNb48V9WgruG7SPWPzVraMNsqVWg9rxEerdgfX732d5hY35l3Nwkr8k31P+KHJBPeIO0+529XeYrL6FSDqD/yt6w82cAq1N1f7o+tXyntnh3MIoRuA43YALfewYW4T4sQRP7/nJqzqRz233uvnH33Q/9VNcZcMbgmLkffwP+LsiI</latexit>

1

L
X

visible@n V n (S0 , SL )
=

L
X
l=1

e

P
1
2

visible
n=0

NM

bond
Sent [Q̂(l)] +

l=1

a
b
J⌅(i) J⌅(i)

L
X1= @n Sn [{Q̂(l), q̂(l)}]spinSent [q̂(l)]M↵=N<latexit sha1_base64="rvGn3eI3HDgpZ8ede2+h9v9WMn0=">AAACeXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsIxSTmFGQk2sakFSUmV/vWVvvVxgsoG+hZWpiaGlsoGOgZGFgaWZoDGZaWlobmhgqGQBEQUGaAgoB8gRUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrLsCqCIfqDMfaEotgwKDqsFVg5UGnw1OGKw2eGnwB6dp1WBTQK6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF14XV3CkMZgAXZtJtD1BWARkD+SIfrLqqZ/DrYKUq1WM1hk8Bro/oUGNw0OA32QV/YleWlgatBsBi5wFEDCWQGTAYuCMCM9QzM9o0ATZQcnaGRwMEgzKDFoAEPcnMGBwYMhgCEUaG8FwzyG5QwrGH8zKTJpMGlBlDIxQvUIM6AAJmMAhqqSaQ==</latexit>

l=1

ab qab (l1)Qab (l)qab (l)@ha (l) @hb (l)

n=0

n
Y

a=1

e

v(ha (l))
ha (l)=0

Hajime Yoshino, SciPostPhys. Core 2, 005 (2020).

SciPost Phys. Core 2, 005 (2020)

Loop correctionz
Figure 2: A loop of interactions in a DNN extended over 3 layers, through 3 perceptrons and 4 bonds. We neglect effects of such loops (and more extended ones) in ourtheory.

Not global but dense coupling - loop corrections become negligible
with

N
i a
X
(J
µ
µ a
µ
Ñ)
a
rÑ,a ⌘ (SÑ )
(S
)
.
p
Ñ(i)
S0,1
N S1,1
i=1

S2,1

(15)

S3,1

c can beS0,2studied by considering
The typical behavior
the n ! 0 limit [36], i.e.
“connectivity”
S1,2<latexit sha1_base64="BubXqNOViuEiOOOYVJKEj5exHJo=">AAACcnichVHLSsNAFD2N7/qquhHdVEvFhciNCIor0Y1LtfYBbSlJnGowTWKSFjX0B/yBLtxYQVD8DDf+gAs/QVwquHHhbRoQLeodmDlz5p47Z+aqtqG7HtFTROrq7unt6x+IDg4Nj4zGxsYzrlV1NJHWLMNycqriCkM3RdrTPUPkbEcoFdUQWfVos3WerQnH1S1zzzu1RbGiHJh6WdcUj6miX1DL8VS95NOCXC/FErRIQcQ7gRyCBMLYtmI3KGAfFjRUUYGACY+xAQUujzxkEGzmivCZcxjpwblAHVHWVjlLcIbC7BHPB7zLh6zJ+1ZNN1CfcIbFSour1BFHkh7pll7pge7omT5+reYHVVpuTnlV21phl0bPJ1Pv/6oqvHo4/FL96dpDGauBW53d2wHTeofW1tfOGq+ptd2kP0dX9ML+m/RE9/wCs/amXe+I3QtEuQXyzw/vBJmlRZnxznJifSNsRj+mMYt5/vEVrGML20jzvcdo4BLNyJs0Jc1IYeekSKiZwLeQFj4B/C+O7g==</latexit><latexit sha1_base64="(null)">(null)</latexit>Nc@n V n (S0 , S L )1

S(0),S(L)<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

where the overline represents the average over the different realiza-

n=0

S0,3

tions of the boundaries (see below for the details.)
As shown in appendix A, following similar steps as in [20], we obtain the replicated freeentropy functional sn ({Q Ñ , qÑ }) in terms of the order parameters Q Ñ and qÑ defined in Eq. (13)S1,N0,N ↵ = M /N . For simplicity,in the limit N , M ! 1 with S
fixed
we limit ourselves to a treeapproximation which neglects the effects of interaction-loops along the z-axis such as the one
shown in Fig. 2. The tree approximation has two essential problems: 1) it cannot describe
faithfully 1-dimensional fluctuations along z-axis 2) it misses microscopic details close to theboundaries where we naturally expect inhomogeneities. Especially it fails to capture the difference of the two opposite boundaries. In principle, this accidental symmetry can be removed↵ = M/c<latexit sha1_base64="(null)">(null)</latexit>

Parisi's RSB ansatz

1<latexit sha1_base64="DX86leatul5UFSW1Q2u7LsfJOc0=">AAACjnichVE9S8NQFD2NX7V+VV0EF7FUnMqLiIogFl06ttWqUIsk8bU+TJOQvBZq8Q/oqjg4KTiIf8HNxT/g0J8gjgouDt6kAdGi3pC888695+a8d3XHFJ5krBVRurp7evui/bGBwaHhkfjo2JZn11yDFwzbtN0dXfO4KSxekEKafMdxuVbVTb6tH677+e06dz1hW5uy4fBSVatYoiwMTRKVU/fiCZZiQUx1AjUECYSRteP32MU+bBiooQoOC5KwCQ0ePUWoYHCIK6FJnEtIBHmOY8RIW6MqThUasYf0rdCuGLIW7f2eXqA26C8mvS4pp5BkT+yWvbJHdsee2cevvZpBD99Lg1a9reXO3sjJxMb7v6oqrRIHX6o/PUuUsRR4FeTdCRj/FEZbXz+6eN1YziebM+yavZD/K9ZiD3QCq/5m3OR4/vIPPw65aFDGvz3PvzsalfpzMJ1gay6lLqTmcvOJ9Fo4tCgmMY1Zmswi0sggiwL15DjFGc6VuLKgrCir7VIlEmrG8S2UzCd5rpIa</latexit><latexit sha1_base64="2dYqp042e1t4Scw10C0g74EXcJY=">AAAC6HicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtEBsZXJybVauRo2sYUl+bGV2faGtTGVWdrG9YCZTJBEhqeYCVx1SDZ2lpdZC5QWa2mQkxhYWliikKOraGOkU5MTkp+SbGOT7yAsoGepYGhpZmhAibDUM8ADJQZoCAgX2APQwxDCkM+QzJDKUMuQypDHkMJkJ3DkMhQDITRDIYMBgwFQLFYhmqgWBGQlQmWT2WoZeAC6i0FqkoFqkgEimYDyXQgLxoqmgfkg8wsButOBtqSA8RFQJ0KDKoGVw1WGnw2OGGw2uClwR+cZlWDzQC5pRJIJ0H0phbE83dJBH8nqCsXSJcwZCB04XVzCUMagwXYrZlAtxeARUC+SIboL6ua/jnYKki1Ws1gkcFroPsXGtw0OAz0QV7Zl+SlgalBs/G4pwDoikqgDCj0ikFhB44qSHwoYDJgURVmpGdopmcUaKLs4ASNNA4GaQYlBg1gzJgzODB4MAQwhALN3M/wlOEdw3umLKYJTNOZZkKUMjFC9QgzoACmRQD3c7W/</latexit>

Qab (l) =<latexit sha1_base64="7ysHn527Gj1DKcrYQ/O+sUrqJwQ=">AAAC6nichVFNT9RQFL1UEeRDRt2YuGmcQIY4TG4nBMVkEqIbSVzwNUAyA03beeDLvH5M3+skY9M/4NKNMaw0sjD+BTfGjX+ABb/AGDYQSNiw4LYzRgkBb9O++8655/a8d+1AcKkQ9/u0Gzf7bw0M3h4aHhm9M5a7e29V+lHosKrjCz9cty3JBPdYVXEl2HoQMsu1BVuzmy9Sfq3NQsl9b0V1ArbhWtse3+KOpQgyc7WWGVt2UhCTlbqMXDPmFUw24+ZjIyGGp0RhPivZjFM2Sab+3VJZMqnXW63IauiiYhTLxbpo+EoWX00ZZi6PpVk0ZmcM/XJilDCLPPRiwc99gzo0wAcHInCBgQeKcgEWSHpqYABCQNgGxISFlPGMZ5DAEGkjqmJUYRHapO827Wo91KN92lNmaof+IugNSanDOO7hFzzGn/gVf+PZlb3irEfqpUOr3dWywBx7+2D59L8ql1YFr/+qrvWsYAueZl45eQ8yJD2F09W337w/Xn62NB5P4Cc8IP8fcR9/0Am89omzu8iWdq7xE5CLDjHp7cn07rJRdeehX07+jGq1XDJmSuXF6fzc897QBuEhPIICTeYJzMFLWIAq9fwOv+AQjjShvdM+aDvdUq2vp7kPF0L7fA7tcrZx</latexit>

qab (l) =

k+1
X

i=0
k+1
X

mi
qi (l)(Iab

i=0

Input/output boundaries
replicated machines are subjected tothe same training data<latexit sha1_base64="RsbJR7Y5sh6pFTlHTSYZ76KZZWk=">AAACenichVHLSsNAFD2N7/qqiiC4KS2KRSg3Ir6gILpx4cJaq4KWksRRg2kSk7SgoT/gD7hwpSCifoYbf8BFP0FcKrhx4W0aESnqDZk5c+aeO2fmqrahux5RLSK1tLa1d3R2Rbt7evv6YwODm65VdjSR1yzDcrZVxRWGboq8p3uG2LYdoZRUQ2ypR8v1/a2KcFzdMje8E1sUSsqBqe/rmuIxVYwNHxd9Ra1OUCoTotVURi7GkpSeJ3l+Ro43AzlNQSQRxpoVu8Eu9mBBQxklCJjwGBtQ4PK3AxkEm7kCfOYcRnqwL1BFlLVlzhKcoTB7xOMBr3ZC1uR1vaYbqDU+xeDfYWUcY/REt/RKj3RPz/Txay0/qFH3csKz2tAKu9h/NpJ7/1dV4tnD4bfqT88e9jEXeNXZux0w9VtoDX3l9Pw1t7A+5o/TFb2w/0uq0QPfwKy8addZsX6BaNCAxivHm8FXAzan0vJMeio7nVxcClvRiVEkMMHvPYtFrGANeT7XxyXucB/5kBJSSppspEqRUDOEHyFNfwIcI5Gq</latexit>

qab (0) = qab (L) = 1

miQi (l)(Iab

mi+1
Iab )

l = 1, 2, . . . , L0x Q(x)<latexit sha1_base64="VKwTklyRFSWcSDHjsrPLraBQugE=">AAACjnichVHLSsNQED3G9zvqRnAjloqrMimiIoiiG5dW7QOqSBJva2heJGmhFn9At4oLVwouxF9w58YfcOEniMsKblw4SQOiok5I7rln5kzOvaO5puEHRE9tUntHZ1d3T29f/8Dg0LA8Mprznaqni6zumI5X0FRfmIYtsoERmKLgekK1NFPktcpamM/XhOcbjr0d1F2xa6ll2ygZuhowlaE9OUEpimLyJ1BikEAcG458hx3sw4GOKiwI2AgYm1Dh81OEAoLL3C4azHmMjCgvcIQ+1la5SnCFymyFv2XeFWPW5n3Y04/UOv/F5Ndj5SSS9Eg31KQHuqVnev+1VyPqEXqp86q1tMLdGz4e33r7V2XxGuDgU/Wn5wAlLEReDfbuRkx4Cr2lrx2eN7cWN5ONabqiF/Z/SU90zyewa6/6dUZsXvzhx2UXdc6Et+eHd8ejUr4P5ifIpVPKXCqdmU2srMZD68EEpjDDk5nHCtaxgSz3FDjBKc4kWZqTlqTlVqnUFmvG8CWk9Q93hJIZ</latexit>

mi+1
Iab )

l = 1, 2, . . . , L

1<latexit sha1_base64="PDEynTrDTzj0DHd/M5fJCZKLhgY=">AAACjnichVE9S8NQFD2NX/WzURfBpVgUp/JSpIogFl06ttWqoCJJfOqjaRKS12It/gFdFQcnBQfxL7i5+Acc/AniqODi4E0aEBX1huSdd+49N+e9a7iW8CVjjzGlrb2jsyve3dPb1z+QUAeHVnyn5pm8bDqW460Zus8tYfOyFNLia67H9aph8VWjshjkV+vc84VjL8uGyzer+q4tdoSpS6KK+1tqiqVZGMmfQItAClEUHPUWG9iGAxM1VMFhQxK2oMOnZx0aGFziNtEkziMkwjzHIXpIW6MqThU6sRX67tJuPWJt2gc9/VBt0l8sej1SJjHOHtg1e2H37IY9sfdfezXDHoGXBq1GS8vdrcTRyNLbv6oqrRJ7n6o/PUvsYCb0Ksi7GzLBKcyWvn5w9rI0WxpvTrBL9kz+L9gju6MT2PVX86rIS+d/+HHJRYMywe35wd3RqLTvg/kJVjJpLZvOFKdSuYVoaHGMYgyTNJlp5JBHAWXqyXGME5wqqpJV5pT5VqkSizTD+BJK/gMTY5Jh</latexit><latexit sha1_base64="Qo0ZJ0HaJgDnBR3E3PRAdJbiLpc=">AAACpnichVHNLgNRFD4df1U/LRYkNo2G1KY5Iw1qJWysRFGVIDUzbmvS6cxk5rZpTfoCXsDCCrEQvIGdjRew8AhiSWJj4cydSUQEZzJzv/ud85357j2qbeguR3yKSB2dXd090d5YX//AYDwxNLzlWnVHYwXNMixnW1VcZugmK3CdG2zbdphSUw1WVKvLfr7YYI6rW+Ymb9lsr6ZUTL2sawonqpQYzad3OWty0clTDUWrtr1me7qUSGEmh3JuVk7+BHIGRaQgjDUrcQe7cAAWaFCHGjAwgRM2QAGXnh2QAcEmbg884hxCusgzaEOMtHWqYlShEFulb4V2OyFr0t7v6Qq1Rn8x6HVImYRJfMQrfMUHvMZn/Pi1lyd6+F5atKqBltml+PHYxvu/qhqtHA6/VH965lCGeeFVJ++2YPxTaIG+cXTyurGwPulN4Tm+kP8zfMJ7OoHZeNMu82z99A8/NrloUca/Pde/u2BUIpIBmMuGIPc1qq2ZjDybyeazqcWlcGhRGIcJSNNk5mARVmANCsLhBdzArZSWVqWCVAxKpUioGYFvIe1/AjKrnR0=</latexit><latexit sha1_base64="OzTC5NT0tIW5aAA0sq2tPGQj8zA=">AAACpnichVE7TsNAEJ2Y/z9AARJNRBQUmmiMEL8qgoYKhYR8JEDBNptgxbGNvYkIVi7ABSioAFEg4AZ0NFyAIkdAlEGioWC8CUIIAWPZO/tm3vjtPtU2dJcjNgJSR2dXd09vX//A4NDwSHB0LONaFUdjac0yLCenKi4zdJOluc4NlrMdppRVg2XV0ppfz1aZ4+qWucVrNtstK0VTL+iawgnKBycOozucHXExyVMNRSvVvaP6bD4YxtgyyssLcuhnIsdQRBjakbCC97AD+2CBBhUoAwMTOOUGKODSsw0yINiE7YJHmEOZLuoM6tBP3Ap1MepQCC3Rt0i77TZq0t6f6Qq2Rn8x6HWIGYIIPuE1NvERb/AZ33+d5YkZvpYarWqLy+z8yMlk6u1fVplWDgdfrD81cyjAktCqk3ZbIP4ptBa/enzaTK0kI94MXuAL6T/HBj7QCczqq3a1yZJnf+ixSUWNKv7tuf7dCatafoR+Jp9WZeZi8kJsbnM+HF9tm9YLUzANUXJmEeKwDglIC4WXcAt3UlTakNJSttUqBdqccfgW0t4HOrucqA==</latexit>q(x)4.2 1st glass transition
1st Glass transitionspindx
P (q) =
dq
<latexit sha1_base64="WTPuyO9LjqLXH7ZYDkruR7cYqeM=">AAACoXichVHNSsNAEP6M//Wv6kXwUiwVvZSpin8giF701larQpWSpFsNpklM0mINfQFfwIMgKIiKr+DNiy/gwUcQjxW8eHCSVkREnWV3Z7+Zb/bbHcXSNcclemqSmlta29o7OkNd3T29feH+gQ3HLNmqyKimbtpbiuwIXTNExtVcXWxZtpCLii42lf1lP75ZFrajmca6W7HETlHeNbSCpsouQ7nwQHLsYHxhu2DLqpc/rHr5g2ouHKU4zc3S5Fzkp5OIU2BRNCxphu+wjTxMqCihCAEDLvs6ZDg8skiAYDG2A48xmz0tiAtUEWJuibMEZ8iM7vO6y6dsAzX47Nd0ArbKt+g8bWZGEKNHuqEaPdAtPdP7r7W8oIavpcK7UucKK9d3PLT29i+ryLuLvS/Wn5pdFDAbaNVYuxUg/ivUOr98dFJbm0/HvFG6oBfWf05PdM8vMMqv6mVKpE//0GOxigpH/N9z/L/jVn32I/K7szERT0zHJ1JT0cWlRtM6MIwRjHFnZrCIFSSR4ZqHOMMVrqWotColpXQ9VWpqcAbxzaTsB/h3mgM=</latexit>

bond

dx
P (Q) =
dQ
<latexit sha1_base64="OB9FtvNM9/5A96XI/T0K45bXtjs=">AAACoXichVHNSsNAEP6M//9VL4KXYlH0UqZV/ANB9KK3tloVqpQk3dZgmoQkLdbQF/AFPAiCgqj4Ct68+AIefATxqODFg5O0IiKts+zu7DfzzX67o1i65rhEzy1Sa1t7R2dXd09vX//AYGhoeNsxS7Yq0qqpm/auIjtC1wyRdjVXF7uWLeSioosd5XDNj++Uhe1oprHlViyxX5QLhpbXVNllKBsaTkwlp5f38rasermjqpdLVrOhCEVpcYFmFsN/nViUAougbgkzdI895GBCRQlFCBhw2dchw+GRQQwEi7F9eIzZ7GlBXKCKHuaWOEtwhszoIa8FPmXqqMFnv6YTsFW+RedpMzOMCXqiW3qjR7qjF/psWMsLavhaKrwrNa6wsoMno5sf/7KKvLs4+GE11ewij4VAq8barQDxX6HW+OXj07fNpdSEN0mX9Mr6L+iZHvgFRvldvUqK1FkTPRarqHDE/z3H/ztu1Xc/wo2d7Xg0NheNJ2cjK6v1pnVhDOOY4s7MYwXrSCDNNY9wjmvcSBFpQ0pIqVqq1FLnjOCXSZkva/eZww==</latexit>

bond
spin

continuous transition to full RSB glass phase
Figure 4: Glass order parameters close to the 1st glass transition: a) spatial profile of
↵
'
2.03
at
1
st
&
(L-1)
th
layer
g
the Edwards-Anderson (EA) order parameter for spins qEA (l) and bonds QEA (l) slightly
layers(filled
remainin the liquid
phase
before ↵ = 2.0 (empty symbols)/after ↵other= 3.125
symbols)
the 1stglass transition.
The depth is L = 10 in this example. b) Variation of the EA order parameters qEA (1) =qk (1) and QEA (1) = Qk (1) at the 1st layer after passing the critical point of the 1st<latexit sha1_base64="Uj5ZNLsPCbzxQewKFMPvBgp/e3I=">AAACfnichVFNL0NBFD2er6qvYkNsGg2xUbclIVYNG0tUkag07z2jJt6X916bVCP2/oCFFUkjIuFH2PgDFv0JYkliY+H29SWC4E5m5syZe+6cmdEcQ3o+Ub1FaW1r7+iMdEW7e3r7+mMDgxueXXJ1kdNtw3a3NNUThrREzpe+IbYcV6imZohN7WCpsb9ZFq4nbWvdrzhix1SLltyTuuozVYiN5FXD2VcL1bxrxovHeU+a4jCeTtJMIZagJAUR/wlSIUggjBU7doU8dmFDRwkmBCz4jA2o8LhtIwWCw9wOqsy5jGSwL3CMKGtLnCU4Q2X2gMcir7ZD1uJ1o6YXqHU+xeDusjKOcXqka3qhB7qhJ3r/tVY1qNHwUuFZa2qFU+g/Hc6+/asyefax/6n607OPPcwHXiV7dwKmcQu9qS8fnb1kF9bGqxN0Sc/s/4LqdM83sMqvem1VrJ0jyh+Q+v7cP8FGOpmaSaZXZxOZxfArIhjFGCb5veeQwTJWkONzT1DDLe4UKBPKlDLdTFVaQs0QvoQy/wETcJJ/</latexit>

4.3

2nd glasstransition

2nd Glass transition

spin

bond

↵g (2) ' 15.38

continuous transition to full RSB glass phaseat 2nd& (L-2)layer transition: a) spatial profile of
Figure 5: Glass order parameters close
to the2ndth glass
the Edwards-Anderson (EA) orderparameter
forglass
spins
qEA (l) andQEA(l) slightly
which also
induce 2ndtransitions
at 1stbonds
and L-thlayer<latexit sha1_base64="64V74Zhkt2+ro2udTFp+qOwDKes=">AAAChHichVFNLwNBGH6sr6qv4iJx0GgIiTSzrYY4iHBxbEuRqDS7a9TEftndNmHTo4s/4OBEIiK98gtc/AEHP0EcSVwcvN1uIgjeycw888z7vPPMjGrrwvUYe2yRWtvaOzojXdHunt6+/tjA4LprVRyNFzRLt5xNVXG5Lkxe8ISn803b4Yqh6nxD3V9u7G9UueMKy1zzDm2+bShlU+wKTfGIKsVGi4pu7ykl3y86Rrxcq02mpoquMPhBXM4k03OlWIIlWRDxn0AOQQJhZK3YFYrYgQUNFRjgMOER1qHApbYFGQw2cdvwiXMIiWCfo4YoaSuUxSlDIXafxjKttkLWpHWjphuoNTpFp+6QMo5x9sCu2Qu7Z3X2xN5/reUHNRpeDmlWm1pul/pPhlff/lUZNHvY+1T96dnDLuYCr4K82wHTuIXW1FePTl9W5/Pj/gS7YM/k/5w9sju6gVl91S5zPH+GKH2A/P25f4L1VFJOJ1O5mcTiUvgVEYxgDJP03rNYxAqyKNC5x6jjBrdShzQtpaVMM1VqCTVD+BLSwgeefZRy</latexit>

Growth of glass phase with increasing training data
spin

bond

1

1

qEA 0.8

0.8

QEA<latexit sha1_base64="RPD8LwSR4z5gJHAWTCHdiCZKVOM=">AAACb3ichVHLSsNAFD2Nr1pfVRcKghRLRTdlWgXFVVUEl221ttiWksRRg3mRpMVa+gOuBReioCAofoYbf8CFnyCuRMGNC2/SgKiod2DmzLn33DkzI5mqYjuMPQSEtvaOzq5gd6int69/IDw4tGEbVUvmOdlQDasgiTZXFZ3nHMVRecG0uKhJKs9Le8tuPl/jlq0Y+rpTN3lZE3d0ZVuRRYeozUylUbK0yMpisxKOsjjzIvITJHwQhR9pI3yFErZgQEYVGjh0OIRViLBpFJEAg0lcGQ3iLEKKl+doIkTaKlVxqhCJ3aN5h3ZFn9Vp7/a0PfU+VRikNKhLExHE2D27Zi/sjt2wR/b+a7eG18V1U6dVamm5WRk4HF17+1el0epg91P1p2sH25j33Crk3vQY9x5yS187OH5ZW8jGGpPsgj2R/3P2wG7pBnrtVb7M8OwJQvQFie8P/hNsJOOJmXgyMxtNLfmfEcQYJjBFLz6HFFaRRo7O1XGEU5wFnoURYVyItEqFgK8ZxpcQpj8Aq2GN6g==</latexit>

0.6

0.6

“Wetting transition”! 0.4

0.4

0.2

0.2

layer-by-layer 2nd order transition0

0

0

5

10

15

2

20

l

4

6

8 10 12 14 16 18 20lSciPost Phys. Core 2, 005 (2020)↵ = 50, 100, 200, 1000, 2000, 4000
<latexit sha1_base64="2dFrtClhBujE7RruPfa1atU1G9U=">AAAChXichVHLSsNAFD3GR2t9Vd0IbopFcVHLTalVBFF049JXtdBKSeKooWkSkrSgxa0Lf8CFKwUR0a3+gBt/wIWfIC4V3LjwNg2IFvWGzD1z5p47Z2ZU29Bdj+ipRWpta+8IhTsjXd09vX3R/oEN16o4mshqlmE5OVVxhaGbIuvpniFytiOUsmqITbW0WF/frArH1S1z3du3xVZZ2TX1HV1TPKaK0VhBMew9ZXaSEjJRIkV+9gEl0jwUo3FKkh+xZiAHII4glq3oJQrYhgUNFZQhYMJjbECBy18eMgg2c1uoMecw0v11gUNEWFvhKsEVCrMlHnd5lg9Yk+f1nq6v1ngXg3+HlTGM0iNd0Ss90DU908evvWp+j7qXfc5qQyvsYt/x0Nr7v6oyZw97X6o/PXvYwbTvVWfvts/UT6E19NWDk9e1mdXR2hid0wv7P6MnuucTmNU37WJFrJ4iwg8g/7zuZrCRSsqZZGolHZ9fCJ4ijGGMYJzvewrzWMIysrzvEW5wizspJE1IaSnTKJVaAs0gvoU09wkLxZE5</latexit>

10000“penetration depth”

1000

g

↵g (l) ⇠ 2.7(3)e100

1.03(2)l<latexit sha1_base64="13Jt2AvwoBcVr58dGy40xBWyYrw=">AAAChHichVHLLgRBFD3ae7wGG4mFjgkhkUmNR4iFCBtLr0GiZdLdChU13ZXumgkmljZ+wMKKRERs+QIbP2DhE8SSxMbCnZ5OBMGtVNWpU/fcOlXlKClCzdhjlVFdU1tX39CYaGpuaW1LtneshH4hcHnW9aUfrDl2yKXweFYLLfmaCriddyRfdXZny/urRR6EwveW9b7iG3l72xNbwrU1Ublkj7UnBixbqh170LRU4Cvtm5b0zApn5pIplmZRmD9BJgYpxDHvJy9hYRM+XBSQB4cHTVjCRkhtHRkwKOI2UCIuICSifY5DJEhboCxOGTaxuzRu02o9Zj1al2uGkdqlUyT1gJQm+tgDu2Iv7J5dsyf2/mutUlSj7GWfZqei5SrXdty19PavKk+zxs6n6k/PGluYiLwK8q4ipnwLt6IvHpy8LE0u9pX62Tl7Jv9n7JHd0Q284qt7scAXT5GgD8h8f+6fYGU4nRlJDy+MpqZn4q9oQDd6MUDvPY5pzGEeWTr3CNe4wa1RZwwZI8ZYJdWoijWd+BLG1AeHR5Tq</latexit>⇠(↵) / ln ↵

10

1
0

1

2

3

4

l

5

6

7

this suggests storage capacity also grows fast with the depth
constl↵J (l) / e

SciPost Physics

Submission

Space-dependent replica-symmetry breaking↵ = 4000<latexit sha1_base64="ghbOttJ0+kI+w09oJPANmemhYCs=">AAACbnichVHLSsNAFD2Nr1ofrQoiiFgsla7KbRUUQSi6canVPqCKJHG0oWkSkrSgxR9wLy4ERUFE/Aw3/oCLfoK4ERTcuPA2DYiKesNkzpy5586ZO4qla45L1AxIHZ1d3T3B3lBf/8BgODI0nHfMmq2KnGrqpl1UZEfomiFyrubqomjZQq4quigoleXWfqEubEczjQ133xJbVXnP0HY1VXaZKm3KulWWF2eJaDsSoyR5Ef0JUj6IwY9VM3KNTezAhIoaqhAw4DLWIcPhr4QUCBZzW2gwZzPSvH2BQ4RYW+MswRkysxX+7/Gq5LMGr1s1HU+t8ik6D5uVUcTpgW7ohe7plh7p/ddaDa9Gy8s+z0pbK6zt8NHY+tu/qirPLsqfqj89u9jFvOdVY++Wx7Ruobb19YOTl/WFbLwxTZf0xP4vqEl3fAOj/qperYnsKUL8AKnv7f4J8ulkaiaZXpuNZZb8pwhiHFNIcL/nkMEKVpHzOnaMM5wHnqVRaUKabKdKAV8zgi8hJT4AWBKNSw==</latexit>

spin

dx
P (q) =
dq
<latexit sha1_base64="WTPuyO9LjqLXH7ZYDkruR7cYqeM=">AAACoXichVHNSsNAEP6M//Wv6kXwUiwVvZSpin8giF701larQpWSpFsNpklM0mINfQFfwIMgKIiKr+DNiy/gwUcQjxW8eHCSVkREnWV3Z7+Zb/bbHcXSNcclemqSmlta29o7OkNd3T29feH+gQ3HLNmqyKimbtpbiuwIXTNExtVcXWxZtpCLii42lf1lP75ZFrajmca6W7HETlHeNbSCpsouQ7nwQHLsYHxhu2DLqpc/rHr5g2ouHKU4zc3S5Fzkp5OIU2BRNCxphu+wjTxMqCihCAEDLvs6ZDg8skiAYDG2A48xmz0tiAtUEWJuibMEZ8iM7vO6y6dsAzX47Nd0ArbKt+g8bWZGEKNHuqEaPdAtPdP7r7W8oIavpcK7UucKK9d3PLT29i+ryLuLvS/Wn5pdFDAbaNVYuxUg/ivUOr98dFJbm0/HvFG6oBfWf05PdM8vMMqv6mVKpE//0GOxigpH/N9z/L/jVn32I/K7szERT0zHJ1JT0cWlRtM6MIwRjHFnZrCIFSSR4ZqHOMMVrqWotColpXQ9VWpqcAbxzaTsB/h3mgM=</latexit>

bond

dx
P (Q) =
dQ
<latexit sha1_base64="OB9FtvNM9/5A96XI/T0K45bXtjs=">AAACoXichVHNSsNAEP6M//9VL4KXYlH0UqZV/ANB9KK3tloVqpQk3dZgmoQkLdbQF/AFPAiCgqj4Ct68+AIefATxqODFg5O0IiKts+zu7DfzzX67o1i65rhEzy1Sa1t7R2dXd09vX//AYGhoeNsxS7Yq0qqpm/auIjtC1wyRdjVXF7uWLeSioosd5XDNj++Uhe1oprHlViyxX5QLhpbXVNllKBsaTkwlp5f38rasermjqpdLVrOhCEVpcYFmFsN/nViUAougbgkzdI895GBCRQlFCBhw2dchw+GRQQwEi7F9eIzZ7GlBXKCKHuaWOEtwhszoIa8FPmXqqMFnv6YTsFW+RedpMzOMCXqiW3qjR7qjF/psWMsLavhaKrwrNa6wsoMno5sf/7KKvLs4+GE11ewij4VAq8barQDxX6HW+OXj07fNpdSEN0mX9Mr6L+iZHvgFRvldvUqK1FkTPRarqHDE/z3H/ztu1Xc/wo2d7Xg0NheNJ2cjK6v1pnVhDOOY4s7MYwXrSCDNNY9wjmvcSBFpQ0pIqVqq1FLnjOCXSZkva/eZww==</latexit>

SciPost Phys. Core 2, 005 (2020)

Figure 15: Schematic picture of the design space of deep neural network based on
the present theory

each DNN is NOT a glass

Scenario (2) Teacher student setting- a statistical inference problem

Randomly quenched

1) Training

random
test
data

noise

Out put of teacher

teacher
Jij

student
Jij

noise

Student is forced toreproduceteacherʼs output

Randomly quenched
2) Testrandomtest
data

teacherJij
Now quenchedstudent
Jij

CompareReplicated Gardner volume<latexit sha1_base64="HpAM+IzvTKaMjs1wp23fhTROu88=">AAAD1HichZG/b9NAFMdfkhaK+dEWFiQWi6hRKkJ0yVAoUqUKFoQY2qZJK9VNdHYvqVX77J7PKeG4CbEwsDIwgYQQ4s9g4R/o0D8BMRaJAQae7ZBSCuUs3717730/9+6dHXpuJAk5yOULY+Nnzk6cM85fuHhpcmr6cisKYuGwphN4gVi3acQ8l7OmdKXH1kPBqG97bM3euZfE1/pMRG7AV+UgZJs+7XG36zpUoqszneMWZ3tO4PuUbynLjjROHnV2ot2YCqYNy2Y9lyu2y6kQdKCNVltZkj2S6dlqwDwv2NOqdoNrbXmsK8sI6JoN3VFEV0b2Q20Jt7ctZ43SQskKRbDVUXThLyCidVtxPDdlDTOTukxlCd9cFQhLqQ8wj+osNoJnKvNIZtnJXTwabWf6IJZhLLX+A9b4HWaaI9yQ48cVjFSoNlhb3cSWSGr2y6JzrFcYbiepelabFg947NtMmIbFsLGj9nWmiqQ6T2rzczXzpFGrknQUYTiWgql3YMEWBOBADD4w4CDR9oBChN8G1IBAiL5NUOgTaLlpnIEGA7UxZjHMoOjdwbmHu42hl+M+YUap2sFTPPwFKk2YIfvkPTkkn8gH8pn8+CdLpYyklgGudqZlYWfy+dXGt/+qfFwlbB+pTq1ZQhdup7W6WHuYepJbOJm+//jlYePOyowqkTfkC9b/mhyQj3gD3v/qvF1mK6/ASB8g67J50vj1AK16tTZXrS/Xi4t3h08xAdfgOpSx37dgEe7DEjTBye3nvufH8uOFVuFJ4WnhWZaazw01V+DYKLz4CZz5GOw=</latexit>

V

1+n(S0 , SL )

n
Y

=

Y
⌅

a=0
teacher-machine
Order parameters

a=0<latexit sha1_base64="kpC3jLtG17/nxwei73ymQvW3yEA=">AAACkHichVHLSsNQED3GV3226kZwUywVV2VSRK0gvjbiylr7ABVJ4m0NTZOQpIVa/AHBrS5cKbgQf8GdG3/ART9BXCq4ceEkrYiIOiG5c8/MmZx7j2obuusRNTukzq7unt5QX//A4NBwODIymnOtqqOJrGYZllNQFVcYuimynu4ZomA7Qqmohsir5TW/nq8Jx9Utc9ur22KvopRMvahrisdQRlmk/UiMEimSU7Ny9GciJyiIGNqxaUXusIsDWNBQRQUCJjzODShw+dmBDILN2B4ajDmc6UFd4Bj9zK1yl+AOhdEyf0u822mjJu/9mW7A1vgvBr8OM6OI0yPd0As90C090fuvsxrBDF9LnVe1xRX2fvhkPPP2L6vCq4fDL9afmj0UMR9o1Vm7HSD+KbQWv3Z0/pJZ2Io3puiKnln/JTXpnk9g1l6167TYuvhDj80q6lzxb8/17y6wquVH9GfyaVUumZBnE8n0TGx5tW1aCBOYxDQ7M4dlrGMTWZ5ZwinOcC6NSvPSkrTSapU62pwxfAtp4wO72ZMW</latexit><latexit sha1_base64="GTxtaHLJa7YedtM7CfyEvvH72cc=">AAADNXichVHNahRBEK4Z/+L6k425CF4Gl8iuyNITQjSBQNCLCAnZrJsEMtmhe9Ibm8zfTs+sxKFfwLPgwZOCB8kD+AAi+AIecvAmgnjwEEEQD9b0jqis0R6mu+rr+r6qrmKxL2RKyIFhHjt+4uSpsdOVM2fPnR+vTlxYk1GWeLzjRX6UbDAquS9C3klF6vONOOE0YD5fZ7u3ivv1AU+kiMK76V7MtwK6E4qe8GiKkFt91Hdzyq45TKoFp5dQL7dVvqQcmQVu7gTZgq266Nfb6GFMt8BUo5vTUYgpy+n3M7pdaf1Fc7nUFFpxWd3R7LpoqELsd48pt1ojzTliz83a1qhhN4leNSjXSlR9DQ5sQwQeZBAAhxBStH2gIPHbBBsIxIhtQY5YgpbQ9xwUVJCbYRTHCIroLu476G2WaIh+oSk128MsPv4JMi2YIm/JC3JI3pB98pF8P1Ir1xpFLXt4siGXx+74w4vtr/9lBXimcO8X6581p9CDG7pWgbXHGile4Q35gwePD9vzq1P5FfKMfML6n5ID8gpfEA6+eM9bfPWJVg+Rc1+/NtD5Q+xvjjjDrKq0/DK3hD5mKfqK/dTDG07IGjV+Dm9tumnPNqdbM7XFm+UYx+ASXIY6zuo6LMJtWIEOqn8zLKNhXDVfmu/M9+aHYahplJxJ+GOZn38A1eXSiA==</latexit>qab,⌅ =

TrJa⌅

!0
@

Y

⌅\output

student-machines

M
X
1

M µ=1

1

TrSa⌅ AY

<latexit sha1_base64="4D54sKxWyCGJfM4gDeVzrB07T3Q=">AAAED3ichVFNaxNBGH6T+FFjta2iCF4GS0tiTZitUq1QKHrx4KFp7Ad00mV2O2mWzn50dxKN4+DdmycPnhQ8iD/Ai4jgRX+Ah/4BQTxW8OLBdzcJJRTrLDvzzvO+zzPPzOtE0ksUpXu5fOHY8RMnR04VT4+eOTs2PnFuNQnbsStW3FCG8brDEyG9QKwoT0mxHsWC+44Ua87O3TS/1hFx4oXBA9WNRMPn24HX9FyuELInck+nSd3WTIlHKlPTXSFl+NBoy8wEZoNpzVpckV1TkmVmrg32td6+UVxgXEYtvqkrliEsafu2lguW2dT3Dalvahb7xAmDLWNnoQgUaqKAzgQaxZkhSqrRJyWRFxwiZS4ahAVh0PYdERPGitOEtZKIu+Kqvu76hlSGTRQF6jVj7uKF9KzpJbljdrO5JCtWudYPywOszCIeK49LW7cQShEzBGVFBrE43MKCBYpHBabIpGiqKsETK8wRipNOacAvG0JY7G23FHlyoIpEe3ySVuepNT9nkcOBVaXZmIT+WArHPwCDLQjBhTb4ICAAhbEEDgl+G2ABhQixBmjEYoy8LC/AQBG5bawSWMER3cF5G3cbfTTAfaqZZGwXT5H4x8gkMEW/0bd0n36h7+gP+uefWjrTSL10cXV6XBHZY88u1X//l+XjqqB1wDrSs4Im3Mq8eug9ypD0Fm6P33n8Yr9+e3lKT9PX9Cf6f0X36Ge8QdD55b6pieWXR/iJ0EUXM+nrJenbZa3q9YMcDgatWp2tWnPV2dqNycU7/aaNwGW4AiXszE1YhHuwBCvg5r7nR/MX8hcLzwvvCx8Ln3ql+Vyfcx6GRuHrXzRgIQw=</latexit>

S1+n [{q̂(l)}, {Q̂(l)}] = ↵
L
X
l=1

1

L
X

visibleQab,⌅ =

N
X
1

N i=1

e

P
1
2

@n V 1+n (S0 , SL )
=

bond
Sent
[Q̂(l)] +l=1

ea = 1, 2, . . . , n<latexit sha1_base64="CR7z0kBBXxEO63X7ypvkC8hVLl8=">AAACm3ichVHLSsNQED3GV31X3QgiFEvFRSmTIvUBguhGxIWvaqE+SNKrBtMkJLeFWvwBf0DBlYIL8RfcufEHXPgJ4lLBjQsnaUWkqBOSO/fMnMm59+iuZfqS6KlJaW5pbWuPdHR2dff09kX7BzZ9p+QZIms4luPldM0XlmmLrDSlJXKuJ7Sibokt/WghqG+Vheebjr0hK67YKWoHtrlvGppkaFebVZPp5LZVcKSftPeicUpNkzqdUWONiZqiMOKox4oTvcM2CnBgoIQiBGxIzi1o8PnJQwXBZWwHVcY8zsywLnCCTuaWuEtwh8boEX8PeJevozbvg5l+yDb4Lxa/HjNjSNAj3dArPdAtPdPHr7Oq4YxAS4VXvcYV7l7f6dD6+7+sIq8Sh9+sPzVL7GMq1GqydjdEglMYNX75+Ox1fWYtUR2jK3ph/Zf0RPd8Arv8ZlyvirWLP/S4rKLCleD2/ODuQqtqfsQaky+rNtMpNZNKr07E5+brpkUwjFGMszOTmMMiVpDlmR7OcYkrZURZUJaU5Vqr0lTnDOJHKNlPWkaXKw==</latexit>µ aµ b(S⌅ ) (S⌅ )F (S0 , SL )NMµ
v(r⌅,a)µ,⌅,a<latexit sha1_base64="MKQ3nZXU7mVIMHgdnBYhPvMuAII=">AAADvXicjVHLahRBFL2d9hHHRya6EbJpHCITjEN1kGiEwaAgLlQyGWcSmJo03W3NTJPqR7prJo5lbVz6Ay5cKQiKK/0FEfwBF/kEcRnBjQtvPzSEELWarjp1655zbtV1Iu4lgpAdbUI/cvTY8ckTpZOnTp+ZKk+fbSfhMHZZyw15GK87dsK4F7CW8ARn61HMbN/hbM3ZvJWer41YnHhh8ECMI9b17X7g9TzXFhiyprUZ2ottV16mDhO2QUNMTrXk7aqkTs9oKksSNf8H31VzakPS2DdGXuKhi1Ly/j1VqucylLOeqBk0smPh2dySgdqTbCNRsEciq1qOGefhtpKmuhSo/zWjsdcfCOMJCtdJbl0v7XPLC2hah1p1qJR0YAujoap8bj7HWymmqmvkBoW+Va6Q2hIxlxZN4yAwayQbFSjGSlj+BBQeQgguDMEHBgEIxBxsSPDrgAkEIox1QWIsRuRl5wwUlJA7xCyGGTZGN3Hu465TRAPcp5pJxnbRheMfI9OAWfKFvCW75DN5R76Sn4dqyUwjrWWMq5NzWWRNPTvf/PFPlo+rgMEe6681C+jBtaxWD2uPskh6Czfnjx4/321eX52VF8kr8g3rf0l2yEe8QTD67r5usNUXmXqAnO3stn7mH+D7Sow76KoKxAvvBLbQJX1XfM+seXmHjIPgd/PaCzVzsbbQuFJZvlm0cRJm4AJUsVdXYRnuwAq0wNWeam+099oH/YbOdK4HeeqEVnDOwb6hb/8CGugEZg==</latexit>

Replicated free-energy(Franz-Parisi potential)

N
X
1 i
µ
µ
µ
p J⌅,a S⌅(i),ar⌅,a = S⌅,aN
i=1<latexit sha1_base64="Q+kIGIpS6mrlAdct66lnao1pWXE=">AAAC/HichVHNahRBEK4Z/+L6k9VcBC+DSyRCXHqC5A8CQS/iQZKsmwQy2aGn7U2anb9096zGpn0AXyAHT4oiErz6Al58AQ95BPEYwYuCtbMrIazRGqa7+qv6qr7uivJYKE3IgeOeOn3m7LmR85ULFy9dHq1eubqqskIy3mRZnMn1iCoei5Q3tdAxX88lp0kU87Woc68XX+tyqUSWPtK7Od9M6FYq2oJRjVBY3ZMtEySFDU0QxZR11E5BJZ+kdsFrnBAJVJGERiz4tmUe2qAtKTO+NYHakRoBG2j+VJfKzJNtobk1D4ZqtIyw9q8NJsQtjHthtUbqc8Sfm/a9Ycevk9JqMLClrPoOAngMGTAoIAEOKWj0Y6Cg8NsAHwjkiG2CQUyiJ8o4BwsV5BaYxTGDItrBdQtPGwM0xXOvpirZDLvE+EtkejBOvpD35JB8JvvkK/l5Yi1T1uhp2cU96nN5Ho6+uNb48V9WgruG7SPWPzVraMNsqVWg9rxEerdgfX732d5hY35l3Nwkr8k31P+KHJBPeIO0+529XeYrL6FSDqD/yt6w82cAq1N1f7o+tXyntnh3MIoRuA43YALfewYW4T4sQRP7/nJqzqRz233uvnH33Q/9VNcZcMbgmLkffwP+LsiI</latexit>abJ⌅(i) J⌅(i)

visible
n=0

NM
L
X1

= @n S1+n [{Q̂(l), q̂(l)}]spinSent [q̂(l)]M↵=N<latexit sha1_base64="rvGn3eI3HDgpZ8ede2+h9v9WMn0=">AAACeXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsIxSTmFGQk2sakFSUmV/vWVvvVxgsoG+hZWpiaGlsoGOgZGFgaWZoDGZaWlobmhgqGQBEQUGaAgoB8gRUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrLsCqCIfqDMfaEotgwKDqsFVg5UGnw1OGKw2eGnwB6dp1WBTQK6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF14XV3CkMZgAXZtJtD1BWARkD+SIfrLqqZ/DrYKUq1WM1hk8Bro/oUGNw0OA32QV/YleWlgatBsBi5wFEDCWQGTAYuCMCM9QzM9o0ATZQcnaGRwMEgzKDFoAEPcnMGBwYMhgCEUaG8FwzyG5QwrGH8zKTJpMGlBlDIxQvUIM6AAJmMAhqqSaQ==</latexit>

l=1

ab qab (l1)Qab (l)qab (l)@ha (l) @hb (l)

n=0

n
Y

a=0

e

v(ha (l))
ha (l)=0

Hajime Yoshino, SciPostPhys. Core 2, 005 (2020).

Parisi's RSB ansatz

k+1
X<latexit sha1_base64="2dYqp042e1t4Scw10C0g74EXcJY=">AAAC6HicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtEBsZXJybVauRo2sYUl+bGV2faGtTGVWdrG9YCZTJBEhqeYCVx1SDZ2lpdZC5QWa2mQkxhYWliikKOraGOkU5MTkp+SbGOT7yAsoGepYGhpZmhAibDUM8ADJQZoCAgX2APQwxDCkM+QzJDKUMuQypDHkMJkJ3DkMhQDITRDIYMBgwFQLFYhmqgWBGQlQmWT2WoZeAC6i0FqkoFqkgEimYDyXQgLxoqmgfkg8wsButOBtqSA8RFQJ0KDKoGVw1WGnw2OGGw2uClwR+cZlWDzQC5pRJIJ0H0phbE83dJBH8nqCsXSJcwZCB04XVzCUMagwXYrZlAtxeARUC+SIboL6ua/jnYKki1Ws1gkcFroPsXGtw0OAz0QV7Zl+SlgalBs/G4pwDoikqgDCj0ikFhB44qSHwoYDJgURVmpGdopmcUaKLs4ASNNA4GaQYlBg1gzJgzODB4MAQwhALN3M/wlOEdw3umLKYJTNOZZkKUMjFC9QgzoACmRQD3c7W/</latexit>

Qab (l) =
k+1
i=0
X
mi
qab (l) =qi (l)(Iab<latexit sha1_base64="7ysHn527Gj1DKcrYQ/O+sUrqJwQ=">AAAC6nichVFNT9RQFL1UEeRDRt2YuGmcQIY4TG4nBMVkEqIbSVzwNUAyA03beeDLvH5M3+skY9M/4NKNMaw0sjD+BTfGjX+ABb/AGDYQSNiw4LYzRgkBb9O++8655/a8d+1AcKkQ9/u0Gzf7bw0M3h4aHhm9M5a7e29V+lHosKrjCz9cty3JBPdYVXEl2HoQMsu1BVuzmy9Sfq3NQsl9b0V1ArbhWtse3+KOpQgyc7WWGVt2UhCTlbqMXDPmFUw24+ZjIyGGp0RhPivZjFM2Sab+3VJZMqnXW63IauiiYhTLxbpo+EoWX00ZZi6PpVk0ZmcM/XJilDCLPPRiwc99gzo0wAcHInCBgQeKcgEWSHpqYABCQNgGxISFlPGMZ5DAEGkjqmJUYRHapO827Wo91KN92lNmaof+IugNSanDOO7hFzzGn/gVf+PZlb3irEfqpUOr3dWywBx7+2D59L8ql1YFr/+qrvWsYAueZl45eQ8yJD2F09W337w/Xn62NB5P4Cc8IP8fcR9/0Am89omzu8iWdq7xE5CLDjHp7cn07rJRdeehX07+jGq1XDJmSuXF6fzc897QBuEhPIICTeYJzMFLWIAq9fwOv+AQjjShvdM+aDvdUq2vp7kPF0L7fA7tcrZx</latexit>

Input/output boundaries

mi+1
Iab )

mi
Qi (l)(Iab

mi+1
Iab )

l = 1, 2, . . . , L
l = 1, 2, . . . , L

1

i=0

Bayes-optimal limit
overlap among students
at the boundaries

overlap between the students
and the teacher

0<latexit sha1_base64="c85Rhcf/vL7n+qeqq+3B6uWj7Vg=">AAACjnichVE9S8NQFD3G7/rRqovgUiwVp3IjUq0gii6O1toPUJEkvtZgmoQkLdTiH9BVcXBScBD/gpuLf8ChP0EcK7g4eJNWRMR6Q/LuO/eem/PeUW1Ddz2iRpfU3dPb1z8wGBoaHhkNR8bGc65VcTSR1SzDcgqq4gpDN0XW0z1DFGxHKGXVEHn1aN2v56vCcXXL3PZqttgrKyVTL+qa4jGUpv1IjBIpklNJOfo7kRMURAzt2LQiD9jFASxoqKAMARMe5wYUuPzsQAbBZmwPdcYczvSgLnCCEHMr3CW4Q2H0iL8l3u20UZP3/kw3YGv8F4Nfh5lRxOmZ7qhJT3RPL/Tx56x6MMPXUuNVbXGFvR8+ncy8/8sq8+rh8JvVUbOHIhYDrTprtwPEP4XW4lePL5uZpa14fYZu6JX1X1ODHvkEZvVNu02LrasOemxWUeOKf3uuf3eBVS0/or+TL6tycwk5mZhLz8dW19qmDWAK05hlZxawig1sIsszBc5wjgspIiWlZWml1Sp1tTkT+BHSxicZyZJk</latexit>

1<latexit sha1_base64="6K9B7q8+yy45EjrLUbI/66LHWzk=">AAACjnichVE9S8NQFD3G7/rRqovgUiwVp3IjUq0gii6O1toPUJEkvtZgmoQkLdTiH9BVcXBScBD/gpuLf8ChP0EcK7g4eJNWRMR6Q/LuO/eem/PeUW1Ddz2iRpfU3dPb1z8wGBoaHhkNR8bGc65VcTSR1SzDcgqq4gpDN0XW0z1DFGxHKGXVEHn1aN2v56vCcXXL3PZqttgrKyVTL+qa4jGUlvcjMUqkSE4l5ejvRE5QEDG0Y9OKPGAXB7CgoYIyBEx4nBtQ4PKzAxkEm7E91BlzONODusAJQsytcJfgDoXRI/6WeLfTRk3e+zPdgK3xXwx+HWZGEadnuqMmPdE9vdDHn7PqwQxfS41XtcUV9n74dDLz/i+rzKuHw29WR80eilgMtOqs3Q4Q/xRai189vmxmlrbi9Rm6oVfWf00NeuQTmNU37TYttq466LFZRY0r/u25/t0FVrX8iP5OvqzKzSXkZGIuPR9bXWubNoApTGOWnVnAKjawiSzPFDjDOS6kiJSUlqWVVqvU1eZM4EdIG58b85Jl</latexit>r<latexit sha1_base64="Um8mYpLa3Y6FWt0b6TcTX8aMvLY=">AAACjnichVHLSsNQED2Nr/quuhHcFIviqkyK1AeIRTddttWqoCJJvNbQNAlJWqjFH9Ct4sKVggvxF9y58Qdc+AniUsGNCycPERF1QnLnnpkzOfce1TZ01yN6jElt7R2dXfHunt6+/oHBxNDwmmvVHU2UNcuwnA1VcYWhm6Ls6Z4hNmxHKDXVEOtqddmvrzeE4+qWueo1bbFdUyqmvqdrisdQ0dlJpCg9R/JcVk7+TOQ0BZFCFAUrcYst7MKChjpqEDDhcW5AgcvPJmQQbMa20WLM4UwP6gKH6GFunbsEdyiMVvlb4d1mhJq892e6AVvjvxj8OsxMYoIe6Jpe6J5u6Inef53VCmb4Wpq8qiFX2DuDR6Mrb/+yarx62P9i/anZwx5mA606a7cDxD+FFvIbB2cvK/OlidYkXdIz67+gR7rjE5iNV+2qKErnf+ixWUWTK/7tuf7dBVaFfiR/Jp9WrWXScjadKU6nckuRaXGMYRxT7MwMcsijgDLPFDjGCU6lhJSVFqTFsFWKRZwRfAsp/wGonZKm</latexit><latexit sha1_base64="RsbJR7Y5sh6pFTlHTSYZ76KZZWk=">AAACenichVHLSsNAFD2N7/qqiiC4KS2KRSg3Ir6gILpx4cJaq4KWksRRg2kSk7SgoT/gD7hwpSCifoYbf8BFP0FcKrhx4W0aESnqDZk5c+aeO2fmqrahux5RLSK1tLa1d3R2Rbt7evv6YwODm65VdjSR1yzDcrZVxRWGboq8p3uG2LYdoZRUQ2ypR8v1/a2KcFzdMje8E1sUSsqBqe/rmuIxVYwNHxd9Ra1OUCoTotVURi7GkpSeJ3l+Ro43AzlNQSQRxpoVu8Eu9mBBQxklCJjwGBtQ4PK3AxkEm7kCfOYcRnqwL1BFlLVlzhKcoTB7xOMBr3ZC1uR1vaYbqDU+xeDfYWUcY/REt/RKj3RPz/Txay0/qFH3csKz2tAKu9h/NpJ7/1dV4tnD4bfqT88e9jEXeNXZux0w9VtoDX3l9Pw1t7A+5o/TFb2w/0uq0QPfwKy8addZsX6BaNCAxivHm8FXAzan0vJMeio7nVxcClvRiVEkMMHvPYtFrGANeT7XxyXucB/5kBJSSppspEqRUDOEHyFNfwIcI5Gq</latexit>

qab (0) = qab (L) = 1<latexit sha1_base64="Gr5q9ATJ0P+Cb8aqfmZFu1sUK8A=">AAACmHichVG7SgNREJ2srxgfidqINsEQiU2YDRKNIAQt1C4P84AYZHe9iUv2xe4mEIM/YC8WgqJgIf6CnY0/YJFPEMsINhbObiIiwTiXvXfumTmz586IhiJbNmLbww0Nj4yOecd9E5NT0/7AzGze0uumxHKSruhmURQspsgay9myrbCiYTJBFRVWEGvbTrzQYKYl69q+3TRYWRWqmlyRJcEmqGRGcGXTjPC0HQZCGE0gn4jzwX6Hj6JrIehZSg88wgEcgQ4S1EEFBhrY5CsggEWrBDwgGISVoUWYSZ7sxhmcgo+4dcpilCEQWqO9SrdSD9Xo7tS0XLZEf1HoM4kZhDC+4D128Bkf8BU//6zVcms4Wpp0il0uMw79Z/PZj39ZKp02HP+wBmq2oQLrrlaZtBsu4rxC6vIbJxed7EYm3FrGW3wj/TfYxid6gdZ4l+7SLHM5QI9BKpoUcbpnOb1zR9WdR7Df+R5VPhbl49FYejWU3OoNzQuLsAQRmswaJGEXUpBze3sOV3DNLXBJbofb66Zynh5nDn4Zl/kCVJOVaw==</latexit>r(0) = r(1) = r

SciPost Phys. Core 2, 005 (2020)

Bayes-optimal case (no noise)
HajimeYoshino,SciPostPhys.
Core
2,
005
(2020).
SciPost Phys. Core 2, 005 (2020)
SciPost Phys. Core 2, 005 (2020)↵of the EA order parameters qEA (l) = q0 (l) = r(l),ution k = 0) at ↵ = 25, 100, 250, 500, 625, 714. Here
> L and the liquid phase disappear, which is a ’finite↵

Figure 14:
The spatial profile of the EA order parameters qEA (l) = q0 (l) = r(l),g) minima, all of which correctly satisfy the constraints
onFigure
14: The spatial profile of the EA order parameters q (l) = q (l) = r(l),EA Here 0
Q EA (l) = Q 0 (l) = R(l) (RS solution k = 0) at ↵ = 25, 100, 250, 500, 625, 714.
Q EA(l)largest= Q 0 (l)= 0) at↵ = 25, 100,
625, 714. HereL = 20. Forthe↵, =⇠ R(l)> L (RSand solution
the liquidk phase
disappear,
which250,
is a500,’finiteBayes
optimal,Nishimoricondition
proposal for the real symmetry breaking field which
works
L
=
20.
For
the
largest
↵,
⇠
>
L
and
the
liquidphase
disappear,
which
is
a
’finitedepth’ effect .
ink of the following unlearning. Suppose that we give thedepth’ effect .

“Wetting transition”

Replica symmetry (RS) holds
figuration of the teacher
machine at the2nd
beginningthenletlayer-by-layer
ordertransition
hidden in the ocean of many (wrong) minima, all of which correctly satisfy the constraints on
he constraint by the training data of size M= ↵N . Ourhiddenin theocean of many (wrong) minima, all of which correctly satisfy the constraints on
the input andoutputboundaries.
chine will keep the configuration of the teacher
machinethe inputweandAt the moment
do output
not haveboundaries.
a proposal for the real symmetry breaking field which works
on of size ⇠ / ln ↵ but forget the teacher machinein theduring learning.
we canthe following
that
we givefield
the which works
At Instead,the moment
we think
do notofhave
a proposalunlearning.
for the realSuppose
symmetrybreaking

student machinecompleteconfiguration
the ofteachermachine unlearning.
at the beginningthen that
let we give theduringthelearning.
Instead,we can ofthink
the following
Supposeme weak correlation between the teacher andmathestudent
studentstudent
machinerelax under
the constraint
by the training
data ofmachinesize M =Ourmachinethe complete
configurationof the teacher
at ↵Nthe. beginningthen let<latexit sha1_base64="2aq61blyQLh1svI2UNt8eIK8Qrg=">AAACbXichVHLSsNAFD2N7/qqiiAoUiw+FlJupFgrCKIbl7ZaLahIEkcNpklM0mIt/oBLNy50oyAofoYbf8CFnyAuXCi4ceFtEhAR9Q7MnDn3nsuZuapt6K5H9BiR6uobGpuaW6Ktbe0dnbGu7hXXKjmayGuWYTkFVXGFoZsi7+meIQq2I5SiaohVdW++ll8tC8fVLXPZq9hio6jsmPq2rikeU4X9GWc8np3JbcYSlMyQnJmU4z+BnCQ/Eghj0YpdYx1bsKChhCIETHiMDShwea1BBsFmbgNV5hxGup8XOEKUtSWuElyhMLvH+w7f1kLW5Hutp+urD7jCYqXFXY4QxzA90A290j3d0hN9/Nqt6nepuanwqQZaYW92Hvctvf+rKvLpYfdL9adrD9uY8t3q7N72mdo7tEBfPjx9XZrODVdH6JKe2f8FPdIdv8Asv2lXWZE7QzQYgR/xAKRTIch8jWBlIilPJlPZVGJ2LhxGM/oxhDH+8TRmsYBF5P2fPsEZziMvUq80IA0GpVIk1PTgW0ijn7MYjXw=</latexit>

q = r, Q = R

Generalization ability
teacher-student overlap
for “test data”

teacher-student overlap of bondafter training<latexit sha1_base64="xlI2Ukunxr03/sSF4qkoPgdyl3k=">AAAConicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsIqxRp5GiqKGQWK+TllyikZVakpsQLKBvoWRoYWpoZKmAyDPUMwECZAQoC8gX2MMQwpDDkMyQzlDLkMqQy5DGUANk5DIkMxUAYzWDIYMBQABSLZagGihUBWZlg+VSGWgYuoN5SoKpUoIpEoGg2kEwH8qKhonlAPsjMYrDuZKAtOUBcBNSpwKBqcNVgpcFngxMGqw1eGvzBaVY12AyQWyqBdBJEb2pBPH+XRPB3grpygXQJQwZCF143lzCkMViA3ZoJdHsBWATki2SI/rKq6Z+DrYJUq9UMFhm8Brp/ocFNg8NAH+SVfUleGpgaNBuPewqArqgEyoBCrxgUduCogsSHAiYDFlVhRnqGZnpGgUbKDk7QSONgkGZQYtAAxow5gwODB0MAQyjQzEqG+QwrGFYyqTB5MQUyBUOUMjFC9QgzoACmGAA7TJkr</latexit>r(l) is not fixed

rtest

R
<latexit sha1_base64="9YHHxoiUk2cJJehYZMmslWomvpU=">AAACjnichVE9S8NQFD2NX7V+VV0El2JRnMpNkWoFUXRx7Ie1gook8bUG0yQkaaEW/4CuioOTgoP4F9xc/AMO/QniqODi4E1aERHrDcm779x7bs57R7UN3fWImiGpq7unty/cHxkYHBoeiY6ObbpW1dFEQbMMy9lSFVcYuikKnu4ZYst2hFJRDVFUD9f8erEmHFe3zA2vbovdilI29ZKuKR5D2dxeNE6JNMnplBz7ncgJCiKOdmSs6D12sA8LGqqoQMCEx7kBBS4/25BBsBnbRYMxhzM9qAscI8LcKncJ7lAYPeRvmXfbbdTkvT/TDdga/8Xg12FmDNP0RLf0So90R8/08eesRjDD11LnVW1xhb03cjKRf/+XVeHVw8E3q6NmDyUsBFp11m4HiH8KrcWvHV285hdz040ZuqYX1n9FTXrgE5i1N+0mK3KXHfTYrKLOFf/2XP/uAqtafsR+J19WbSYTciqRzM7FV1bbpoUxiSnMsjPzWME6MijwTIFTnOFcikopaUlabrVKoTZnHD9CWv8EY12Shg==</latexit><latexit sha1_base64="hmbergAs3iiZfu7w1UtWvi+xJZo=">AAACmXichVG7SgNRED2ur/iO2gg2waBYhdkgvqqojVipMSokIeyu17i4L3ZvAjH4A36AFhY+wEL8BTsbf8DCTxBLBRsLZzcREVFn2b1zz8yZPfce3bPMQBI9tiitbe0dnbGu7p7evv6B+ODQZuBWfEPkDNdy/W1dC4RlOiInTWmJbc8Xmq1bYkvfXwrrW1XhB6brbMiaJ4q2VnbMXdPQJEMFv1Qv+HZCikAeluJJSs2ROjetJn4maoqiSKIZq278FgXswIWBCmwIOJCcW9AQ8JOHCoLHWBF1xnzOzKgucIhu5la4S3CHxug+f8u8yzdRh/fhzCBiG/wXi1+fmQmM0wNd0wvd0w090fuvs+rRjFBLjVe9wRVeaeBoJPv2L8vmVWLvi/WnZoldzEZaTdbuRUh4CqPBrx6cvGTn18frE3RJz6z/gh7pjk/gVF+NqzWxfvqHHo9V1LgS3l4Q3l1kVcOPxM/k06rNdEqdTqXXppKZxaZpMYxiDJPszAwyWMYqcjzTwzHOcK6MKgvKsrLSaFVampxhfAsl+wH/xpeG</latexit>

non-zero solution!

↵ = 250<latexit sha1_base64="viC0BYmC5inzntcX6nq+FzZw58M=">AAACl3ichVHLSsNAFD3G97vqRnFTLIqrcBPrE0RREJdWrYoPJImjhqZJSNJCLf6Aa8GFKCi4EH/BnRt/wIWfIC4V3LjwJqmIiHqHmblz7j13zszVXcv0A6LHKqm6prauvqGxqbmlta090dG54jsFzxBZw7Ecb03XfGGZtsgGZmCJNdcTWl63xKqemw3jq0Xh+aZjLwclV2zltT3b3DUNLWBofVOz3H1tUh2m7USKZBpJK6qSJHmYlDGVYmd8dCipyBRZChVbcBK32MQOHBgoIA8BGwH7FjT4PDaggOAytoUyYx57ZhQXOEQTcwucJThDYzTH6x6fNiqozeewph+xDb7F4ukxM4l+eqBreqF7uqEnev+1VjmqEWop8a7HXOFutx91L739y8rzHmD/i/Wn5gC7GIu0mqzdjZDwFUbMLx6cvCxNLPaXB+iSnln/BT3SHb/ALr4aVxmxePqHHpdVlDgS/p4f/l3cqsiSP53PVq2osjIiq5l0anqm0rQG9KIPg9yZUUxjHgvIck0bxzjDudQjTUlz0nycKlVVOF34ZlLmA9dala0=</latexit>on this side zerosolution also exist↵ = 25<latexit sha1_base64="wBxtesIidFV/k+ZGLrhH3AEZoag=">AAAClnichVHLLgRBFD3T3uM12ExiMzEhVpPbE++ECBGWMxiPINLdaujoV7p7JhkTP2BrYYEFiYX4BTsbP2DhE8SSxMbC7Z5GLHArVXXr3HtunaqrOobu+USPMamuvqGxqbkl3trW3tGZ6Ope8eySq4mCZhu2u6YqnjB0SxR83TfEmuMKxVQNsaruzwbx1bJwPd22lv2KI7ZMZdfSi7qm+AytbyqGs6dMZoe3E2nKDJM8PiKnKEOhpb4QOULSiCxnJ26xiR3Y0FCCCQELPvsGFHg8NiCD4DC2hSpjLnt6GBc4RJy5Jc4SnKEwus/rLp82ItTic1DTC9ka32LwdJmZQj890DW90D3d0BO9/1qrGtYItFR4V2tc4Wx3HiWX3v5lmbz72Ptm/anZRxFjoVadtTshErxCq/HLBycvSxOL/dUBuqRn1n9Bj3THL7DKr9pVXiye/qHHYRUVjgS/5wV/x6367Efqd2clm5FHMtn8UHp6JmpaM3rRh0HuzCimsYAcClzTxDHOcC4lpSlpTpqvpUqxiNODHyblPgAxfpVr</latexit>l<latexit sha1_base64="CxfxIN62aDLVL2SK4iF4qeGEQlI=">AAACjnichVE9S8NQFD3G7/rRqovgUiwVp3IjUq0gii6O1toPUJEkvtZgmoQkLdTiH9BVcXBScBD/gpuLf8ChP0EcK7g4eJNWRMR6Q/LuO/eem/PeUW1Ddz2iRpfU3dPb1z8wGBoaHhkNR8bGc65VcTSR1SzDcgqq4gpDN0XW0z1DFGxHKGXVEHn1aN2v56vCcXXL3PZqttgrKyVTL+qa4jGUNvYjMUqkSE4l5ejvRE5QEDG0Y9OKPGAXB7CgoYIyBEx4nBtQ4PKzAxkEm7E91BlzONODusAJQsytcJfgDoXRI/6WeLfTRk3e+zPdgK3xXwx+HWZGEadnuqMmPdE9vdDHn7PqwQxfS41XtcUV9n74dDLz/i+rzKuHw29WR80eilgMtOqs3Q4Q/xRai189vmxmlrbi9Rm6oVfWf00NeuQTmNU37TYttq466LFZRY0r/u25/t0FVrX8iP5OvqzKzSXkZGIuPR9bXWubNoApTGOWnVnAKjawiSzPFDjDOS6kiJSUlqWVVqvU1eZM4EdIG5+boZKg</latexit>l<latexit sha1_base64="CxfxIN62aDLVL2SK4iF4qeGEQlI=">AAACjnichVE9S8NQFD3G7/rRqovgUiwVp3IjUq0gii6O1toPUJEkvtZgmoQkLdTiH9BVcXBScBD/gpuLf8ChP0EcK7g4eJNWRMR6Q/LuO/eem/PeUW1Ddz2iRpfU3dPb1z8wGBoaHhkNR8bGc65VcTSR1SzDcgqq4gpDN0XW0z1DFGxHKGXVEHn1aN2v56vCcXXL3PZqttgrKyVTL+qa4jGUNvYjMUqkSE4l5ejvRE5QEDG0Y9OKPGAXB7CgoYIyBEx4nBtQ4PKzAxkEm7E91BlzONODusAJQsytcJfgDoXRI/6WeLfTRk3e+zPdgK3xXwx+HWZGEadnuqMmPdE9vdDHn7PqwQxfS41XtcUV9n74dDLz/i+rzKuHw29WR80eilgMtOqs3Q4Q/xRai189vmxmlrbi9Rm6oVfWf00NeuQTmNU37TYttq466LFZRY0r/u25/t0FVrX8iP5OvqzKzSXkZGIuPR9bXWubNoApTGOWnVnAKjawiSzPFDjDOS6kiJSUlqWVVqvU1eZM4EdIG5+boZKg</latexit>

over-parametrized DNN with “central liquid region” generalizes!
(if symmetry breaking ﬁeld helps).

With noise
Spatial profile of the EA order parameter

noise<latexit sha1_base64="J89iRxYYkWTWlFnr+Fd39gm+AJk=">AAACdnichVHJSgNBEH0Z97gk6kUQJBjicgk1EtQIgguCR7eooBJmxlYHZ3NmEtSQH/AHPHhSFBQ/w4s/4MFPEI8KevBgZWZARNRq6H79ql7xukt1DN3ziR5jUl19Q2NTc0u8ta29I5Hs7Fr17JKriYJmG7a7riqeMHRLFHzdN8S64wrFVA2xpu7P1vJrZeF6um2t+EeO2DKVXUvf0TXFZ6qYTBwUK5uumZqbrk4eDMnDxWSasnmS86Ny6ieQsxREGlEs2MlrbGIbNjSUYELAgs/YgAKP1wZkEBzmtlBhzmWkB3mBKuKsLXGV4AqF2X3ed/m2EbEW32s9vUB9yBU2K23uUkUKGXqgG3qhe7qlJ/r4tVsl6FJzc8SnGmqFU0yc9Cy//asy+fSx96X607WPHYwHbnV27wRM7R1aqC8fn74sTyxlKgN0Qc/s/5we6Y5fYJVftatFsXSGeDiCIFIhGMtFIP81gtWRrDyazS3m0lMz0TCa0Yt+DPGPj2EK81hAIZjKGS5xFXuX+qSMNBiWSrFI041vIdEnfcKQgA==</latexit>

qEA = q(1)<latexit sha1_base64="VYw+7kwY4fFRl9gAQKzDqykc8cg=">AAACaHichVHLSsNAFD2Nr1pfVRcqbopFcVVupLRWEEQ3Lm1rVVCRJE5rME1CkhZr8QcEt7pwpSAofoYbf8CFn6AuFdy48DYJiIh6B2bOnLnnzpm5qm3orkf0GJHa2js6u6LdsZ7evv6B+ODQmmvVHE2UNMuwnA1VcYWhm6Lk6Z4hNmxHKFXVEOvq/lLrfL0uHFe3zFWvYYvtqlIx9bKuKR5TRWde3oknKZUjOZeREz+BnCI/kghjxYpfYwu7sKChhioETHiMDShweWxCBsFmbhtN5hxGun8ucIQYa2ucJThDYXaf5wrvNkPW5H2rpuurDzjDYqXFVY6QwCQ90A290j3d0hN9/Fqt6VdpuWnwqgZaYe8MHI8W3/9VVXn1sPel+tO1hzJmfbc6u7d9pvUOLdDXD89ei3OFyeYUXdIL+7+gR7rjF5j1N+0qLwrniAUt8CMRgGw6BLmvFqzNpORMKp1PJxcWw2ZEMY4JTPOPZ7GAZaygxPdWcIJTnEWepbg0Io0FqVIk1AzjW0gTnxV0i94=</latexit>

r=1<latexit sha1_base64="kJ0LeUMJXBNwA8p+HWdZxD2GgRE=">AAACanichVHLLgRBFD3T3uM12BCbiUGsOrdlPEYiETaWXoMEke5WQ0VPd6e7Z4KJH7CxRKxIJMRn2PgBC58g7EhsLNzp7kREcCupOnXqnlun6hquJf2A6DGh1NTW1Tc0NiWbW1rb2lMdncu+U/JMkTcdy/FWDd0XlrRFPpCBJVZdT+hFwxIrxu5M9XylLDxfOvZSsO+KjaK+bcuCNPWAqbw3SerIZipDao603KiW/gk0lcLIII45J3WNdWzBgYkSihCwETC2oMPnsQYNBJe5DVSY8xjJ8FzgEEnWljhLcIbO7C7P27xbi1mb99Wafqje4wyHlQ5XOUQaA/RAN/RK93RLT/Txa7VKWKXqZp9XI9IKd7P9qHvx/V9VkdcAO1+qP10HKGA8dCvZvRsy1XeYkb58cPK6OLEwUBmkS3pm/xf0SHf8Arv8Zl7Ni4VzJKMWhJGOwFg2BrmvFiwPq9qomp3PZqam42Y0ohd9GOIfH8MUZjGHPN8rcYxTnCVelE6lR+mNUpVErOnCt1D6PwEnAYxU</latexit>r = 0.5q=r<latexit sha1_base64="CCKBdmc5U46ZWS1bC9ZcH5fM2yQ=">AAACanichVHLLgRBFD3TXmO8xtgQm4lBrDq3ZYKRSISNpdcgQaS7FSp6ujvdPRNM/ICNJWJFIiE+w8YPWPgEYUdiY+FOdycigltJ1alT99w6VddwLekHRI8Jpa6+obEp2ZxqaW1r70h3ZpZ8p+yZomg6luOtGLovLGmLYiADS6y4ntBLhiWWjd3p2vlyRXi+dOzFYN8V6yV925Zb0tQDporeBKnaRjpHaoG0woiW/Qk0lcLIIY5ZJ32NNWzCgYkyShCwETC2oMPnsQoNBJe5dVSZ8xjJ8FzgECnWljlLcIbO7C7P27xbjVmb97Wafqje4wyHlQ5XOUQWA/RAN/RK93RLT/Txa7VqWKXmZp9XI9IKd6PjqHvh/V9VidcAO1+qP10H2MJY6Fayezdkau8wI33l4OR1YXx+oDpIl/TM/i/oke74BXblzbyaE/PnSEUtCCMbgdF8DApfLVgaVrURNT+Xz01Oxc1Iohd9GOIfH8UkZjCLIt8rcYxTnCVelIzSo/RGqUoi1nThWyj9nx75jFA=</latexit>r = 0.1qqrr<latexit sha1_base64="uDEOnnLcPUpgxTkLNsDO/amAZbY=">AAACZnichVFNLwNRFD0dX1VUESGxaTTEqrkjjaqVsLFUFEk1MjNemZjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLidmUREcF/y3nnn3XPfee+qtqG7HtFjRGpr7+jsinbHenr74v2JgcF116o6miholmE5m6riCkM3RcHTPUNs2o5QKqohNtT9xdb5Rk04rm6Za17dFqWKsmvqZV1TPKbyxnYiRekcybkZOfkTyGnyI4Uwlq3ENbawAwsaqqhAwITH2IACl0cRMgg2cyU0mHMY6f65QBMx1lY5S3CGwuw+z7u8K4asyftWTddXH3CGxUqLqzSRxAQ90A290j3d0jN9/Fqt4VdpuanzqgZaYW/3H42uvv+rqvDqYe9L9adrD2XM+m51dm/7TOsdWqCvHZ69rs6tTDQm6ZJe2P8FPdIdv8CsvWlXebFyjljQAj+SAchmQpD7asH6dFqeSWfymdT8QtiMKMYwjin+8SzmsYRlFPhegWOc4DTyJMWlYWkkSJUioWYI30JKfgLgh4tW</latexit><latexit sha1_base64="uDEOnnLcPUpgxTkLNsDO/amAZbY=">AAACZnichVFNLwNRFD0dX1VUESGxaTTEqrkjjaqVsLFUFEk1MjNemZjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLidmUREcF/y3nnn3XPfee+qtqG7HtFjRGpr7+jsinbHenr74v2JgcF116o6miholmE5m6riCkM3RcHTPUNs2o5QKqohNtT9xdb5Rk04rm6Za17dFqWKsmvqZV1TPKbyxnYiRekcybkZOfkTyGnyI4Uwlq3ENbawAwsaqqhAwITH2IACl0cRMgg2cyU0mHMY6f65QBMx1lY5S3CGwuw+z7u8K4asyftWTddXH3CGxUqLqzSRxAQ90A290j3d0jN9/Fqt4VdpuanzqgZaYW/3H42uvv+rqvDqYe9L9adrD2XM+m51dm/7TOsdWqCvHZ69rs6tTDQm6ZJe2P8FPdIdv8CsvWlXebFyjljQAj+SAchmQpD7asH6dFqeSWfymdT8QtiMKMYwjin+8SzmsYRlFPhegWOc4DTyJMWlYWkkSJUioWYI30JKfgLgh4tW</latexit><latexit sha1_base64="uDEOnnLcPUpgxTkLNsDO/amAZbY=">AAACZnichVFNLwNRFD0dX1VUESGxaTTEqrkjjaqVsLFUFEk1MjNemZjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLidmUREcF/y3nnn3XPfee+qtqG7HtFjRGpr7+jsinbHenr74v2JgcF116o6miholmE5m6riCkM3RcHTPUNs2o5QKqohNtT9xdb5Rk04rm6Za17dFqWKsmvqZV1TPKbyxnYiRekcybkZOfkTyGnyI4Uwlq3ENbawAwsaqqhAwITH2IACl0cRMgg2cyU0mHMY6f65QBMx1lY5S3CGwuw+z7u8K4asyftWTddXH3CGxUqLqzSRxAQ90A290j3d0jN9/Fqt4VdpuanzqgZaYW/3H42uvv+rqvDqYe9L9adrD2XM+m51dm/7TOsdWqCvHZ69rs6tTDQm6ZJe2P8FPdIdv8CsvWlXebFyjljQAj+SAchmQpD7asH6dFqeSWfymdT8QtiMKMYwjin+8SzmsYRlFPhegWOc4DTyJMWlYWkkSJUioWYI30JKfgLgh4tW</latexit>l↵ = 25<latexit sha1_base64="qgim9nWoWZNIW3jvS9G3YAjBQM8=">AAAClnichVFLLwNRFP6MVz1bbCQ2jYZYNWcaSiWkIcKyHvUIIjPjtibmlZlpk2r8AVsLCyxILMRfsLPxByz8BLEksbFwZloREZzJzD33O+c78937qY6hez7RY4PU2NTc0hppa+/o7OqOxnp6Vz275Goir9mG7a6riicM3RJ5X/cNse64QjFVQ6yp+7NBfa0sXE+3rRW/4ohtUylaekHXFJ+hjS3FcPaUqdTYTixByQzJmbQc/5nISQojgXrk7NgttrALGxpKMCFgwefcgAKPn03IIDiMbaPKmMuZHtYFDtHO3BJ3Ce5QGN3nb5F3m3XU4n0w0wvZGv/F4NdlZhxD9EDX9EL3dENP9P7rrGo4I9BS4VWtcYWzEz3qX377l2Xy6mPvi/WnZh8FTIRaddbuhEhwCq3GLx+cvCxPLg1Vh+mSnln/BT3SHZ/AKr9qV4ti6fQPPQ6rqHAluD0vuLvQqpof8Z/Jp1WrqaScTqYWRxPZmbppEQxgECPszDiyWEAOeZ5p4hhnOJf6pWlpTpqvtUoNdU4fvoWU+wB3hJWK</latexit>llHierarchical structure of the solutions<latexit sha1_base64="UsJEU170eew+4aXKx9eY3+Qu1EA=">AAACaXichVHLSsNAFD2N7/po1U3RTTEouik3UtS6Et249FUVVCSJo4amSUzSohZ/wIXbCq4UBMXPcOMPuOgnSJcKblx48wARUe/AzJkz99w5M1dzTMPziRoJqaW1rb2jsyvZ3dPbl0r3D6x7dsXVRVG3Tdvd1FRPmIYlir7hm2LTcYVa1kyxoZUWgvONqnA9w7bW/BNH7JTVA8vYN3TVD6ij8eOJ3bRMuQIphSkl+xMoOQpDRhxLdvoO29iDDR0VlCFgwWdsQoXHYwsKCA5zO6gx5zIywnOBMyRZW+EswRkqsyWeD3i3FbMW74OaXqg+5gyblTZXOUMWo/RM9/RKT/RAL/Txa7VaWCVwc8KrFmmFs5s6z6y+/6sq8+rj8Ev1p2sf+5gJ3Rrs3gmZ4B16pK+e1l9XZ1dGa2N0Q032f00NeuQXWNU3/XZZrFwhGbUgjGwEpvMxKHy1YH0yp0zl8st5eW4+bkYnhjGCcf7xacxhEUso8r2HuEAdl4mm1C9lpKEoVUrEmkF8C0n+BPCIjEI=</latexit>

q(x)<latexit sha1_base64="VYw+7kwY4fFRl9gAQKzDqykc8cg=">AAACaHichVHLSsNAFD2Nr1pfVRcqbopFcVVupLRWEEQ3Lm1rVVCRJE5rME1CkhZr8QcEt7pwpSAofoYbf8CFn6AuFdy48DYJiIh6B2bOnLnnzpm5qm3orkf0GJHa2js6u6LdsZ7evv6B+ODQmmvVHE2UNMuwnA1VcYWhm6Lk6Z4hNmxHKFXVEOvq/lLrfL0uHFe3zFWvYYvtqlIx9bKuKR5TRWde3oknKZUjOZeREz+BnCI/kghjxYpfYwu7sKChhioETHiMDShweWxCBsFmbhtN5hxGun8ucIQYa2ucJThDYXaf5wrvNkPW5H2rpuurDzjDYqXFVY6QwCQ90A290j3d0hN9/Fqt6VdpuWnwqgZaYe8MHI8W3/9VVXn1sPel+tO1hzJmfbc6u7d9pvUOLdDXD89ei3OFyeYUXdIL+7+gR7rjF5j1N+0qLwrniAUt8CMRgGw6BLmvFqzNpORMKp1PJxcWw2ZEMY4JTPOPZ7GAZaygxPdWcIJTnEWepbg0Io0FqVIk1AzjW0gTnxV0i94=</latexit>

r=1

l=1

<latexit sha1_base64="kJ0LeUMJXBNwA8p+HWdZxD2GgRE=">AAACanichVHLLgRBFD3T3uM12BCbiUGsOrdlPEYiETaWXoMEke5WQ0VPd6e7Z4KJH7CxRKxIJMRn2PgBC58g7EhsLNzp7kREcCupOnXqnlun6hquJf2A6DGh1NTW1Tc0NiWbW1rb2lMdncu+U/JMkTcdy/FWDd0XlrRFPpCBJVZdT+hFwxIrxu5M9XylLDxfOvZSsO+KjaK+bcuCNPWAqbw3SerIZipDao603KiW/gk0lcLIII45J3WNdWzBgYkSihCwETC2oMPnsQYNBJe5DVSY8xjJ8FzgEEnWljhLcIbO7C7P27xbi1mb99Wafqje4wyHlQ5XOUQaA/RAN/RK93RLT/Txa7VKWKXqZp9XI9IKd7P9qHvx/V9VkdcAO1+qP10HKGA8dCvZvRsy1XeYkb58cPK6OLEwUBmkS3pm/xf0SHf8Arv8Zl7Ni4VzJKMWhJGOwFg2BrmvFiwPq9qomp3PZqam42Y0ohd9GOIfH8MUZjGHPN8rcYxTnCVelE6lR+mNUpVErOnCt1D6PwEnAYxU</latexit>r = 0.5

l=1

l=2

x
Replica symmetricr = 0.1l=1

l=2<latexit sha1_base64="gM5HNvXWs4sEIUlPnTZxllDm22I=">AAACZnichVHLSsNAFD2N7/qqiii4KRbFVbmRotaV6MZlH9YWqkgSRw2mSUjSohZ/QLeKC1cKguJnuPEHXPQPFJcKblx4mwSKiHoHZs6cuefOmbmqbeiuR9SISG3tHZ1d3T3R3r7+gcHY0PC6a1UdTRQ0y7Cckqq4wtBNUfB0zxAl2xFKRTVEUd1faZ4Xa8Jxdctc8w5tsVlRdk19R9cUj6nswVYsQck0yek5Of4TyEnyI4EwMlbsFhvYhgUNVVQgYMJjbECBy6MMGQSbuU3UmXMY6f65wDGirK1yluAMhdl9nnd5Vw5Zk/fNmq6vPuAMi5UWVzlGHFP0RHf0Ro90Ty/0+Wu1ul+l6eaQVzXQCntr8GQ8//GvqsKrh72W6k/XHnaw4LvV2b3tM813aIG+dnTxll/MTdWn6Zpe2f8VNeiBX2DW3rWbrMhdIhq0wI94AOZTIUi3WrA+m5TnkqlsKrG0HDajGxOYxAz/+DyWsIoMCnyvwCnOcB55lgakUWksSJUioWYE30KKfwH4n4ti</latexit><latexit sha1_base64="CCKBdmc5U46ZWS1bC9ZcH5fM2yQ=">AAACanichVHLLgRBFD3TXmO8xtgQm4lBrDq3ZYKRSISNpdcgQaS7FSp6ujvdPRNM/ICNJWJFIiE+w8YPWPgEYUdiY+FOdycigltJ1alT99w6VddwLekHRI8Jpa6+obEp2ZxqaW1r70h3ZpZ8p+yZomg6luOtGLovLGmLYiADS6y4ntBLhiWWjd3p2vlyRXi+dOzFYN8V6yV925Zb0tQDporeBKnaRjpHaoG0woiW/Qk0lcLIIY5ZJ32NNWzCgYkyShCwETC2oMPnsQoNBJe5dVSZ8xjJ8FzgECnWljlLcIbO7C7P27xbjVmb97Wafqje4wyHlQ5XOUQWA/RAN/RK93RLT/Txa7VqWKXmZp9XI9IKd6PjqHvh/V9VidcAO1+qP10H2MJY6Fayezdkau8wI33l4OR1YXx+oDpIl/TM/i/oke74BXblzbyaE/PnSEUtCCMbgdF8DApfLVgaVrURNT+Xz01Oxc1Iohd9GOIfH8UkZjCLIt8rcYxTnCVelIzSo/RGqUoi1nThWyj9nx75jFA=</latexit>

l=2x<latexit sha1_base64="gM5HNvXWs4sEIUlPnTZxllDm22I=">AAACZnichVHLSsNAFD2N7/qqiii4KRbFVbmRotaV6MZlH9YWqkgSRw2mSUjSohZ/QLeKC1cKguJnuPEHXPQPFJcKblx4mwSKiHoHZs6cuefOmbmqbeiuR9SISG3tHZ1d3T3R3r7+gcHY0PC6a1UdTRQ0y7Cckqq4wtBNUfB0zxAl2xFKRTVEUd1faZ4Xa8Jxdctc8w5tsVlRdk19R9cUj6nswVYsQck0yek5Of4TyEnyI4EwMlbsFhvYhgUNVVQgYMJjbECBy6MMGQSbuU3UmXMY6f65wDGirK1yluAMhdl9nnd5Vw5Zk/fNmq6vPuAMi5UWVzlGHFP0RHf0Ro90Ty/0+Wu1ul+l6eaQVzXQCntr8GQ8//GvqsKrh72W6k/XHnaw4LvV2b3tM813aIG+dnTxll/MTdWn6Zpe2f8VNeiBX2DW3rWbrMhdIhq0wI94AOZTIUi3WrA+m5TnkqlsKrG0HDajGxOYxAz/+DyWsIoMCnyvwCnOcB55lgakUWksSJUioWYE30KKfwH4n4ti</latexit>x<latexit sha1_base64="gM5HNvXWs4sEIUlPnTZxllDm22I=">AAACZnichVHLSsNAFD2N7/qqiii4KRbFVbmRotaV6MZlH9YWqkgSRw2mSUjSohZ/QLeKC1cKguJnuPEHXPQPFJcKblx4mwSKiHoHZs6cuefOmbmqbeiuR9SISG3tHZ1d3T3R3r7+gcHY0PC6a1UdTRQ0y7Cckqq4wtBNUfB0zxAl2xFKRTVEUd1faZ4Xa8Jxdctc8w5tsVlRdk19R9cUj6nswVYsQck0yek5Of4TyEnyI4EwMlbsFhvYhgUNVVQgYMJjbECBy6MMGQSbuU3UmXMY6f65wDGirK1yluAMhdl9nnd5Vw5Zk/fNmq6vPuAMi5UWVzlGHFP0RHf0Ro90Ty/0+Wu1ul+l6eaQVzXQCntr8GQ8//GvqsKrh72W6k/XHnaw4LvV2b3tM813aIG+dnTxll/MTdWn6Zpe2f8VNeiBX2DW3rWbrMhdIhq0wI94AOZTIUi3WrA+m5TnkqlsKrG0HDajGxOYxAz/+DyWsIoMCnyvwCnOcB55lgakUWksSJUioWYE30KKfwH4n4ti</latexit>

Replica symmetry broken

<latexit sha1_base64="CCKBdmc5U46ZWS1bC9ZcH5fM2yQ=">AAACanichVHLLgRBFD3TXmO8xtgQm4lBrDq3ZYKRSISNpdcgQaS7FSp6ujvdPRNM/ICNJWJFIiE+w8YPWPgEYUdiY+FOdycigltJ1alT99w6VddwLekHRI8Jpa6+obEp2ZxqaW1r70h3ZpZ8p+yZomg6luOtGLovLGmLYiADS6y4ntBLhiWWjd3p2vlyRXi+dOzFYN8V6yV925Zb0tQDporeBKnaRjpHaoG0woiW/Qk0lcLIIY5ZJ32NNWzCgYkyShCwETC2oMPnsQoNBJe5dVSZ8xjJ8FzgECnWljlLcIbO7C7P27xbjVmb97Wafqje4wyHlQ5XOUQWA/RAN/RK93RLT/Txa7VqWKXmZp9XI9IKd6PjqHvh/V9VidcAO1+qP10H2MJY6Fayezdkau8wI33l4OR1YXx+oDpIl/TM/i/oke74BXblzbyaE/PnSEUtCCMbgdF8DApfLVgaVrURNT+Xz01Oxc1Iohd9GOIfH8UkZjCLIt8rcYxTnCVelIzSo/RGqUoi1nThWyj9nx75jFA=</latexit><latexit sha1_base64="CCKBdmc5U46ZWS1bC9ZcH5fM2yQ=">AAACanichVHLLgRBFD3TXmO8xtgQm4lBrDq3ZYKRSISNpdcgQaS7FSp6ujvdPRNM/ICNJWJFIiE+w8YPWPgEYUdiY+FOdycigltJ1alT99w6VddwLekHRI8Jpa6+obEp2ZxqaW1r70h3ZpZ8p+yZomg6luOtGLovLGmLYiADS6y4ntBLhiWWjd3p2vlyRXi+dOzFYN8V6yV925Zb0tQDporeBKnaRjpHaoG0woiW/Qk0lcLIIY5ZJ32NNWzCgYkyShCwETC2oMPnsQoNBJe5dVSZ8xjJ8FzgECnWljlLcIbO7C7P27xbjVmb97Wafqje4wyHlQ5XOUQWA/RAN/RK93RLT/Txa7VqWKXmZp9XI9IKd6PjqHvh/V9VidcAO1+qP10H2MJY6Fayezdkau8wI33l4OR1YXx+oDpIl/TM/i/oke74BXblzbyaE/PnSEUtCCMbgdF8DApfLVgaVrURNT+Xz01Oxc1Iohd9GOIfH8UkZjCLIt8rcYxTnCVelIzSo/RGqUoi1nThWyj9nx75jFA=</latexit>r = 0.1q<latexit sha1_base64="J89iRxYYkWTWlFnr+Fd39gm+AJk=">AAACdnichVHJSgNBEH0Z97gk6kUQJBjicgk1EtQIgguCR7eooBJmxlYHZ3NmEtSQH/AHPHhSFBQ/w4s/4MFPEI8KevBgZWZARNRq6H79ql7xukt1DN3ziR5jUl19Q2NTc0u8ta29I5Hs7Fr17JKriYJmG7a7riqeMHRLFHzdN8S64wrFVA2xpu7P1vJrZeF6um2t+EeO2DKVXUvf0TXFZ6qYTBwUK5uumZqbrk4eDMnDxWSasnmS86Ny6ieQsxREGlEs2MlrbGIbNjSUYELAgs/YgAKP1wZkEBzmtlBhzmWkB3mBKuKsLXGV4AqF2X3ed/m2EbEW32s9vUB9yBU2K23uUkUKGXqgG3qhe7qlJ/r4tVsl6FJzc8SnGmqFU0yc9Cy//asy+fSx96X607WPHYwHbnV27wRM7R1aqC8fn74sTyxlKgN0Qc/s/5we6Y5fYJVftatFsXSGeDiCIFIhGMtFIP81gtWRrDyazS3m0lMz0TCa0Yt+DPGPj2EK81hAIZjKGS5xFXuX+qSMNBiWSrFI041vIdEnfcKQgA==</latexit>

qEA = q(1)r = 0.1q<latexit sha1_base64="J89iRxYYkWTWlFnr+Fd39gm+AJk=">AAACdnichVHJSgNBEH0Z97gk6kUQJBjicgk1EtQIgguCR7eooBJmxlYHZ3NmEtSQH/AHPHhSFBQ/w4s/4MFPEI8KevBgZWZARNRq6H79ql7xukt1DN3ziR5jUl19Q2NTc0u8ta29I5Hs7Fr17JKriYJmG7a7riqeMHRLFHzdN8S64wrFVA2xpu7P1vJrZeF6um2t+EeO2DKVXUvf0TXFZ6qYTBwUK5uumZqbrk4eDMnDxWSasnmS86Ny6ieQsxREGlEs2MlrbGIbNjSUYELAgs/YgAKP1wZkEBzmtlBhzmWkB3mBKuKsLXGV4AqF2X3ed/m2EbEW32s9vUB9yBU2K23uUkUKGXqgG3qhe7qlJ/r4tVsl6FJzc8SnGmqFU0yc9Cy//asy+fSx96X607WPHYwHbnV27wRM7R1aqC8fn74sTyxlKgN0Qc/s/5we6Y5fYJVftatFsXSGeDiCIFIhGMtFIP81gtWRrDyazS3m0lMz0TCa0Yt+DPGPj2EK81hAIZjKGS5xFXuX+qSMNBiWSrFI041vIdEnfcKQgA==</latexit>

qEA = q(1)rr<latexit sha1_base64="uDEOnnLcPUpgxTkLNsDO/amAZbY=">AAACZnichVFNLwNRFD0dX1VUESGxaTTEqrkjjaqVsLFUFEk1MjNemZjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLidmUREcF/y3nnn3XPfee+qtqG7HtFjRGpr7+jsinbHenr74v2JgcF116o6miholmE5m6riCkM3RcHTPUNs2o5QKqohNtT9xdb5Rk04rm6Za17dFqWKsmvqZV1TPKbyxnYiRekcybkZOfkTyGnyI4Uwlq3ENbawAwsaqqhAwITH2IACl0cRMgg2cyU0mHMY6f65QBMx1lY5S3CGwuw+z7u8K4asyftWTddXH3CGxUqLqzSRxAQ90A290j3d0jN9/Fqt4VdpuanzqgZaYW/3H42uvv+rqvDqYe9L9adrD2XM+m51dm/7TOsdWqCvHZ69rs6tTDQm6ZJe2P8FPdIdv8CsvWlXebFyjljQAj+SAchmQpD7asH6dFqeSWfymdT8QtiMKMYwjin+8SzmsYRlFPhegWOc4DTyJMWlYWkkSJUioWYI30JKfgLgh4tW</latexit><latexit sha1_base64="uDEOnnLcPUpgxTkLNsDO/amAZbY=">AAACZnichVFNLwNRFD0dX1VUESGxaTTEqrkjjaqVsLFUFEk1MjNemZjOTGamjWr6B9gSCysSCfEzbPwBC/+AWJLYWLidmUREcF/y3nnn3XPfee+qtqG7HtFjRGpr7+jsinbHenr74v2JgcF116o6miholmE5m6riCkM3RcHTPUNs2o5QKqohNtT9xdb5Rk04rm6Za17dFqWKsmvqZV1TPKbyxnYiRekcybkZOfkTyGnyI4Uwlq3ENbawAwsaqqhAwITH2IACl0cRMgg2cyU0mHMY6f65QBMx1lY5S3CGwuw+z7u8K4asyftWTddXH3CGxUqLqzSRxAQ90A290j3d0jN9/Fqt4VdpuanzqgZaYW/3H42uvv+rqvDqYe9L9adrD2XM+m51dm/7TOsdWqCvHZ69rs6tTDQm6ZJe2P8FPdIdv8CsvWlXebFyjljQAj+SAchmQpD7asH6dFqeSWfymdT8QtiMKMYwjin+8SzmsYRlFPhegWOc4DTyJMWlYWkkSJUioWYI30JKfgLgh4tW</latexit>ll<latexit sha1_base64="UsJEU170eew+4aXKx9eY3+Qu1EA=">AAACaXichVHLSsNAFD2N7/po1U3RTTEouik3UtS6Et249FUVVCSJo4amSUzSohZ/wIXbCq4UBMXPcOMPuOgnSJcKblx48wARUe/AzJkz99w5M1dzTMPziRoJqaW1rb2jsyvZ3dPbl0r3D6x7dsXVRVG3Tdvd1FRPmIYlir7hm2LTcYVa1kyxoZUWgvONqnA9w7bW/BNH7JTVA8vYN3TVD6ij8eOJ3bRMuQIphSkl+xMoOQpDRhxLdvoO29iDDR0VlCFgwWdsQoXHYwsKCA5zO6gx5zIywnOBMyRZW+EswRkqsyWeD3i3FbMW74OaXqg+5gyblTZXOUMWo/RM9/RKT/RAL/Txa7VaWCVwc8KrFmmFs5s6z6y+/6sq8+rj8Ev1p2sf+5gJ3Rrs3gmZ4B16pK+e1l9XZ1dGa2N0Q032f00NeuQXWNU3/XZZrFwhGbUgjGwEpvMxKHy1YH0yp0zl8st5eW4+bkYnhjGCcf7xacxhEUso8r2HuEAdl4mm1C9lpKEoVUrEmkF8C0n+BPCIjEI=</latexit><latexit sha1_base64="UsJEU170eew+4aXKx9eY3+Qu1EA=">AAACaXichVHLSsNAFD2N7/po1U3RTTEouik3UtS6Et249FUVVCSJo4amSUzSohZ/wIXbCq4UBMXPcOMPuOgnSJcKblx48wARUe/AzJkz99w5M1dzTMPziRoJqaW1rb2jsyvZ3dPbl0r3D6x7dsXVRVG3Tdvd1FRPmIYlir7hm2LTcYVa1kyxoZUWgvONqnA9w7bW/BNH7JTVA8vYN3TVD6ij8eOJ3bRMuQIphSkl+xMoOQpDRhxLdvoO29iDDR0VlCFgwWdsQoXHYwsKCA5zO6gx5zIywnOBMyRZW+EswRkqsyWeD3i3FbMW74OaXqg+5gyblTZXOUMWo/RM9/RKT/RAL/Txa7VaWCVwc8KrFmmFs5s6z6y+/6sq8+rj8Ev1p2sf+5gJ3Rrs3gmZ4B16pK+e1l9XZ1dGa2N0Q032f00NeuQXWNU3/XZZrFwhGbUgjGwEpvMxKHy1YH0yp0zl8st5eW4+bkYnhjGCcf7xacxhEUso8r2HuEAdl4mm1C9lpKEoVUrEmkF8C0n+BPCIjEI=</latexit>

q(x)Replica symmetric!

q(x)

l=1

l=1

l=2

l=2

x
<latexit sha1_base64="gM5HNvXWs4sEIUlPnTZxllDm22I=">AAACZnichVHLSsNAFD2N7/qqiii4KRbFVbmRotaV6MZlH9YWqkgSRw2mSUjSohZ/QLeKC1cKguJnuPEHXPQPFJcKblx4mwSKiHoHZs6cuefOmbmqbeiuR9SISG3tHZ1d3T3R3r7+gcHY0PC6a1UdTRQ0y7Cckqq4wtBNUfB0zxAl2xFKRTVEUd1faZ4Xa8Jxdctc8w5tsVlRdk19R9cUj6nswVYsQck0yek5Of4TyEnyI4EwMlbsFhvYhgUNVVQgYMJjbECBy6MMGQSbuU3UmXMY6f65wDGirK1yluAMhdl9nnd5Vw5Zk/fNmq6vPuAMi5UWVzlGHFP0RHf0Ro90Ty/0+Wu1ul+l6eaQVzXQCntr8GQ8//GvqsKrh72W6k/XHnaw4LvV2b3tM813aIG+dnTxll/MTdWn6Zpe2f8VNeiBX2DW3rWbrMhdIhq0wI94AOZTIUi3WrA+m5TnkqlsKrG0HDajGxOYxAz/+DyWsIoMCnyvwCnOcB55lgakUWksSJUioWYE30KKfwH4n4ti</latexit>x<latexit sha1_base64="gM5HNvXWs4sEIUlPnTZxllDm22I=">AAACZnichVHLSsNAFD2N7/qqiii4KRbFVbmRotaV6MZlH9YWqkgSRw2mSUjSohZ/QLeKC1cKguJnuPEHXPQPFJcKblx4mwSKiHoHZs6cuefOmbmqbeiuR9SISG3tHZ1d3T3R3r7+gcHY0PC6a1UdTRQ0y7Cckqq4wtBNUfB0zxAl2xFKRTVEUd1faZ4Xa8Jxdctc8w5tsVlRdk19R9cUj6nswVYsQck0yek5Of4TyEnyI4EwMlbsFhvYhgUNVVQgYMJjbECBy6MMGQSbuU3UmXMY6f65wDGirK1yluAMhdl9nnd5Vw5Zk/fNmq6vPuAMi5UWVzlGHFP0RHf0Ro90Ty/0+Wu1ul+l6eaQVzXQCntr8GQ8//GvqsKrh72W6k/XHnaw4LvV2b3tM813aIG+dnTxll/MTdWn6Zpe2f8VNeiBX2DW3rWbrMhdIhq0wI94AOZTIUi3WrA+m5TnkqlsKrG0HDajGxOYxAz/+DyWsIoMCnyvwCnOcB55lgakUWksSJUioWYE30KKfwH4n4ti</latexit>

Simulation of learning in a teacher-student settingt<latexit sha1_base64="HMz+Z9SCkdv5eJHEYg8IaM7c964=">AAACjnichVE9S8NQFD3G7/pVdRFcikVxKjci9QNE0cXRWtsKKpLE1xpMk5C8FmrxD+iqODgpOIh/wc3FP+DQnyCOFVwcvEkrImK9IXn3nXvPzXnv6K5l+pKo1qa0d3R2dff0Rvr6BwaHosMjWd8peYbIGI7leNu65gvLtEVGmtIS264ntKJuiZx+tBbUc2Xh+aZjb8mKK/aKWsE286ahSYZScj8ap8QCqQtJNfY7URMURhzN2HCiD9jFARwYKKEIARuScwsafH52oILgMraHKmMeZ2ZYFzhBhLkl7hLcoTF6xN8C73aaqM37YKYfsg3+i8Wvx8wYJumZ7qhOT3RPL/Tx56xqOCPQUuFVb3CFuz90OpZ+/5dV5FXi8JvVUrNEHvOhVpO1uyESnMJo8MvHl/X04uZkdYpu6JX1X1ONHvkEdvnNuE2JzasWelxWUeFKcHt+cHehVQ0/Yr+TL6uyMwk1mZhJzcZXVpum9WAcE5hmZ+awgnVsIMMzBc5wjgslqiSVJWW50aq0NTmj+BHK+ies8ZKo</latexit>

T=0 greedy Monte Carlo
random teacher + student “a” and “b”

loss function

N X
M ⇣
X
µ
E=
SL,i
i=1 µ=1µ(S⇤ )L,i

1. “Unlearning” : start from teacher’s configuration
2. “Learning” : start from random configuration

N = 10, M = 10, L = 10

E
NM

t

⌘2

Permutation-invariant overlap

N
X
1
2
2
q ⌘
qij
N i,j=1

1
↵

qij =

M
X
1

M µ=1

µ a
µ b
(Si ) (Sj )

↵=1
test

training

N = 10, L = 10

N = 10, L = 10
teacher-student

2
⌅ 2

r
⌅ r

t = 1, 10, 102 , 103 , 104
unlearning

t

2
2

q
⇤
⇤ q

teacher-student• r2

unlearning

student-studentttlearning

learninglll↵ = 32
N = 10, L = 10

training
2

3

t = 1, 10, 10 , 10 , 10teacher-student

2

⌅

r

⇤

2

q

N = 10, L = 10

test

4

unlearning

unlearningteacher-student

• r

2

t

student-studenttlearning
learningllteacher-student overlap in the test

Generalization ability<latexit sha1_base64="4ii7slCLhEEv61aOuqvj8jzvxo4=">AAADEnichVG/TxRBFH67oiKKd2KDsbl4wWBzeUsMggkJ0YYGApwHJCxsdoc5mLC/MjN7ybnZys6ahMIKEwvjf0BsDI2FrQV/ArHExMbCtz+MEALOZne+9733vf1mnhf7QmnEY8O8NnD9xs3BW0O37wzfrdXvjayoKJGMd1jkR3LNcxX3Rcg7Wmifr8WSu4Hn81Vv92WeX+1xqUQUvtL9mG8E7nYouoK5miin/kY6qS2DhuZKZzN2V7ostbJ0YT6zVRI4qZixss10oYrsICni+Wy8TTlCxGRPzrTYrKDLdrg8X1WmlE62eKizMxqn3sTWNFrTk1bjIrBaWKwmVGsxqn8GG7YgAgYJBMAhBE3YBxcUPetgAUJM3AakxElCoshzyGCItAlVcapwid2l7zZF6xUbUpz3VIWa0V98eiUpGzCG3/EjnuJX/IQn+PvSXmnRI/fSp90rtTx2am9H27/+qwpo17DzT3WlZw1dmCq8CvIeF0x+Clbqe6/3T9vPl8fSx/gef5D/AzzGIzpB2PvJPizx5XdX+InJRZ8y+e2p/O6KUZXzaFwEf0e1MtGyJlsTS0+bsy+qoQ3CQ3gE4zSZZzALc7AIHep5YtSMUeOBuWceml/Mo7LUNCrNfTi3zG9/AEiNyEQ=</latexit>

N X
M
X

1
µ teacherµ student
rtest =(Si )test (Si )test
N M i=1 µ=1

N = 10, L = 5

unlearningrlearningtteacher-student overlap in the test

Generalization ability

N X
M
X

<latexit sha1_base64="4ii7slCLhEEv61aOuqvj8jzvxo4=">AAADEnichVG/TxRBFH67oiKKd2KDsbl4wWBzeUsMggkJ0YYGApwHJCxsdoc5mLC/MjN7ybnZys6ahMIKEwvjf0BsDI2FrQV/ArHExMbCtz+MEALOZne+9733vf1mnhf7QmnEY8O8NnD9xs3BW0O37wzfrdXvjayoKJGMd1jkR3LNcxX3Rcg7Wmifr8WSu4Hn81Vv92WeX+1xqUQUvtL9mG8E7nYouoK5miin/kY6qS2DhuZKZzN2V7ostbJ0YT6zVRI4qZixss10oYrsICni+Wy8TTlCxGRPzrTYrKDLdrg8X1WmlE62eKizMxqn3sTWNFrTk1bjIrBaWKwmVGsxqn8GG7YgAgYJBMAhBE3YBxcUPetgAUJM3AakxElCoshzyGCItAlVcapwid2l7zZF6xUbUpz3VIWa0V98eiUpGzCG3/EjnuJX/IQn+PvSXmnRI/fSp90rtTx2am9H27/+qwpo17DzT3WlZw1dmCq8CvIeF0x+Clbqe6/3T9vPl8fSx/gef5D/AzzGIzpB2PvJPizx5XdX+InJRZ8y+e2p/O6KUZXzaFwEf0e1MtGyJlsTS0+bsy+qoQ3CQ3gE4zSZZzALc7AIHep5YtSMUeOBuWceml/Mo7LUNCrNfTi3zG9/AEiNyEQ=</latexit>1µ teacherµ student
rtest =(Si )test (Si )test
N M i=1 µ=1

N = 10, L = 5

N = 10, L = 10rr

unlearninglearningtt
deeper systems generalize as well…

Summary

Construction of replica theory for a deep perceptron network- random input/output (random constraint satisfaction problem)- teacher-student scenario (statistical inference) with noise“Wetting transition” in the design space with/without RSB
NumN
Numerical simulations of theteacher-student scenario

OutlookGoldt, S., Mézard, M., Krzakala, F., & Zdeborová, L. (2020). PRX, 10(4), 041044.

# Finite width N effect, hidden manifold model: loop corrections…
MNIST N = 784 but De↵ ' 14c.f.
# mismatch of architecture# other activation functions: sigmoid, ReLU,….
# Simulations with “real data”, various algorithms, architectures…