Hopfield/Mixer correspondence
towards a better understanding of MetaFormers architecture design

Toshihiro Ota
CyberAgent, Inc. / RIKEN iTHEMS

December, 2023

Mainly based on 2304.13061 with Masato Taki (Rikkyo Univ./RIKEN iTHEMS)

Introduction

Attention is All You Need

HN is All You Need

iMixer

Biography
Apr. 2016 - Mar. 2021

Osaka Univ., Ph.D. in Physics
AdS/CFT, class S, integrability

Apr. 2021 - July 2021

UTokyo, Math. Sci.
Low-dim. topology, quantum algebra

Aug. 2021 - Nov. 2022

TokyoTech, School of Computing
Machine Learning, Deep LearningDec. 2022 - present

CyberAgent, AI Lab
Machine Learning, Deep Learning

Apr. 2019 - present

RIKEN, iTHEMS

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerToday’s main message:• Hopfield/Mixer correspondence as an approach for

MetaFormes architecture design

Based on the correspondence, we theoretically predict
iMixer: a novel MetaFormer model fromhierarchical Hopfield network [TO-Taki, 2304.13061]

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaSummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerAttention is All You Need
Transformer in our everyday life [Vaswani+ NeurIPS17, Fig. 1]

Large success across nearly all domains
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerAttention is All You Need
An image is worth 16x16 words: Vision Transformer[Dosovitskiy+ ICLR21; Touvron+ ICML21; . . . ]Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You NeediMixer

SummaryAttention is All You Need?
MetaFormers (MLP-Mixer, Conv/Pool/Rand/Identity-Former,. . . ) [Tolstikhin+ NeurIPS21; Melas-Kyriazi 21; Yu+ 22][Yu+ CVPR22, Fig. 1a]

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaIntroduction

Attention is All You Need

HN is All You Need

iMixerSummaryiMixer: invertible, implicit and iterative MLP-Mixer• Derive a new MetaFormer model from

Hopfield/Mixer correspondence• Provide a direction for incorporating implicit NNs• Empirical study supports the validity of our formulation
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaIntroduction

Attention is All You Need

HN is All You NeediMixerContents
1. Introduction
2. Attention is All You Need
– CNN vs Vision Transformer– Attention is All You Need?

3. Hopfield Networks is All You Need
4. iMixer: invertible, implicit and iterative MLP-Mixer
from modern Hopfield network

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You NeedHN is All You Need

iMixerConvolutional Neural Networkhttps://en.wikipedia.org/wiki/Convolutional neural network

respects• Locality• Translation invariance“inductive bias”

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerVision Transformer
An image is worth 16x16 words [Dosovitskiy+ ICLR21, Fig. 1]

Quite less inductive bias than CNN
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You NeediMixer

Vision Transformer

Attention mechanism:

Y = Attn(X) = V > softmax KQ>
Q = WQ X,

K = WK X,

Hopfield/Mixer correspondence for MetaFormers architecture design

V = WV X
Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerAttention is All You Need?
Casted doubt on the role of attention module: MLP-Mixer
Mixer Layer[Tolstikhin+ NeurIPS21, Fig. 1]

Spatial MLP:
Y = W2 σ(W1 X)

Simpler than attention mechanism and yet less inductive bias
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerAttention is All You Need?
MetaFormers [Yu+ CVPR22, Fig. 1a]

Token-mixing block:Y = X + TokenMixer(Norm(X))
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerContents
1. Introduction
2. Attention is All You Need
– CNN vs Vision Transformer– Attention is All You Need?

3. Hopfield Networks is All You Need
– Modern Hopfield networks to the rescue

4. iMixer: invertible, implicit and iterative MLP-Mixer
from modern Hopfield network

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerClassical Hopfield network
A classical associative memory model [Hopfield 82]

Update rule ~s ← sgn(J~s) minimizes the energy function,XXE(~s) = −Jij si sj , J :=ξ~µ ξ~µ> , si ∈ {±1}i6=jµHopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerClassical Hopfield network
A classical associative memory model [Hopfield 82]

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaSummary

Introduction

Attention is All You Need

HN is All You Need

iMixerHopfield Networks is All You Need
Attention = a Hopfield update rule [Ramsauer+ ICLR21, Fig. A.7]
vi ∈ R,ξ = (ξ~1 , . . . , ξ~N )>Update rulevi ←X

ξiµ softmaxµXξµj vjjminimizes an energy functionXX
1X 2ξµi viE({vi }) =vi − logexp
2µi

Hopfield/Mixer correspondence for MetaFormers architecture design!

iToshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixer

Generalized Hopfield network
Unification of energy-based associative memory models[Krotov-Hopfield ICLR21]
Nτv

hdvi (t) X=
ξiµ fµ (h(t)) − vi (t)
dt

µ=1
Nτhv
dhµ (t) X=
ξµi gi (v(t)) − hµ (t)dt

i=1

Activation functions f , g are determined by “Lagrangians”:
fµ (h) =∂Lh (h),∂hµgi (v) =Hopfield/Mixer correspondence for MetaFormers architecture design∂Lv (v)∂viToshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixer

Generalized Hopfield network
Unification of energy-based associative memory models[Krotov-Hopfield ICLR21]
Nτv

hdvi (t) X=
ξiµ fµ (h(t)) − vi (t)
dt

µ=1
Nτhv
dhµ (t) X=
ξµi gi (v(t)) − hµ (t)dt

i=1

Activation functions f , g are determined by “Lagrangians”:fµ (h) =∂Lh (h),∂hµgi (v) =Hopfield/Mixer correspondence for MetaFormers architecture design∂Lv (v)∂viToshihiro Ota

SummaryIntroductionAttention is All You Need

HN is All You NeediMixer

Generalized Hopfield network
The dynamical equations (update rules for the neurons)
Nhdvi (t) Xτv=
ξiµ fµ (h(t)) − vi (t)
dt

µ=1
Nv
dhµ (t) Xτh=ξµi gi (v(t)) − hµ (t)dt

i=1minimize the energy function
E(v, h) =Xivi gi − Lv +Xµhµ fµ − Lh −Xfµ ξµi giµ,i

Lagrangians Lv , Lh define a model
Generate a family of Hopfield networks
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixer

Generalized Hopfield network
The dynamical equations (update rules for the neurons)
Nhdvi (t) Xτv=
ξiµ fµ (h(t)) − vi (t)
dt

µ=1
Nv
dhµ (t) Xτh=ξµi gi (v(t)) − hµ (t)dt

i=1minimize the energy function
E(v, h) =Xivi gi − Lv +Xµhµ fµ − Lh −Xfµ ξµi giµ,iLagrangians Lv , Lh define a model
Generate a family of Hopfield networks
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaSummary

Introduction

Attention is All You Need

HN is All You Need

iMixerAttention as a modern Hopfield network“Model B” in [Krotov-Hopfield ICLR21]
Lv (v) =1X 2vi ,2Lh (h) = log

X

exp(hµ )µi

Integrate out hidden neurons hµ , discretize the ODE, thenXXξµj vj (t)vi (t + 1) =ξiµ softmax
µjXX1X 2E({vi }) =exp
ξµi vivi − log
2µi!

ireproduce [Ramsauer+ ICLR21]
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerSummary

Attention as a modern Hopfield network
Applications along this line:• Immune repertoire classification [Widrich+ NeurIPS20]• Exponential capacity of dense associative memories[Lucibello-Mezard 23]• Learning with partial forgetting in modern Hopfield

networks [TO-Sato-Kawakami-Tanaka-Inoue AISTATS23]• A family of Boltzmann machines from modern Hopfieldnetworks [TO-Karakida NECO23]– Attentional Boltzmann machine is an exactly solvable model

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaIntroductionAttention is All You Need

HN is All You NeediMixerContents
1. Introduction
2. Attention is All You need
– CNN vs Vision Transformer– Attention is All You Need?

3. Hopfield Networks is All You Need
– Modern Hopfield networks to the rescue

4. iMixer: invertible, implicit and iterative MLP-Mixer
from modern Hopfield network

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaSummary

IntroductionAttention is All You Need

HN is All You NeediMixer

Hopfield/Mixer correspondence
MLP-Mixer as Model C of the generalized Hopfield network[Krotov-Hopfield ICLR21; Tang-Kopp 21]

sXLv (v) =(vi − v̄)2 ,Lh (h) =X

φ(hµ )µi

Integrate out hidden neurons hµ , discretize the ODE, thenXX
ξµj LayerNorm(v(t))j vi (t + 1) = vi (t) +ξiµ φ0 µjToken-mixing block of MLP-Mixer [Tolstikhin+ NeurIPS21]
Y = X + W2 σ(W1 LayerNorm(X))
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaSummary

Introduction

Attention is All You Need

HN is All You NeediMixer

Hopfield/Mixer correspondence
MLP-Mixer as Model C of the generalized Hopfield network[Krotov-Hopfield ICLR21; Tang-Kopp 21]

sXLv (v) =(vi − v̄)2 ,Lh (h) =X

φ(hµ )µi

Integrate out hidden neurons hµ , discretize the ODE, thenXXξµj LayerNorm(v(t))j vi (t + 1) = vi (t) +ξiµ φ0 µj

Token-mixing block of MLP-Mixer [Tolstikhin+ NeurIPS21]
Y = X + W2 σ(W1 LayerNorm(X))
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerSummary

The generalized Hopfield network can reproduce many of known
NN models. So far so good
A natural question:
The generalized Hopfield network can even predicta novel MetaFormer architecture?

Model-C Hopfield network
Model-C hierarchical extension

Hopfield/Mixer correspondence for MetaFormers architecture design

MLP-Mixer???

Toshihiro OtaIntroduction

Attention is All You Need

HN is All You Need

iMixerSummary

The generalized Hopfield network can reproduce many of known
NN models. So far so good
A natural question:
The generalized Hopfield network can even predicta novel MetaFormer architecture?

Model-C Hopfield network
Model-C hierarchical extension

Hopfield/Mixer correspondence for MetaFormers architecture design

MLP-Mixer???

Toshihiro OtaIntroduction

Attention is All You Need

HN is All You Need

iMixerSummary

iMixerHierarchical extension

Lv (v) =sXi(vi − v̄)2 ,Lx (x) =X

φx (xα ),αLh (h) =Xφh (hµ )µv(t + 1) = v(t) + ξ (v,x) φ0x (1 − F)−1 ξ (x,v) LayerNorm(v(t))
F = (ξ (x,h) φ0h ) ◦ (ξ (h,x) φ0x )Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaIntroduction

Attention is All You Need

HN is All You Need

iMixeriMixerInverted ResNet is an example of implicit NNs[Behrmann+ ICML19; Bai+ NeurIPS19; El Ghaoui+ 19]

Fixed-point iteration method enables us to easily implement &train the model
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaSummaryIntroduction

Attention is All You Need

HN is All You Need

iMixeriMixer
The iMLP module looks somewhat unconventional from CV
viewpoint. Experimental evaluation?
Top-1 accuracy (%), trained on CIFAR-10 from scratch
Model
Mixer (baseline)
iMixer (ours)

SmallBaseLarge

88.08 ±0.51
88.56 ±0.30

89.03 ±0.24
89.07 ±0.33

86.67 ±0.30
87.48 ±0.40

Top-1 accuracy (%) for other datasets, trained from scratch for Small models
Model

CIFAR-100

Food-101

ImageNet-1k

Mixer-S
iMixer-S

68.13 ±0.46
68.26 ±0.30

76.11 ±0.32
76.08 ±0.20

73.91
74.10

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerOutlook
Lots of further directions like
• More hidden layers and different Lagrangians• Practical applications for real computer vision tasks• Boltzmann machine counterparts of hierarchical Hopfieldnetworks• More direct relation with associative memory model (inprogress with Taki and Karakida)Any discussions/comments are very welcome

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

SummaryIntroduction

Attention is All You Need

HN is All You Need

iMixerSummary

iMixer: invertible, implicit and iterative MLP-Mixer• Derive a new MetaFormer model from

Hopfield/Mixer correspondence• Provide a direction for incorporating implicit NNs• Empirical study supports the validity of our formulation
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaIntroduction

Attention is All You Need

HN is All You NeediMixer

Contents
1. Introduction
2. Attention is All You need
– CNN vs Vision Transformer– Attention is All You Need?

3. Hopfield Networks is All You Need
– Modern Hopfield networks to the rescue

4. iMixer: invertible, implicit and iterative MLP-Mixer
from modern Hopfield network

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

Summary

Backup

BackupHopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

Backup

LwPF
Learning with partial forgetting in modern Hopfield networks[TO-Sato-Kawakami-Tanaka-Inoue AISTATS23]• Propose learning with partial forgetting (LwPF) mechanism• Derive the expression for partially forgetting attention• Demonstrate the effectiveness of LwPF in diverse domains

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

Backup

AttnBM
Attention in a family of Boltzmann machines emerging frommodern Hopfield networks [TO-Karakida NECO23]

Image reconstruction

Receptive fields• Propose a family of Boltzmann machines from thegeneralized Hopfield network• Investigate the basic properties of attentional BM and verify

its integrability and trainability
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaBackup

Generalized Hopfield network
Model A: Dense associative memory models [Hopfield 82;
Krotov-Hopfield NeurIPS16; Demircigil +17]

Lv (v) =X
i|vi |,Lh (h) =X

F(hµ )µIntegrate out hidden neurons hµ , discretize the ODE, thenXXvi (t + 1) =ξiµ F0 
ξµj sgn vj (t) µj!
E({vi }) = −XµFX

ξµi sgn(vi )iHopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

BackupGeneralized Hopfield network
Model A: Dense associative memory models [Hopfield 82;
Krotov-Hopfield NeurIPS16; Demircigil+ 17]

vi (t + 1) =XXξiµ F0 
ξµj sgn vj (t) µj!
E({vi }) = −XµFX

ξµi sgn(vi )i• F(x) = x2 : the classical Hopfield network, sgn(vi (t)) =: si (t)• F(x) = xn : the network can store O(Nvn−1 ) memories• F(x) = ex : exponential storage capacityHopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

Backupx1 (t + 1)

iMixer: a general formulation
H2

Fully-connected

Fully-connected

F2

Fully-connected
3F
One of the most general formulations of iMixer
fromG
L-layer hierarchical Hopfield network:
MLP block

MLP block

Fully-connected

F4

F5

MLP block

MLP block

1

Fully-connected

x1 (t + 1)

Fully-connected

Fully-connected

x1 (t)
H2

Fully-connectedblock

Fully-connected

Fully-connected

F2block

G
Fully-connected

F3

Fully-connected

block

F4

Fully-connected

block

F5

Fully-connected

Fully-connected

L : odd… …

1

Fully-connected

Fully-connected

Fully-connected

Fully-connected
Fully-connected

L : even

x1 (t)

x1 (t + 1) = x1 (t) + iMLPs(x1 (t))

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota

BackupiMixer: experimental details
Hyperparameters commonly used for the vanilla Mixer and iMixer for fair comparison.
Training configuration
optimizer
training epochs
batch sizebase learning rate
weight decay
optimizer 
optimizer momentum
learning rate schedule
lower learning rate boundwarmup epochs
warmup schedule
warmup learning rate
cooldown epochs
crop ratio
RandAugment
mixup αcutmix α
random erasinglabel smoothing
stochastic depth

Small/Base/Large
AdamW
300
512/256/64
5e-4/2.5e-4/6.25e-5
0.05
1e-8
β1 = 0.9, β2 = 0.999
cosine decay
1e-620linear1e-6
100.875(9, 0.5)
0.8
1.0
0.25
0.1
0.1/0.2/0.3

Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro OtaBackup

iMixer: experimental detailsHyperparameter search for hr and n in iMixer-S, trained on CIFAR-10 from scratch
hrn=1n=2

n=40.250.5
1288.26 ±0.28
88.32 ±0.39
88.36 ±0.31
88.54 ±0.34

88.22 ±0.33
88.21 ±0.45
88.32 ±0.32
88.56 ±0.30

88.29 ±0.37
88.22 ±0.43
88.32 ±0.32
88.46 ±0.26

Convergence rate of L2 -norm (left) and cosine similarity (right)
between two successive feature vectors in fixed-point iteration in iMLP-0
Hopfield/Mixer correspondence for MetaFormers architecture design

Toshihiro Ota