Large Associative Memory Problem in Neurobiology and Machine Learning
Authors: Dmitry Krotov
John J. Hopfield
Venue: ICLR 2021 Poster (2021)
Keywords: associative memory
Hopfield networks
modern Hopfield networks
neuroscience

Abstract: Dense Associative Memories or modern Hopfield networks permit storage and reliable  retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons.  We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in "Hopfield Networks is All You Need" paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.

Published as a conference paper at ICLR 2021
LARGE ASSOCIATIVE MEMORY PROBLEM IN NEURO- BIOLOGY AND MACHINE LEARNING
Dmitry Krotov
John Hopfield
MIT-IBM Watson Al Lab IBM Research
Princeton Neuroscience Institute
Princeton University
k krotovê i.m. com
ABSTRACT
Dense Associative Memories modern Hopfield networks permit storage and re- liable retrieval of an exponentially large (in the dimension of feature space) num- ber of memories. At the same time, their naive implementation is non-biological. since it seemingly requires the existence of many many-body synaptic junctions be- tween the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason proposed microscopic theory is valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one recover many of the models that were previ- ously discussed in the literature, e.g. the model presented in "Hopfield Networks is All You Need" paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.
INTRODUCTION
Associative memory is defined in psychology as the ability to remember (link) many sets, called memories, of unrelated items. Prompted by a large enough subset of items taken from one memory, an animal or computer with an associative memory can retrieve the rest of the items belonging to that memory. The diverse human cognitive abilities which involve making appropriate responses to stimulus patterns often be understood the operation of associative memory, with the "memories" often being distillations and consolidations of multiple experiences rather than merely corresponding to single event.
The intuitive idea associative memory can be described using a "feature space". In a mathemat- ical model abstracted from neurobiology, the presence (or absence) of each particular feature is denoted by the activity (or lack of activity) of a model neuron v; due to being directly driven by a feature signal. If there N, possible features, there be only most N2 distinct connections (synapses) in neural circuit involving only these neurons. Typical cortical synapses reliable, and can store only a few bits of information]] The description of a particular memory re- quires roughly Ny bits of information. Such system can therefore store at most ~ N₁ unrelated memories. Artificial neural network models of associative memory (based on attractor dynamics of feature neurons and understood through energy function) exhibit this limitation even with precise synapses, with limits of memory storage to less than ~ 0.14N, memories (Hopfield) 1982).
not highly
For instance, a recent study (Bromer et al. 2018 reports the information content of individual synapses ranging between 2.7 and 4.7 bits. based on electron microscopy imaging, see also (Bartol Jr et al. 2015). These numbers refer to the structural accuracy of synapses. There is also electrical and chemical currents induced by the biophysical details vesicle release and neurotransmitter binding. The unreliability of the fusion of pre-synaptic vesicles (containing neurotransmitter) with the pre-synaptic neuron membrane is the dominant source of trial- to-l-to-iie to-i-al synaptic current variation (Allen & Stevens] 1994). This noise decreases the electrical information capacity of individual synapses from the maximal value that the synaptic structure would otherwise provide.
in synaptic

---

Published as a conference paper at ICLR 2021
E =
Tipaas
==
Σ
pl E (uaa) =- --Er To
Y12
Ti3
Ta T25
ds
13
biological
X non-biological
Figure Two binary networks consisting of three neurons 01, 02,02,03 {+1}. On the left is the classical Hopfield network (Hopfield 1982) with the matrix Tij E2 Eushuj being the outer product of memory vectors (see section 2 for the definitions of notations). In this case the matrix Tiy interpreted
matrix of synaptic connections between cells and j. On the right is a Dense Associative Memory network of (Krotov & Hopfield (016) with cubic interaction term F(x) = x². this case the corresponding tensor Tisk Σn كuكEكōكu has three indices, thus cannot be interpreted biological synapse, which can only connect two cells.
Situations arise in which the number y is small and the desired number of memories far exceeds Ny, see some examples from biological and AI systems in Section 4 In these situations the associative memory model of (Hophilld 1982) would be insufficient, since it would not be able to memorize the required number of patterns. At the same time, models of associative memory with large storage capacity considered in paper, can easily solve these problems.
The starting point of this paper is a machine learning approach to associative memory based on an energy function and attractor dynamies in the space of N3 variables, called Dense Associative Memory (Krotov& & Hopfield 2016). This idea has been shown to dramatically increase the memory storage capacity of the corresponding neural network (Krotov & Hopfield [016 Demircigil al., 2017) and was proposed to be useful for increasing robustness of neural networks to adversarial attacks (rotov& & Hopfield 2018). Recently, an extension of this idea to continuous variables, called modern Hopfield network, demonstrated remarkably successful results on the immune repertoire classification (Widrich et al., 2020), and provided valuable insights into the properties of attention heads in Transformer architectures (Ramsauer et al. 2020).
Dense Associative Memories or modern Hopfield networks, however, cannot describe biological neural networks in terms of true microscopic degrees of freedom, since they contain many-body interaction terms in equations describing their dynamics and the corresponding energy functions. To illustrate this point consider two networks: conventional Hopfield network (Hopfield 1982) and a Dense Associative Memory with cubic interaction term in the energy function (see Fig. D). In the conventional network the dynamics is encoded in the matrix Tej, which represents the strengths of the synaptic connections between feature neurons and j. Thus, this network is manifestly describable in terms of only two-body synapses, which is approximately true for many biological synapses. In contrast, Dense Associative Memory network with cubic energy function naively requires the synaptic connections to be tensors Teyk with three indices, which are harder, although not impossible, to implement biologically. Many-body synapses become even more problematic in situations when the interaction term is described by a more complicated function than a simple power (in this case the Taylor expansion of that function would generate a series of terms with increasing powers).
Many-body synapses typically appear in situations when one starts with microscopic theory de- scribed by only two-body synapses and integrates out some the degrees of freedom (hidden neu- rons). The argument described above based counting the information stored in synapses in con- junction with the fact that modern Hopfield nets and Dense Associative Memories can have a huge storage capacity hints the same solution. The reason why these networks have a storage capacity much greater than N₂ is because they do not describe the dynamics of only Ny neurons, but rather involve additional neurons and synapses.
Thus, there remains a thcoretical question: what does this hidden circuitry look like? Is it possible to introduce set of hidden neurons with appropriately chosen interaction terms and activation
2

---

Published as a conference paper at ICLR 2021
memory neurons hpp
O
Spi
Siju
biological
O feature neurons
Figure 2: An example of a continuous network with Ny 5 feature neurons and N₂, 11 complex memory (hidden) neurons with symmetric synaptic connections between them.
functions so that the resulting theory has both large memory storage capacity (significantly bigger than f), and, at the same time, is manifestly describable in terms on only two-body synapses?
The main contributions of this current paper are the following. First, we extend the model of (Krotov & Hopfield 2016) to continuous state variables and continuous time, so that the state of the network is described by a system of non-linear differential equations. Second, we couple an additional set of Nr. "complex neurons" or "memory neurons" or hidden neurons the Ny feature neurons. When the synaptic couplings and neuron activation functions are appropriately chosen, this dynamical system in Ny + N₂ variables has energy function describing its dynamics. The minima (stable points) of this dynamics are at the same locations in the N, dimensional feature subspace as the minima in the corresponding Dense Associative Memory system. Importantly, the resulting dynamical system has a mathematical structure of conventional recurrent neural network, in which the neurons interact only in pairs through two-body matrix of synaptic connections. We study three limiting cases of this new theory, which we call models A, B. and C. In one limit (model A) it reduces to Dense Associative Memory model of (Krotov Hopfield] 2016) or (Demircigil al. 2017) depending on the choice of the activation function. In another limit (model B) our model reduces to the network of (Ramsauer et al. 2020). Finally, we present third limit (model C) which call Spherical Memory model. To the best of our knowledge this model has not been studied in the literature. However, it has a high degree of symmetry and for this reason might be useful for future explorations of various models of large associative memory and recurrent neural networks in machine learning.
For the purposes of this paper we defined "biological plausiblity" as the absence of many- body synapses. It important to note that there other aspects in which our model described by equations (I) below is biologically implausible. For instance, it assumes that the strengths of two physically different synapses μ - i and i + l are equal. This assumption is necessary for the existence of the energy function, which makes easy to prove the convergence a fixed point. It can be relaxed in equations (I). which makes them even more biological, but, at the same time, more difficult to analyse.
MATHEMATICAL FORMULATION
In this section, we present simple mathematical model in continuous time, which, on one hand, permits the storage of a huge number of patterns in the artificial neural network, and, at the same time, involves only pairwise interactions between the neurons through synaptic junctions. Thus, this system has the useful associative memory properties of the AI system, while maintaining conven- tional neural network dynamics and thus degree of biological plausibility.
The spikes of action potentials in a pre-synaptic cell produce input currents into postsynaptic neuron. As result of single spike the pre-synaptic cell the current the post-synaptic neuron rises instantaneously and then falls off exponentially with a time constant т. In the following the currents of the feature neurons are denoted by v; (which are enumerated by the latin indices), and the currents of the complex memory neurons are denoted by h, (h stands for hidden neurons, which are enumerated by the greek indices). simple cartoon of the network that we discuss is shown in Fig There synaptic connections among the feature neurons the memory neurons. A matrix &, denotes the strength of synapses from a feature neuron to the memory neuron p. The synapses assumed to be symmetric, so that the same value Eip Epé characterizes different physical
3

---

Published as a conference paper at ICLR 2021
synapse from the memory neuron jl to the feature neuron i. The outputs of the memory neurons and the feature neurons are denoted by fu and 9t. which are non-linear functions of the corresponding currents. In some situations (model A) these outputs
be interpreted as activation functions for
the corresponding neurons, so that f J(hp) and 9i = g(1)) with some non-linear functions f(x and s(:). In other cases (models B and C) these outputs involve contrastive normalization, e.g. softmax, and can depend on the currents of all the neurons in that layer. In these cases fu = f((h,)) and gi { For the most part of this paper one can think about them as firing rates of the corresponding neurons. In some limiting cases, however, the function g(;)) will have both positive and negative signs. Then should be interpreted as the input current from a pre-synaptic neuron. The functions / (h,) and g(v:) are the only nonlinearities that appear in our model. Finally, the time constants for the two groups of neurons are denoted by Ty and Th- With these notations our model can be written as
NA. Ssaf - V/3 + +
du ff
Σ
dt
(1)
N₁ Euils 11
dh
Th de
where I; denotes the input current into the feature neurons.
The connectivity of our network has the structure of bipartite graph, that the connections exist between two groups of neurons, but not within each of the two groups. This design of neural network is inspired by the class of models called Restricted Boltzmann Machines (RBM) (Smolen- sky 1986). There is a body of literature studying thermodynamic properties of these systems and leariig rules for the synaptic weights. In contrast, the goal of our work is to write down general dynamical system and an energy function so that the network has useful properties of associative memories with a large memory storage capacity, is describable only in terms of manifestly two-body synapses, and is sufficiently general so that it can be reduced to various models of this class previ- ously discussed in the literature. We also note that although we use the notation vú (v stands for visible neurons), commonly used in the RBM literature. it is more appropriate to think about as higher level features. For example the input to our network can be a latent representation produced by convolutional neural network 2018) rather than raw input data. Additionally, our general formulation makes it possible to use a much broader class of activation functions (e.g. involving contrastive spherical normalization) than those typically used in the RBM literature. Also, the relationship between Dense Associative Memories and RBMs has been previously studied in (Barra et al.) 2018 Agliari & De Marzo 2020). We also note that a Hopfield network with exponential capacity was studied in (Chaudhuri & Ficte 2019, but their construction requires specifically engineered memory vectors and cannot be applied to general arbitrary memory vectors.
latent representation of a BERT-like system (Devlin et al.,
Mathematically, equations II] describe temporal evolution of two groups of neurons. For each neuron its temporal updates are determined by the inputs from other neurons and its own state (the decay term on the right hand side of the dynamical equations). For this reason, an energy function for this system is expected to be represented as a sum of three terms: two terms describing the neurons in each specific group, and the interaction term between the two groups of ncurons. We have chosen the specific mathematical form of these three terms that the energy function decreases the dynamical trajectory. With these choices the energy function for the network T
can be written as
Ny (u
NR. hufr
E(t
1)g
Lu]
Ln]
E FuEwigh 14,3
Here we introduced two Lagrangian functions L. ((vx)) and Ln ((hn,)) for the feature and the hidden neurons. They are defined through the following equations, so that derivatives of the Lagrangian functions correspond the outputs of neurons
f. = aL,
aL and 91 Dv.
Th
With these notations expressions in the square brackets in have familiar from classical mechan- ics structure of the Legendre transform between a Lagrangian and an energy function. By taking time derivative of the energy and using dynamical equations (T) one show (see Appendix A for
4

---

Published as a conference paper at ICLR 2021
details) that the energy monotonically decreases the dynamical trajectory
N₁ du; 8 L. duz No E
dE(l)
dhy 8 dhy dt Oh dt
Σ dt dv,ōv, dt
(4)
dt
p,01
The last inequality sign holds provided that the Hessian matrices of the Lagrangian functions are positive semi-definite.
In addition to decrease of the energy function on the dynamical trajectory it is important to check that for a specific choice of the activation functions (or Lagrangian functions) the corresponding energy bounded from below. This can be achieved for example by using bounded activation function for the feature neurons (v)), c.g. hyperbolic tangent or a sigmoid. Provided that the energy is bounded, the dynamics of the neural network will eventually reach a fixed point, which corresponds to one of the local minima of the energy functior 2
The proposed energy function has three terms in it: the first term depends only the feature rons, the second term depends only on the hidden neurons, and the third term is the "interaction" term between the groups of neurons. Note, that this third term is manifestly describable by two- body synapses a function of the activity of the feature neurons is coupled to another function of the activity of the memory neurons, and the strength of this coupling characterized by the parameters Eus. The absence of many-body interaction terms in the energy function results in the conventional structure (with unconventional activation functions) collects outputs of other neurons, weights them with coefficients & and generates its own output. Thus, the network described by equations I is biologically plausible according to our definition (see Introduction).
the dynamical equations (T). Each neuron
Lastly, note that the memory patterns Eμi of our network (T) can be interpreted as the strengths of the synapses connecting feature and memory neurons. This interpretation is different from the conventional interpretation, in which the strengths of the synapses is determined by matrices Ti En SuōSuj (see Fig. D. which are outer products of the memory vectors (or higher order generalizations of the outer products).
3 EFFECTIVE THEORY FOR FEATURE NEURONS
In this section we start with the general theory proposed in the previous section and integrate out hidden neurons. We show that depending on the choice of the activation functions this general theory reduces to some of the models of associative memory previously studied in the literature, such as classical Hopfield networks, Dense Associative Memories, and modern Hopfield networks. The update rule in the latter case has the same mathematical structure as the dot-product attention [Bahdanau et al. 2014) and is also used in Transformer networks (Vaswani et al. 2017
3.1 MODEL. A. DENSE ASSOCIATIVE MEMORY LIMIT.
Consider the situation when the dynamics of memory neurons h2 is fast. Mathematically this corre- sponds to the limit Th + O. In this case the second equation in (T) equilibrates quickly, and can be solved as
hp
Enigi
(5)
Additionally, assume that the Lagrangian functions for the feature and the memory neurons are additive for individual neurons
La Σ F(hy): and L, -.=Io(w)
(6)
where F(x) and G(x) some non-linear functions. In this limit we set G(x) |z]] Since, the outputs of the feature neurons are derivatives of the Lagrangian (3), they are given by the sign
"There is also border case possibility that the dynamics cycles without decreasing the energy (limit cycle), but this requires that the Hessian matrix in (4) has a zero mode everywhere along the trajectory. This border case possibility should be checked for specific choice of the activation functions.
5

---

Published as a conference paper at ICLR 2021
functions of their currents, which gives set of binary variables that are denoted by d;
0, ー - - ((v)
Sign [e]]
On
Since (i;)) |u;| the only term that survives in the first square bracket in equation ( is the one proportional to the input current 4, The first term in the second bracket equation (2 cancels the interaction term because of the steady state condition (5). Thus, in this limit the energy function reduces to
N
E(L)
Loo Σ P ( Spios)
If there are no input currents 4, = 0 this is exactly the energy function for Dense Associative Memory from (Krotov & Hopfield 2016). If F(x) x" power function, the network can store N mem N memories, if F(x) exp(x) the network has exponential storage capacity Demircigil al. 2017). If power = 2 this model further reduces the classical Hopfield network
(Hopfield 1982).
It is important to emphasize that the capacity estimates given above express the maximal number of memories that the associative memory can store given the dimensions of the input, but assuming no limits the number of hidden neurons. In all the models considered in this work this capacity is also bounded by the number of those hidden neurons so that Nmem I Nu. With this constraint the capacity of model with power function F(x) x" should be written as
(9) In many practical applications (see examples in Section 4 the number of hidden neurons can be assumed to be larger than the bound defined by the dimensionality of the input space Ny. It is for this class of problems that Dense Associative Memories or modern Hopfield networks offer powerful solution to the capacity limitation compared to the standard models of associative memory (Hopfield) 1982 1984).
Nmen min ,,)
Lastly, for the class of additive models o which we call models A, the equation for the temporal evolution of the energy function reduces to
dE(() di
N dv; di
No.
dhy di
g(u))
S(h,))
00
(10)
1=1
Thus, the condition that the Hessians are positive definite is equivalent to the condition that the activation functions g(v;) and fhy,) are monotonically increasing.
Additionally, in Appendix B. we show how standard continuous Hopfield networks (Hopfield 1984) can be derived as a limiting case of the general theory (
3.2 MODEL B. MODERN HOPFIELD NETWORKS LIMIT AND ATTENTION MECHANISM.
Models B are defined as models having contrastive normalization in the hidden layer. Specifically we are interested in
La = log ( and L, -
(11)
so that Tog still additive, but Tup not. Using the general definition of the activation functions one obtains
aLn softmax(hu)
ch
fa Th
Eeh
(12)
aL.
9: Ov:
U:
Similarly to the previous case, consider the limit Th - O, so that equation (5) is satisfied. this limit the energy function (2) reduces to (currents 4; are assumed to be zero)
N₂
E
U
log Σ exp Σ Euet;)
(13)

---

Published as a conference paper at ICLR 2021
This is exactly the energy function studied in (Ramsauer at. 2020) up to additive constants (inverse temperature 3 was assumed to be equal to one in this derivation). Notice that we used the notations from (Krotov & Hopfield, 2016), which are different from the notations of (Ramsauer al. 2020). In the latter paper the state vector v; is denoted by & and the memory matrix Eyi is denoted by the matrix X
Making substitutions (12) in the first equation of [T. using steady state condition ( and setting input current I; = results in the following effective equations for the feature neurons, when the memory neurons integrated out
dv; Tf di
Sausottmax
(14)
This is continuous time counterpart of the update rule of (Ramsauer al. 2020). Writing it in finite differences gives
Nn.
N Enou! - 2.51
(++1)
vi
Sopsoftaax
(15)
Tf
which for di Ty reduces to
NJ Swyaf()
1++1)
En softmax
(16)
This is exactly the update rule derived in (Ramsauer et al., 2020 which, if applied once, is equiv- alent to the familiar dot-product attention (Bahdanau et al. 2014) and is also used in Transformer networks (Vaswani et al. 2017).
The derivation this result in (Ramsauer et al. 2020 begins with the energy function for a Dense Associative Memory model with exponential interactions F(z) exp(z). Then it is proposed to take a logarithm that energy (with a minus sign) and add a quadratic term in the state vector v to ensure that it remains finite and the energy is bounded from below. While this is a possible logic, it requires a heuristic step taking the logarithm, and makes the connection with Dense Associative Memories less transparent. In contrast, our derivation follows from the general principles specified by equations 1 for the specifically chosen Lagrangians.
also important note, that the Hessian matrix for the hidden neurons has zero mode (zero cigenvalue) for this limit of our model.
3.3 MODEL C. SPHERICAL MEMORY.
Models C are defined as having spherical normalization in the feature layer. We are not aware of discussion of this class of associative memory models in the literature. Specifically,
LA Σ F(hm), and L, = Σ
(17)
1
so that L, is additive, but T.,, is not. Using the general definition of the activation functions 3 obtains
one
f, F F'(hr)
OL.
1/
(18)
9i Dv:
V ΣS m
Equations 1 for model C are given by (I; is assumed to be zero)
Tj
dug dt
=
Shw)( av,
(19)
dhg Th
Σ
EusHi - h1
dt
7

---

Published as a conference paper at ICLR 2021
Notice, that since the Hessian matrix for the feature neurons has mode proportional to v, in this model.
L My dv,Oo;
dig Σ vray] so that M 0,
(20)
Σ (EvR)
3/2
we can write an arbitrary coefficient a, which can be equal to zero, in front of the decay term for the feature neurons. Taking the limit Th - and excluding hy gives the effective energy
E(I) Σ Enr
12
(21)
and the corresponding effective dynamical equations
du E Σ Eus 41 VEx
= EEnu
C02
(22)
dt
J
is also important to notice that the activation function Ji that appears in equation 18 implements canonical computation of divisive normalization widely studied in neuroscience (Carandini & Heeger 2012).. Divisive normalization has also been shown to be beneficial in deep CNNs and RNNs trained image classification and language modelling tasks (Ren al. 2016).
A FEW EXAMPLES OF LARGE ASSOCIATIVE MEMORY PROBLEMS
In this section provide some examples of problems in Al and biology which may benefit from thinking about them through the lens of associative memory.
Pattern memorization. Consider small gray scale image 64 64 pixels. If one treats the intensity of each pixel as an input to a feature neuron the standard associative memory (Hopfield, 1982) would be able only memorize approximately 0.14 4096 x 573 distinct patters. Yet, the number of all possible patterns of this size that one can imagine is far bigger. For instance, Kuzushiji- Kanjii dataset (Clanuwat al. 2018) includes over 140,000 characters representing 3832 classes with most of the characters recognizable by humans. A well educated Japanese person can recognize about 3000-5000 character classes, which means that those classes are represented in his/her memory. In addition, for many characters person would be able to complete it if only portion of that character shown. Moreover, possible patterns of 64 x pixels are not necessarily Kanji characters, also include digits, smileys, emojis, etc. Thus, the overall number of patterns that one might want to memorize is even bigger.
In the problem of immune repertoire classification, considered in (Widrich et al. 2020). the num- ber of immune repertoire sequences (number of memories the modern Hopfield network) is N >> 10000, while the size of the sequence embedding dimension d, 32, or Ny 32 using the notations of this current paper. The ability to solve this problem requires the associative memory used in the aforementioned paper have storage capacity much larger than the dimensionality of the feature space.
Cortical-hippocampal system. The hippocampus has long been hypothesised to be responsible for formation and retrieval of associative memories, see for example (Rolls) 2018 Treves & Rolls 1994). Damage the hippocampus results in deficiencies in learning about places and memory recognition visual tasks. For instance (Parkinson et al. 1988) reports deficiencies in object-memory tasks, which require memory of an object and the place where that object was seen. One candidate for associative memory network in the hippocampus is the CA3 area, which consists of a large population of pyramidal neurons, approximately 3- 10 the rat brain (Amaral & Witter 1989). and 2.3-10% (Seress] 1988) in human, in conjunction with an inhibitory network that keeps the firing rates under control. There a substantial recurrent connectivity among the pyramidal neurons (Rolls) 2018), which is necessary for associative memory network. There are also several classes of responses of those neurons in behaving animals, one class being place cells (OeKee & Dostrovsky, 1971). In addition to place cells (Ferguson et al., 2011) report existence of cells in the hippocampus that do not respond in experiments designed to drive place cells, but presumably are useful for other tasks. One possible way of connecting the mathematical model proposed in this paper with
8

---

Published as a conference paper at ICLR 2021
the existing anatomical network in the brain is to assume that some of the pyramidal cells in CA3 correspond to the feature neurons in our model, while the remaining pyramidal cells are the memory neurons. For example, place cells are believed to emerge
result of aggregating imputs from the grid cells and environmental features, e.g. landmark objects, environment boundaries, visual and olfactory cues, etc., (Moser et al., 2013). Thus, it is tempting to think about them as memory neurons (which aggregate information from feature neurons to form a stable memory) in the proposed model.
Another area of the hippocampus potentially related to the mathematical model described in this paper is the area CAI, which, in addition to receiving inputs from CA3, also receives inputs directly from the entorhinal cortex, and projects back it. In this interpretation pyramidal cells of the CA1 would be interpreted as the memory neurons in our mathematical model, while the cells in the layer III of the entorhinal cortex would be the feature neurons. The feedhack projections from CA1 go primarily to layer V of the entorhinal cortex (Rolls) 2018), but there are also projections to layers and III (Witter et al., 2017). While is possible to connect the proposed mathematical model of Dense Associative Memory with existing networks in the hippocampus, it is important to emphasize that the hippocampus is involved in many tasks, for example imagining the future (Hassabis et al.) 2007]. and not only in retrieving the memories about the past. For this reason it is difficult at present to separate the network motifs responsible for memory retrievals from the circuitry required for other functions.
Human color vision has three dimensions so that every color sensation can be achieved by mix- ing three primary lights (Mollon 2003). From the neuron's perspective every color is detected by three kinds of cone photoreceptors (Ny = 3) in the retina, so that the degree of excitation of each photoreceptor described by a continuous number. Most people know many colors with names for them (e.g. red, orange, yellow, green, blue, indigo, violet, pink, lavender, copper, gold, etc.), descriptions for others. e.g. "the color of the sky". Experimentally. humans can distinguish about 10th different colors (Masaoka et al. 2013), although may not be able to "memorize" all of them. See also (Meister) 2015) for the discussion of this problem. Thus, if one thinks about this system as an associative memory for color discrimination, the model of (Hopfield) 1982) and its extensions with O(N,) y) storage capacity would be inadequate since they can only "remember" a few colors. It is important to emphasize that the memories of the colors are stored in higher areas of the brain, while the color sensation is conveyed to the brain through the cone cells in the retina. Thus, in this example there are many intermediate neurons and synapses between in feature neurons and memory neurons. For this reason it is only appropriate to think about this example as a direct associative memory if all these intermediate neurons and synapses are integrated out from this system.
5 DISCUSSION AND CONCLUSIONS
We have proposed a general dynamical system and an energy function that has a large memory storage capacity, and, at the same time, is manifestly deseribable in terms of two-body synaptic connections. From the perspective of neuroscience it suggests that Dense Associative Memory models are not just mathematical tools useful in AI, but have degree of biological plausibility similar to that of the conventional continuous Hopfield networks (Hopficld 1984). Compared to the latter, these models have a greater degree of psychological plausibility, since they can store a much larger number of memories, which is necessary to explain memory-basee based animal behavior.
We want to emphasize that the increase in the memory storage capacity that is achieved by modern Hopfield networks is a result of unfolding the effective theory and addition of (hidden) neurons. By adding these extra neurons we have also added synapses. Coming back to the information counting argument that we presented in the introduction, the reason why these unfolded models have larger storage capacity than the conventional Hopfield networks with the same number of input neurons because they have more synapses, but each of those synapses has the same information capacity as in the conventional case.
From the perspective of Al research our paper provides a conceptually grounded derivation of var- ious associative memory models discussed in the literature, and relationships between them. We hope that the more general formulation, presented in this work, will assist in the development of new models of this class that could be used as building components of new recurrent neural network architectures.

---

Published as a conference paper at ICLR 2021
REFERENCES
Elena Agliari and Giordano De Marzo. Tolerance versus synaptic noise in dense associative memo- ries. arXiv preprint wrXiv:2077.08849 2020.
Christina Allen and Charles Stevens. An evaluation of causes for unreliability of synaptic trans-
mission. Proceedings of the National Academy of Sciences, 10300-13300- 1994.
David G Amaral and Menno P Witter. The three-dimensional organization of the hippocampal formation: a review of anatomical data. Neuroscience, 31 313):571-551, 1989.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint orXiv:/409.04: 2014.
Adriano Barra, Matteo Beccaria, and Alherto Fachechi. A mechanical approach handle generalized hopfield neural networks. Neural Networks, 106:205-222, 2018.
Thomas M Bartol Jr, Cailey Bromer, Justin Kinney, Michael A Chirillo. Jennifer N Bourne, Kris- M Harris, and Terrence Sejnowski. Nanoconnectomic upper bound on the variability of synaptic plasticity. Elife, 1:770778, 2015.
Cailey Bromer, Thomas M Bartol, Jared B Bowden, Dusten D Hubbard, Dakota C Hanka, Paola V Gonzalez, Masaaki Kuwajima, John M Mendenhall, Patrick H Parker, Wickliffe C Abraham, et al. Long-tere term potentiation expands information content of hippocampal dentate gyrus synapses. Proceedings of the National Academy of Sciences, (15(10):E2410 E2418, 2018.
Matteo Carandini and David Heeger. Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1):51-62, 2012.
Rishidev Chaudhuri and Ila Fiete. Bipartite expander hopfield networks as self-decoding high- capacity error correcting codes. Advances in neural information processing systems, 32, 2019.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint 1rXiv:1812.01178,, 2018.
Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168(2):288-299, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 180.0008, 2018.
John E Ferguson, Jadin C Jackson, and David Redish. An inside look hippocampal silent cells. Neuron, 70(1):3-5, 2011.
Demis Hassabis, Dharshan Kumaran, Seralynne D Vann, and Eleanor A Maguire. Patients with hippocampal amnesia cannot imagine new experiences. Proceedings of the National Academy of Sciences, 104(5):1226-1331 2007.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2555 25588, 1982.
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10)33088-309, 1984.
Dmitry Krotov and John Hopfield. Dense associative memory is robust to adversarial inputs. Neural computation, 30(12):3151
2018
Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. In Advances in neural information processing systems, pp. 1172-1180, 2016.
Kenichiro Masaoka, Roy S Berns, Mark D Fairchild, and Farhad Moghareh Abed. Number of discernible object colors is conundrum. JOSA A, 2002))26442277,222 2013.
Markus Meister. On the dimensionality of odor space. Elife, 4:c07865, 2015.
10

---

Published as a conference paper at ICLR 2021
John D Mollon. The origins of modern color science. The science of color, 2:133,,0003.
May-Britt Moser, David C Rowland, and Edvard Moser. Place cells, grid cells, and memory. Cold Spring Harbor perspectives in biology, 7(2):a021808 2015.
John O' O'Keefe and Jonathan Dostrovsky. The hippocampus as spatial map: Preliminary evidence from unit activity in the freely-moving rat. Brain research, 1971.
JK Parkinson, EA Murray, and Mishkin. A selective mnemonic role for the hippocampus in monkeys: memory for the location of objects. Journal of Neuroscience, 8(11):4159-4) 1988.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gru- ber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.022: 2020.
Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H Sinz, and Richard S Zemel. Normaliz- ing the normalizers: Comparing and extending network normalization schemes, arXiv preprint arXiv: /6/ 1.04520, 2016.
Edmund T Rolls. The storage and recall of memories in the hippocampo-cortical system. Cell and fissue research, 373(3)5577-66) 2018.
I. Seress. Interspecies comparison of the hippocampal formation shows increased emphasis the regio superior in the ammon's 's horn of the human brain. Journal fur Hirnforschung, 29(3):335- 340, 1988.
Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.
Alessandro Treves and Edmund T Rolls. Computational analysis of the role of the hippocampus in memory. Hippocampus, 4(3):374-391, 1994.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-60088 2017.
Michael Widrich, Bernhard Schäfl, Hubert Ramsauer, Milena Pavlović, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern hopfield networks and attention for immune repertoire classification. arXiv preprint arXiv:2007. 13505, 2020.
Menno Witter, Thanh P Doan, Bente Jacobsen, Eirik Nilssen, and Shinya Ohara. Architecture of the entorhinal cortex review of entorhinal anatomy in rodents with some comparative notes. Frontiers in Systems Neuroscience, 11:46, 2017.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We thankful to J. Brandstetter, S. Hochreiter, M. Kopp, D. Kreil, H.Rasauerr D. Springer, and Tang for useful discussions.
APPENDIX A
In this appendix we show step by step derivation of the change of the energy function (2) under dynamics ① Time derivative of the energy function can be expressed through time derivatives of the neuron's activities o and h, (the input current is assumed to be time-independent in the calculation below). Using the definition of the functions fa and gi in 3 one can obtain
dE dt
E(m (mr 1:) Lo duy + Σ by
8 Ln dh Oh, dt duž 8 Lo
Dv;dvg dt
1
ma
dhu O
Σ M.
Emii) i
-
<E (ESnt))=
dt dhp
du,
(23)
duj Lo
Esun + vz E dhe Ln
Σ
Σ Euigi km
di OuzOu;
di Ohu
Nr du; L duj 7 dt ov,ōv, dt
Th > dhp Ln dhu
0
dt Oh dt
p.v=l
In the last equality sign the right hand sides of dynamical equations ① are used to replace ex- pressions in the square brackets by the corresponding time derivatives of the neuron's activities. This completes the proof that the energy function decreases on the dynamical trajectory described by equations (T) for arbitrary time constants ry and Ty provided that the Hessians for feature and memory neurons are positive semi-definite.
APPENDIX B. THE LIMIT OF STANDARD CONTINUOUS HOPFIELD NETWORKS.
In this section we explain how the classical formulation of continuous Hopfield networks (Hopfield 1984) emerges from the general theory ( Continuous Hopfield networks for neurons with graded response are typically described by the dynamical equations
N
de: Tf dt
T Tigg
+ I
(24)
1=1
and the energy function
N₃
N₁
N₂
E =
T Tig99 - Σ 9 + Σ/ 92)d2 (z)dz
(25)
2
1.j
i=1
where, in Section 3.1
9: g(u). and (2) is the inverse of the activation function g(x).
According to our classification, this model special limit of the class of models that we call models A, with the following choice of the Lagrangian functions
L Σ ((z)dr, and L
(26)
2
that, according to the definition
3. lead to the activation functions
gi =0))) and
fn h2
(27)
12

---

Published as a conference paper at ICLR 2021
Similarly to Section 3.1
integrate out the hidden neurons to demonstrate that the system of
Nu.
equations I reduces to the equations on the feature neurons (24) with Tú SpiSag. The
general expression for the energy reduces to the effective energy
N E
N
N
Tyg:9
9: +
04gh
g(x)drr
(28)
2 1
2=1
2=1
While the first two terms in equation (25 are the same those in equation (8). the third terms look superficially different. equation (2 it a Legendre transform the Lagrangian for the feature neurons, while in (25) the third term is integral of the inverse activation function. Nevertheless, these two expressions are in fact equivalent, since the derivatives of function and its Legendre transform are inverse functions of each other. The easiest way see that these terms equal explicitly is to differentiate each one with respect to 0,. The results of these differentiations for both expressions are equal veg()))'. Thus, the two expressions are equal up an additive constant. This completes the proof that the classical Hopfield network with continuous states (Hopfield 1984)
a special limiting case of the general theory 2
13


Reviews:
[{"review_id": "fk30n1lUP2Z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a dynamical system that unifies several associative memory models, including the classical Hopfield network and two recently proposed modern Hopfield networks. The dynamical system is described as interactions between two groups of neurons (feature and memory neurons), providing a more biological interpretation of modern Hopfield networks. The proposed system reduces to different associative memory models by choosing different generalized activation functions, each of which maps inputs to a group of neurons into output activity. This manuscript provides sufficient details for understanding, its derivations are correct, and its results are useful in bringing modern Hopfield networks closer to biology.\n\n### Major:\n\n(1)\tMy main concern is that the tradeoff of the modern Hopfield networks discussed is not clearly stated.\n\nFor example, the manuscript at multiple places described modern Hopfield networks as having large or huge memory capacity with respect to $N_f$. I understand that these statements have been used in the literature. However, since the authors are explicitly trying to interpret modern Hopfield networks as network of neurons, it would be appropriate to discuss the memory capacity with respect to both $N_f$ and $N_h$. For example in model B, if I understood correctly, the memory capacity scales linearly with $N_h$.\n\nRelated to the previous point, the number of memory neurons needed should be clearly described. Is it the number of memory patterns stored across all models?\n\n(2)\tThe energy function should be better motivated.\n\nIn equation 2, the authors introduce an energy function containing two Lagrangian functions, and demonstrate that it reduces to energy functions from several previous models when using different choices of Lagrangian functions. While this is intriguing, it can make the reader feels like this energy function came out of nowhere and magically unifies multiple models. I believe the reader would have a better understanding if the authors provide a stronger motivation for the energy function. For example, is the energy function constructed by explicitly trying to connect the Krotov & Hopfield 2016 model with the Ramsauer 2020 model?\n\n### Additional major points:\n\n(3)\tI think the authors should offer richer references to the literature. Here are some examples:\n\nIn the intro, “a cortical synapse stores no more than one or two bits of information” should be followed by a reference since this is not common knowledge. For example, Bartol 2015 eLife estimates a lower bound of 4.7 bits per synapse.\n\nIn p.4, the authors mention BERT-like system without citing the BERT paper.\n\nThe activation function in Eq. 17 is a form of divisive normalization widely studied in neuroscience (Carandini & Heeger 2012).\n\n(4)\tI find it somewhat misleading to refer to the update rule in Eq. 15 as the “attention mechanism in Transformer networks”. I understand this is what Ramsauer et al 2020 said as well, but I think the update rule in Eq. 15 should be simply referred to as dot-product attention, which is used in machine learning at least as early as Bahdanau, Cho, Bengio 2014 and in many settings besides Transformers.\n\n### Minor:\n\nP.2 “integrates our some of the degrees of freedom”  “integrates out…”\n\nIn P.1, the sentence “a small part of a high-resolution photograph may contain only 1000 pixels, but the number of describable “objects” which might occur in such a fragment is far larger” is a bit confusing. \n\n\n-----------------------------------\nPost revision update:\nAll concerns addressed. Score updated.", "extra_scores": {}, "extra_text": {}}, {"review_id": "e40Qv7kQ1jg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: In the current work, the authors describe a novel memory structure, and mathematically show how this is a superset of previously published models. The paper could be of interest to anyone investigating the theoretical side of learning and activity rules.\n\nStrong Points: Unifying the multiple discussed architectures, as well as relating back to RBM and classical Hopfield networks is a nontrivial task. Additionally, the use of integrator units (eq 1) provides a clear path how such an energy optimizing network might be implemented in biological networks (eg: LIF neurons).\n\nWeak Points: Biological plausibility is discussed at several points in the paper, specifically regarding the number of units involved in an interaction. However, this plausibility doesn't influence the functional design of the network, and thus feels extraneous. \n\nAdditional Comments:\nRegarding the above: If the feature units are to represent distributed patterns (eg: BERT features), then it should be possible for more than 2 abstract units to interact at a single \"synapse\", so long as the interaction of the actual units (neurons or RELU units) has only two members?\nAre the memory capacity estimates (directly following eq 8) only applicable when the input current is 0? If so, is it possible to derive more general estimates of capacity?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KrqCfh2D10z", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel class of associative memory models. The model is expressed as a network with two-body interactions (synapses) and a well-defined energy function, and it is shown to generalise and unify several existing approaches (Hopfield Networks, Dense Associative Memories and Modern Hopfield Networks). Besides its theoretical and computational properties, the model is presented as being more biologically valid/plausible than some of the existing approaches it generalizes.\n\nOverall the paper is solid, well written and highly relevant to the ICLR community. My initial recommendation is to accept. I only have a few nitpicks concerning the \"biologically plausible\" angle.\n\nMore specifically, while I agree that the proposed model is *more* plausible than some of the other approaches discussed, its absolute level of biological plausibility remains limited. The authors recognise this point and devote a paragraph to discussing it at the end of section 2 (\"For the purposes of this paper we defined biological plausibility as…\"), but it seems odd to have this passage buried at the end of the mathematical derivation, rather than up front in the introduction. I suggest that this passage is moved forward to a more prominent location.\n\nFurthermore, given that this is essentially a paper on the theory of abstract associative memory systems, the emphasis given to the biological angle in the title seems eccessive, and the choice of the word \"neurobiology\" somewhat puzzling. In my opinion the title would be a better description of the content of the paper if the reference to biology was toned down.\n\nFinally, in the introduction: \"typical synapses are not highly reliable, and a cortical synapse stores no more than one or two bits of information\". While I agree with the general spirit of the observation that synapses are not typically very reliable, some source should provided to back up the quantitative statement about synaptic storage capacity. I am not a specialist on the matter, but this seems surprising in light of work showing that the capacity of hippocampal synapses can be up to 3 to 5 bits (Bartol et al 2015, Bromer et al 2018).\n\n### References \n\nBartol Jr, T. M., Bromer, C., Kinney, J., Chirillo, M. A., Bourne, J. N., Harris, K. M., & Sejnowski, T. J. (2015). Nanoconnectomic upper bound on the variability of synaptic plasticity. Elife, 4, e10778.\n\nBromer, C., Bartol, T. M., Bowden, J. B., Hubbard, D. D., Hanka, D. C., Gonzalez, P. V., … & Sejnowski, T. J. (2018). Long-term potentiation expands information content of hippocampal dentate gyrus synapses. Proceedings of the National Academy of Sciences, 115(10), E2410-E2418. \n\n----\n\nPost revision update: my concerns have all been addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "B8VGUaQdJGy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a mechanism by which recent extensions to the classic Hopfield network model, which as written involve many-body interactions, can be implemented by a more biologically plausible network that uses only two-body synaptic interactions (but more neurons).  \n\nPros:\n\n-- The derivations are sound and well-explained, from what I can tell\n-- Given that Hopfield networks are an important model in the neuroscience community, it is nice to see these extensions to them brought back in to the realm of biological implementation\n\nCons:\n\n-- My main concern with this paper is that I feel like it has not established its significance to neuroscience modelers -- and since the paper is primarily concerned with connecting existing algorithms to biologically plausible implementations, rather than introducing new algorithms, I think it is important to do so.  Dense associative memories and modern Hopfield networks are appealing because of their ability to store memories beyond the O(N) scaling limit of regular Hopfield networks (N = # of neurons).  But in these two-body implementations, the models once again have O(N) scaling in the total number of neurons due to the addition of hidden neurons.  Given this, what is the advantage of these models over the standard Hopfield network?  Are they still more efficient in some way?  More robust?  Do they provide faster recall?  I think a thorough comparison along these lines would be valuable and without it, the paper is less compelling to me.\n\n-- The connection to attention mechanisms feels a bit off in that the equivalence only holds when the update rule is applied exactly once and no more times (I realize this was first shown in prior work).  But given that this paper is attempting to unify several mechanisms under a common framework, it feels like the paper should concern itself with models that really are variants of that framework, rather than introducing additional modifications like cutting off the dynamics early.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attends to the problem of how to implement dense associative memories (i.e. modern Hopfield networks) using only two-body synapses. This is interesting because modern Hopfield networks have much higher capacity, but at face value, they require synapses with cubic interactions between neurons, which to the best of our knowledge, is not a common feature in neurophysiology (though it should be noted: it is not by any means impossible from a physiological perspective to have cubic interactions at synapses, see e.g. Halassa, M. M., Fellin, T., & Haydon, P. G. (2007). The tripartite synapse: roles for gliotransmission in health and disease. Trends in molecular medicine, 13(2), 54-63.). 

The authors show how the use of a layer of hidden neurons, akin to a restricted Boltzmann machine architecture, coupled with the right energy function, can be used to recover dense associative memory models using only two-body synapses. They also demonstrate how this connects to recent work on the relationship between attention mechanisms in modern ML models and Hopfield network dynamics.

Overall, the reviewers were positive on this paper. The most common critique related to the question of "biological plausibility". The authors addressed these concerns by adding some more recognition as to the lack of biological plausibility and more discussions of the relevance to neuroscience. To be candid with the authors, if the goal is indeed to make a more biologically plausible model of modern Hopfield networks, than a fair bit more work would be needed to connect the paper to biology well. As it stands, the only connection is the shift to two-body synapses by using hidden neurons, but this provides limited insight for most neuroscientists, as noted by Reviewer 2. Also, some of the biological examples provided seem strained (e.g. the colour memory example, where there is no physiological reason to posit that we store colour memories using our retina, or the MNIST example, since there is no reason to suppose that animals can memorise thousands of specific MNIST images). But overall, the critique regarding biological plausibility was attended to. The other concerns raises were also largely addressed. 

Given the interesting contributions from this paper, the overall positive reviews, and the decent job at addressing reviewer concerns, the AC believes that this paper should certainly be accepted. A decision of "Accept (Poster)" seems appropriate, though (as opposed to an oral or spotlight), given the lack of biological connections in a paper with a stated goal of achieving a more biologically realistic model.
Decision: Accept (Poster)