Simplicial Hopfield networks
Authors: Thomas F Burns
Tomoki Fukai
Venue: ICLR 2023 poster (2023)
Keywords: Hopfield network
associative memory
attention
computational neuroscience
simplicial complex
topology
memory capacity

Abstract: Hopfield networks are artificial neural networks which store memory patterns on the states of their neurons by choosing recurrent connection weights and update rules such that the energy landscape of the network forms attractors around the memories. How many stable, sufficiently-attracting memory patterns can we store in such a network using $N$ neurons? The answer depends on the choice of weights and update rule. Inspired by setwise connectivity in biology, we extend Hopfield networks by adding setwise connections and embedding these connections in a simplicial complex. Simplicial complexes are higher dimensional analogues of graphs which naturally represent collections of pairwise and setwise relationships. We show that our simplicial Hopfield networks increase memory storage capacity. Surprisingly, even when connections are limited to a small random subset of equivalent size to an all-pairwise network, our networks still outperform their pairwise counterparts. Such scenarios include non-trivial simplicial topology. We also test analogous modern continuous Hopfield networks, offering a potentially promising avenue for improving the attention mechanism in Transformer models.

Published as a conference paper at ICLR 2023
SIMPLICIAL HOPFIELD NETWORKS
Thomas Burns
Tomoki Fukai
Neural Coding and Brain Computing Unit OIST Graduate University, Okinawa, Japan thomas burns@oist.jp
Neural Coding and Brain Computing Unit OIST Graduate University, Okinawa, Japan tomoki,. fukai@oist.
ABSTRACT
Hopfield networks are artificial neural networks which store memory patterns the states of their neurons by choosing recurrent connection weights and update rules such that the energy landscape of the network forms attractors around the memories. How many stable, sufficiently-attee attraacing memory patterns can we store in such network using N neurons? The answer depends on the choice of weights and update rule. Inspired by setwise connectivity in biology, we extend Hopfield networks by adding setwise connections and embedding these connections in a simplicial complex. Simplicial complexes are higher dimensional analogues of graphs which naturally represent collections of pairwise and setwise relationships. We show that simplicial Hopfield networks increase memory storage capacity. Surprisingly, even when connections are limited to a small random subset equivalent size
all all-pairwise network, networks still outperform their pairwise counterparts. Such scenarios include non-trivial simplicial topology. We also test analogous modern continuous Hopfield networks, offering potentially promising avenue for improving the attention mechanism in Transformer models.
INTRODUCTION
Hopfield networks (Hopfield 1982 store memory patterns in the weights of connections between neurons. In the case of pairwise connections, these weights translate to the synaptic strength between pairs of neurons in biological neural networks. In such a Hopfield network with N neurons, there will be (N) of these pairwise connections, forming a complete graph. Each edge is weighted by procedure which considers P' memory patterns and which, based on these patterns, seeks to minimise a defined energy funetion such that the network's dynamics are attracted to and ideally exactly settles in the memory pattern which is nearest to the current states of the neurons. The network therefore acts as content addressable memory given a partial or noise-corrupted memory, the network can update its states through recurrent dynamics to retrieve the full memory. Since its introduction, the Hopfield network has been extended and studied widely by ncuroscientists (Griniasty et al., [993] Schneidman et al. [2006] Sridhar et al., 2021 Burns et al. (2022), ) physicists (Amit et al. 1985 Agliari ct al. 2013 Leonetti et al.] (021), and computer scientists (Widrich et 2020 Millidge al. 2022 Of particular interest to the machine learning community the recent development of modern
Hopfield networks (Krotov Hopfield 2016) and their close correspondence (Ramsauer et al. 2021
the attention mechanism of Transformers /Vaswani al. 2017).
An early (Amit et al., 1985; McEliece et al., 1987 and ongoing (Hillar & Tran 2018) theme in the
study of Hopfield networks has been their memory storage capacity, i.e., determining the number of memory patterns which can be reliably stored and later recalled via the dynamics. As discussed in Appendix A.. this theoretical and computational exercise serves two purposes: (i) improving the memory capacity of such models for theoretical purposes and computational applications; and (ii) gaining an abstract understanding of neurobiological mechanisms and their implications for biological memory systems. Traditional Hopfield networks with binary neuron states, in the limit of N + co
and - X, maintain associative memories for up to approximately 0.14N patterns (Amit et al.
After the proposal of Mar (1971 many similar models of associative memory were proposed, e.g., those of Nakano ( (1722) Amari [1972). Little (1974). and |Stanley (1976) all before Hopfield (1822. Nevertheless. much of the research literature refers and seems more proximally inspired by HHoficld] (182). Many of these models also he considered instances of the Lenz-Ising model (Brush 1967) with infnite-range interactions.

---

Published as a conference paper at ICLR 2023
1985
McEliece et al. 1987), and fewer if the patterns statistically spatially correlated (Löwe However, by a clever reformulation of the update rule based on the network energy. this capacity can be improved Nd-1, where d 2 2 (Krotov & Hopfield 2016). and even further to 2 (Demircigil et al., (017). Networks using these types of energy-based update rules are called modern 1984)) also investigated neurons which took on continuous states. Upon generalising this model by using the softmax activation function, Ramsauer et al. (2021) showed a connection to the attention mechanism of Transformers (Vaswani al. (2017). However, the best of our knowledge, these modern Hopfield networks have not been extended further to include explicit setwise connections between neurons, as has been studied and shown improve memory capacity in traditional Hopfield networks (Peretto & Niez 1986 Lee et al.
1998).
Hopfield networks, Krotov Hopfield (2016) (like [okfield (19)ieed((99))))))
1986 Baldi & Venkatesh/ 1987 Newman 1988). Indeed. Krotov & Hopfield (2016) who introduced modern Hopfield networks, make a mathematical analogy between their energy-basee update rule and setwise connections given their energy -aargyd--dddddd update rule can be interpreted as allowing individual pairs of pre- and post-synaptic neurons to make multiple synapses with each other making pairwise connections mathematically as strong equivalently-ordens (2017) later proved this analogy to be accurate in terms of theoretical memory capacity. By adding explicit setwise connections to modern Hopfield networks, we essentially allow all connections (pairwise and higher) to increase their strength following the same interpretation, this can be thought of as allowing both pairwise and setwise connections between all neurons, any of which may be precisely controlled.
setwise connection Demircigil et al.
Functionally, setwise connections appear in abundance in biological neural networks. What's more, these setwise interactions often modulate and interact with one another in highly complex and nonlinear fashions, adding their potential computational expressiveness. We discuss these biological mechanisms in Appendix A.2 There are many contemporary models in deep learning which implicitly model particular types setwise interactions (Jayakumar et al.) 2020). To explicitly model such interactions, we have multiple options. For reasons we discuss in Appendix A.3 we choose to model our setwise connections using simplicial complex.
We therefore develop and study Simplicial Hopfield Networks. We weight the simplices of the simplicial complex to store memory patterns and generalise the energy functions and update rules of traditional and modern Hopfield networks. Our main contributions are:
We introduce extensions of various Hopfield networks with setwise connections. In addition to generalising Hopfield networks to include explicit, controllable setwise connections based on an underlying simplicial structure, we also study whether the topological features the underlying structure influences performance.
We prove and discuss higher memory capacity in the general case of simplicial Hopfield networks. For the fully-connected simplicial Hopfield network, prove a larger memory capacity than previously shown by Newman (1988); Demircigil et al. (2017 for higher- degree Hopfield networks.
We empirically show improved performance under parameter constraints. By restricting the total number of connections to that of pairwise Hopfield networks with a mixture of pairwise and setwise connections, we show simplicial Hopfield networks retain surprising amount of improved performance over pairwise networks but with fewer parameters, and are robust to topological variability.
2 SIMPLICIAL HOPFIELD NETWORKS
2.1 SIMPLICIAL COMPLEXES
Simplicial complexes are mathematical objects which naturally represent collections of setwise relationships. Here to build intuition and visualise the simplicial complex, we also refer to their geometric realisations.
the combinatorial form, called abstract simplicial complex. Although,
Definition 2.1. Let K a subset of 2|, The subset is an abstract simplicial complex for any K, the condition p gives p K, for any p S σ.
Work by Horn, D. & Usher, M. 1988) study almost the same system but with slight modification to the traditional update rule, whereas Krotov & Hoptecld (2016) use their modern, energy-based update rule.
2

---

Published as a conference paper at ICLR 2023
{ij.k}
5
Ki.jk,,1
1
G
4 1
{i
{i.}}
C
+
0.35
reletive frequency
G
0.1
0.0
0.1
-0.1
0.0
0.1
-0.1
0.0
0.1
1-simplex weights
2-simpllxx weights
3-simplex weights
Figure 1: A. Comparative illustrations of connections in a pairwise Hopfield network (left) and simplicial Hopfield network (right) with N 4. In simplicial Hopfield network, o { is an edge (1 (1-simplex), a = {i }} is triangle (2-simplex), a = j, k, 1) is tetrahedron (3-simplex), and B. Connection weight histograms of 1-, 2-, and mnd3-siin 3-simplices in simplicial Hopfield network. In the binary case, the x-axis range is |PP/N, +PNN Here, N = 100 and / = 10, thus the range is [-0.1, +0.1]. Note that each dimension shows a similar, Gaussian distribution weights (although there are different absolute numbers of these weights; see 'Mixed diluted networks' in Section 2.2p. C. Illustration of the hierarchical relationship between elements in the complex, up to 3 simplices, with arrows indicating potential sources of weight modulation or interaction, e.g.. between (co)faces or using Hodge Laplacians within the same dimension. Such modulations and interactions (including their biological interpretations) are discussed in Appendices A.2 and A.3
In other words, abstract simplicial complex K is collection of finite sets closed under taking subsets. A member of K called simplex a. didiiensionaa simplex (or k-simplex) has cardinality & + 1 and & + faces which are (k- 1i-simplices (obtained by omitting one element from σ). If a simplex a is a face of another simplex we say that is coface of a. We denote the of all k-simplices in K as Kg.
Geometrically, for = 0, 1, and 3, a k-simplex is, respectively, a point, line segment, triangle, and tetrahedron. Therefore, one may visualise simplicial complex as being constructed by gluing together simplices such that every finite set of vertices in K form the vertices of at most one simplex. This structure makes possible to associate every setwise relationship uniquely with a k-simplex identified by its elements, which in our case are neurons (see Figure [). contained in no higher dimensional simplices, i.c., they have no cofaces, are called the facets of K. The dimension of K, dim(K) is the dimension of its largest facet. We call simplicial complex K k-skeleton when all possible faces of dimension exist and dim (K) = k.
(A). Simplices in K which are
2.2 MODEL
network of neurons modelled by N spins. Let K be a simplicial complex on vertices. In the binary neuron case, 5(0) = +1 at time-step t. Given a set neurons a (which contains the neuron and a unique ([ol - 1) simmlxx in K), w(a) is the associated simplicial weight and s((t) the product of their spins. Spin configurations correspond to patterns of neural firing, with dynamics

---

Published as a conference paper at ICLR 2023
governed by defined energy. The traditional model is defined by energy and weight functions
E Σ w( oEK
Σ S0 m((r) N
(1)
with 5 (= =1) static variables being the P binary memory patterns stored in the simplicial weights. Similar for spins, Eh is the product of the static pattern variables for the set of neurons a in the pattern J.. Figure 3 shows examples the resulting Gaussian distributions of weights at each dimension of the simplicial complex. We use these weights to update the state of neuron by applying the traditional Hopfield update rule
$/
( KEE
EE ((o)s 0/
(xx)
{
i i
When K is 1-skeleton, this becomes the traditional pairwise Hopfield network (Hopfilld 1982). In the modern Hopfield case, the energy function and update rule are
P Σ
E
Σ
(3)
w=loek
1 sgn Σ
F(1 F(1.g)
5(1-1) - oEK
E S ole
where the function can chosen, for example, to of a polynomial F(x) = x" or exponential F(z)= ex form. When K is 1-skeleton, this becomes the modern pairwise Hopfield network
(Krotov Hopfield 2016).
In the continuous modern Hopfield case, spins and patterss take real values Sn, E R. Patterns arranged matrix = (5), ...,""" and we define the log-sum-exp function (Ise) for T
P Σ
Ise( =" s(t), K) = T log
Σ exp exp(T~-=nS!0)
(5)
w=
The energy function is
1 S g(t) 2
= -s
(6)
For each simplex a E K, we denote the submatrix of the patterns stored on that simplex as E, (which has dimensions P x σ). Using the dot product measure the similarity between the patterns and spins, the update rule is
5:t softmax
E oEK
T
E stt
In practice, however, the dot product has been found under-perform in modern continuous Hopfield networks compared to Euclidean or Manhattan distances (Millidge et al. (022). Transformer models in natural language tasks have also seen performance improvements by replacing the dot product with cosine similarity (Henry et al. (020). again measure with more geometric flavour. However, these similarity measures generalise distances between pairs of elements rather than sets of elements.
We therefore use digeee--imeesioonll geometric similarity measures, cumulative Euclidean distance (ced) and Cayley-Menger distance (emd). Let d, the (Fuclidean or Manhattan) distance between pattern E/ and spins 5 for pattern μ and spins p C a. Let Kÿ be the subset of K such that all elements in KT are 1-simplex faces of a. We define the cumulative Euclidean distance as
cen s(t))
Σ (d,)². PEKT
We define cmd(EH (EH, ( as the Cayley- determinant of all p Kÿ, with distances set dp-
4

---

Published as a conference paper at ICLR 2023
Mixed diluted networks. A computational concern in the above models is that the number of unique possible k-simplices is (c41), c.g.. with N 100 there approximately 9.89 x 10as possible 50- simpliess, compared to just 4, 950 edges (1-simplices) found in a pairwise Hopfield network. If we allow all possible simplices for a simplicial Hopfield network with N neurons, the total number of simplices (excluding (-simplices, i.e., autapses) will be Σd (2) Simultanooull, there is also an open question to how many setwise connections is viooggiallly-raalise to model. We also note that setwise connections can be functionally built from combinations of pairwise connections by introducing additional hidden neurons, as shown by Krotov & Hopfield (202I). Therefore, we might in fact be under-estimating the total number of "functional" setwise connections, which may appear via common network motifs or 'synapsembles' (Buzsáki 2010).
Conservatively, we evaluate classes of simplicial Hopfield networks which are low- now-dimensional. i.c., dim(K) ) is small, and where the total number of weighted simplices not greater than those normally found in a pairwise Hopfield network, i.e., the number of non-zero weights (2). We randomly choose weights to be non-zero, with each weight of dimension having an equal probability and according to Table T] (See Appendix [A.4for a small worked example.) Such random networks have previously been studied in the traditional pairwise case as 'diluted networks' (Treves & Amit [988] Bovier & Gayrard] 1993a b Löwe & Vermet (20TT). Here we study mixed diluted networks, since use mixture of connections of different degrees. We believe we are also the first study such networks beyond pairwise connections, as well as in modern and continuous cases.
Topology. Different collections of simplices in simplicial complex can result in different Fuler characteristics (a homotopy invariant property). [ this from a parameter perspective via counting only the simplices with non-zero weights. However, even when using the same proportion of 1- and 2-simplices, the choices of which vertices those simplices contain can be different due to randomness. Therefore, the topologies of each network may vary (and so too may their subsequent dynamics and performance). One well-studied and often important topological property in the context of simplicial complexes, homology, counts the number of holes in each dimension. the Oth dimension, this is the number of connected components; in the 1st dimension, this is the number of triangles formed by edges which don't also have a 2-simplex 'filling in' the interior surface that triangle; in the 2nd dimension, this is the number of tetrahedra formed by triangles which don't 't also have a 3-simplex 'iilling in' the interior volume of that tetrahedron; and so on. The exact number of these holes in dimension & can be calculated by the k-th Betti number, 3k (see Appendix A.55). We calculate these for our networks to observe the relationship between homology and memory capacity.
2.3 THEORETICAL MEMORY CAPACITY
Mixed networks. Much is already known about the theoretical memory capacity of various Hopfield networks, including those with explicit (Newman 1988) implicit (Demircigil et al.) (017) setwise connectivity. However, we wish to point out somewhat underappreciated relationship between memory capacity and the explicit or implicit number of network connections which, in the fully- connected network, is determined by the degree of the connections (see Appendix A.6 for proof).
Corollary 2.2 (Memory capacity is proportional to the number of network connections). If the connection weights in a Hopfield network are symmetric, then the order of the network's 's memory capacity is proportional to the number of its connections.
What happens when there are connections between the same neurons at multiple degrees, i.c.. what we call mixed network? To the best of our knowledge, the theoretical memory capacity of such networks has not been well-studied. However, we found one classical study by |Dreyfus et al.) (1987) which showed, numerically, adding triplet connections to a pairwise model improved attractivity and memory capacity. Most prior formal studies have only considered connections at single higher degrees (Newman 1988 Bengtsson 1990). Although, higher order neural networks have historically considered such mixtures of interactions different degrees simultaneously (Zhang) 2012). but as regular neural networks (e.g.. feed-forward networks), not Hopfield networks. Higher order Boltzmann machines (HOBMs) (Sejnowski 1986) have also been studied with mixed connections
5

---

disconnected its dynamics Published as a conference paper at ICLR 2023
However, HOBMs unlike Hopfield networks in that they typically have hidden units, are trained differently, and have stochastic neural activations Modern Hopfield networks also include implicit mixture of connections of different degrees (but (but
(Amari et al. 1992. Leisink & Kappen 1999
and see Theorem of Df et al. (2017). which remains unproven the mixture is unbalanced and not particularly natural, especially for F(x) = x² when odd). Therefore, we include the following result demonstrating fixed points, large basins of attraction (i.e., convergence) to those fixed points mixed networks, and memory capacity which is linear in the number of fully-connected degrees of connections (a proof is provided in Appendix [A.6p.
Lemma 2.3 (Funly-connected mixed Hopfield networks). A ffllly mixed Hopfield network based on a D-skeleton with neurons and P patterns has, when N - oo and P is finite: (i) fixed point attractors at the memory patterns; and (ii) dynamic convergence towards the fixed point attractors within a finite Hamming distance When P + oo with + co, the network has capacity to store up to | (2 In N) memory patterns (with small retrieval errors) or (Et- / In N) (without retrieval errors).
This naturally comports with Theorem 2 from Demircigil al. (2017), except here show an increased capacity in the mixed network, courtesy of Corollary 2.2
Mixed diluted networks. As mentioned earlier, full setwise connectivity is not necessarily tractable nor realistic. Löwe & Vermet (201 show for pairwise diluted networks constructed as Erdös-Rens Renyi graphs (constructed by including each possible edge on the vertex set with probability p) that the memory capacity is proportional to p.V. Crucial for this result is that the random graph must be asymptotically connected. This makes sense, given that if any vertex could never be influenced. Empirically, it does seem that a certain threshold of mean connectivity in pairwise random networks crucial for attractor dynamics (Treves & Amit 1988
Remark 2.4. By a straightforward generalisation of Löwe & Vermet (2011) 's result, diluted networks constructed as pure Erdös-Reny. hypergraphs may store the order of p-Nd-1 memory patterns, where d is the degree of the connections.
In the case of an unbounded number of allowable connections, Remark 2.4 would suggest picking as many higher-degree connections as possible when choosing between connections of lower or higher degrees in mixed diluted networks. However, in the bounded case (our case). are non-trivin non-trivially changing the asymptotic behaviour in terms connectivity and dynamics when we use mixture of connection degrees. We also need to beware of asymmetries which may arise (Kanter 1988). This makes the analysis of mixed diluted networks not particularly straightforward (also see Section
2.4 NUMERICAL SIMULATIONS AND PERFORMANCE METRICS
Given the large space of possible network settings, in the main text we focus primarily on conditions listed in Table I Additional experiments are also shown in Appendix A.8
Table I: List of network condition keys (top row), their number of non-zero weights for 1- and 2 simpplices (second and third rows), and their "functional" Fuler characteristic (x, bottom row). N is the number of neurons. are rounded to the nearest integer.
(N 1)N. For simulation, the number of simplices each dimension
K1
RI2
RI2
R12
R2
--simplices
(
0.75())
0.50 (
0.25 (
0
2-simplices
0.25 (N)
(x)
0.75 (
(N)
X
N - (1/2)C
N 0.25C
N
N + 0.25C
N + (1/2)C
è HHBMMs also suffer the same problem as we face here, one of having many high-order parameters between the neurons to keep a track of. Possibly a factoring trick like in Memiseevic Hinton (2010) for HOBMs could be helpful in simplicial Hopfield networks.
Despptte this, there are equivalences (Lconelli
2021 Marullo Agliari 2021
Smart & Zilman 2021
RRcall that (E,a) a.) = -,a .+1

---

Published as a conference paper at ICLR 2023
In our numerical simulations, perform updates synchronously until E is non-decreasing until a maximum number of steps is reached, whichever comes first. When a simulation concludes we compute the overlap (for binary patterns) mean squared error (MSE) (for continuous patterns) of the final spin configuration with respect to all stored patterns using
N
m
1
MSE
( 5
We say the network recalls (or attempts to recall) whichever pattern has the largest overlap (where r = indicates perfect recall) or smallest MSE (where MSE" 0 indicates perfect recall).
3 NUMERICAL SIMULATIONS
3.1 BINARY MEMORY PATTERNS
After embedding random binary patterns, we started the network in random initial states and recorded the final overlap of the closest pattern. Table 2 shows the final overlaps for traditional simplicial Hopfield networks (N = 100). Our simplicial Hopfield networks significantly outperform the pairwise Hopfield networks (KI). In fact, the R12 model performs well at 0.3NN patterns as the the pairwise network performs on 0.05NN patterns, a six-fold increase in the number of patterns and more than double the theoretical capacity of the pairwise network, 0.14N (Amit al. 1985). Surprisingly, Table 3 shows homology accounts for very little of the variance in network performance.
Table 2: Mean + standard deviation of overlap distributions (n = 100) from traditional simplicial Hopfield networks with varying numbers (top row) of random binary patterns, KI is the traditional pairwise Hopfield network. R12 significantly outperforms K1 at all tested levels (one-way t-tests p 10-11,110000 F > 50.13). At all pattern loadings, a one one-way way ANOVA showed significant variance
between the networks (p < 10 20
F 26.35). Box and whisker plots shown in Figure
No. patterns
0.05.N
0.1N
0.15.N
0.2N
0.3NV
Kl
0.87 + 0.18
0.81 + 0.16
0.66 = 0.10
0.65 + 0.10
0.59 + 0.08
RI2
0.96 0.10
0.94 0.14
0.82 = 0.20
0.71 = 0.17
0.64 + 0.13
RT2
0.98 + 0.10
0.99 + 0.03
0.97 + 0.10
0.91 + 0.15
0.76 + 0.16
R12
1 0
0.99 + 0.04
0.99 + 0.05
0.98 + 0.08
0.87 + 0.16
R2
0
0.99 + 0.18
0.94 + 0.18
0.74 + 0.29
0.53 + 0.23
3.2
CONTINUOUS MEMORY PATTERNS
Energy landscape. Using Equation 3 and given set of patterns, simplicial complex K, and an inverse temperature T- we may calculate the energy of network states. To inspect changes in the energy landscapes of different network conditions, we set N = 10 and P = 10 random patterns. We performed principle component analysis (PCA) to create low dimensional projection
the patterns. Then, we generated network states evenly spaced in a 10 x 10 grid which spanned the projected memory patterns in the first two dimensions of PCA space. We calculated each state's energy by transforming these points from PCA space back into the N Nddimensional space, across the network conditions at 7-1 = 1, 2, 10 (Figure 7. At T = 1, differences between the network conditions' clear change: those with more 2-simplices possess more sophisticated, pattern-separating landscapes.
energy landscapes are very subtle. However, at T-1 2 and T-1 10,
Recall as a function of memory loading. We tested the performance of our simplicial Hopfield networks by embedding data from the MNIST (LeCun et al.) 2010), CIFAR-10 (Krizhevsky & Hinton 2009), and Tiny ImageNet (Le & Yang 2015) datasets as memories. We followed the protocol of Millidge et al. (2022) to test recall under increasing memory load memory capacities. To embed the memories, we normalise the pixel values between 0 and 1, and
indication of the networks'
treat them as continuous- valued neurons, e.g., for MNIST we have N 28 28 784 neurons.
7

---

Published as a conference paper at ICLR 2023
We initialise S one of the memory patterns corrupted by Gaussian noise with variance 0.5. After allowing the network to settle an energy minima, we measured the performance as the fraction of correctly recalled memories (over all tested memories) of the uncorrupted patterns, where "conrcc recall' was defined as a sum of the squared difference being < 50. In all tests, we used 7 = 100. Also see Appendix A.7 for further simulation details.
Figure compares pairwise architecture, K1, with higher-order architecture, R12. The perfor- mance the K1 networks is comparable to that shown in Millidge et al. (0022), however, R12 significantly outperforms KI across all datasets. Since the MNIST dataset is relatively simple and K1 already performs well, the performance improvement is small, albeit significant. the CIFAR-10 and Tiny ImageNet datasets, the performance improvements are more noticeable, with most distance functions seeing improvements of 2 10% in the fraction of correctly retrieved memories.
MNIST
CIFAR-10 10
Tiny ImageNet
Fraction correctly recalled
1.0
0.6
0.4
0.0
Fraction correctly recalled
1.0
0.8
0.6
Euclidean
0.4
Manhattan
Dot Product
0.2
cmd
0.0
800
1000
800
1000
200
800 1000
Memories stored
Memories stored
Memories stored
Figure 2: Recall (mean + S.D. over 10 trials) function of memory loading using the MNIST, CIFAR-10, and Tiny ImageNet datasets, using different distance functions (see legend). Here we compare the performance of modern continuous pairwise networks (top row) and modern continuous simplicial networks (bottom row). The simplicial networks are R12 networks (see Table for information). R12 significantly outperforms the pairwise network (K1) at all tested levels where there was not already perfect recall (one-way t-tests < 10-9, F > 16.01). At all memory loadings, a one-way ANOVA showed significant variance between the networks (p < 10-5 Tabulated results are shown in Tables 6 and
11.95).
Also noticeable the results for CIFAR-10 and Tiny ImageNet (see Figure is the relatively high performance of the ced and cmd distance measures. Indeed, cmd performs as well or better than Manhattan distance in our experiments. And both ced and cmd (along with the Euclidean and Manhattan distances) outperform the dot product in CIFAR-10 and Tiny ImageNet at high memory loadings. This further supports the intuition and results of Millidge et al. (2022), that more 'geometric' distances perform better similarity measures in modern Hopfield networks.
DISCUSSION
We have introduced new class of Hopfield networks which generalises and extends traditional, modern, and continuous Hopfield networks. Our main finding is that mixed diluted networks improve performance in terms memory recall, even when there is no increase in the number of parameters. This improvement therefore comes from the topology rather than additional information
8

---

Published as a conference paper at ICLR 2023
in the form of parameters. We also show how distance measures of more 'geometric flavour' can further improve performance in these networks. This simplicial framework (in diluted or undiluted forms) now opens up new avenues for researchers in neuroscience and machine learning. In neuroscience, we can now model how setwise connections, such as those provided by astrocytes and dendrites, may improve memory function and may interact to form important topological structures to guide memory dynamics. In machine learning, such topological structures may now be utilised in improved attention mechanisms or Transformer models, such as in Ramsauer et al. (202T). At the intersection of these fields, we may now further study how the topology of networks in neuroscience and machine learning systems may correspond to one another and share functional characteristics, such as how the activity of "pairwise" Transformer models have shown similarities to activities in auditory cortex (Millet et al. 2022). Could 'setwise' Transformer models correspond more closely? Or to a more diverse range of cell types? These and related questions are now open for exploration, and may lead to improved performance in applications (Clift et al. 2020).
Convolution operations and higher-order neural networks. From the perspective of modern deep learning, considering higher order correlations between downstream inputs to a neuron is quite classical. For example, convolutional neural networks have incorporated specialised setwise operations since their inception (Fukushima) 1980; Lecun et al., 1998), and more general setwise relationships have also been introduced in higher-order neural networks (Pinedal 1987 Reid et al.)
1989: Ghosh& & Shun 1992 Zhang 2012). Although our setwise connections are not explicitly convolutional, they are one notable sense conceptually similar: they collect information from a particular subset of neurons and only become active when those particular neurons are active in the right way. One of the main differences, however, is that unlike typical convolution operations - we don't 't restrict the connection locations to some particular locations or arrangements within the input space. Our results therefore suggest that, in some cases, replacing regular feedforward connections with random convolutions may offer improved performance in some cireumstances.
Improvements and extensions. Our study focusses on random choices of weighted simplices. What if we choose more carefully? Indeed, it seems quite likely biological setwise connections are not random, and are almost certainly not randomly chosen to replace random pairwise connections.
now seems natural to study how online weight modulations (e.g., based on spectral theories) could generate new connections between Hopfield networks and, e.g., geometric deep learning. Such modulations may have novel biological interpretations, e-g., spatial and anti-Hebbian memory structures may be modelled by strategically inserting inhibitory interactions (Haga & Fukai 2019 Burns et al. 2022) between higher simplices (and may also model disinhibition).
Further analytic studies. Our numerical results suggest diluted mixed networks have larger memory capacities than pairwise networks. In a fairly intuitive sense, this is not particularly surprising are adding degrees of freedom to the energy landscape, within which may build more stable and nicely-behaved attractors. However, we have not yet proven this increased capacity analytically for the diluted case, only given some theoretical indications undiluted case. We hypothesise it is possible to do so using generalised methods from replica- symmetric theory (Amit et al., 1985) or self-consistent sognal-to-noise analysis (Shino & Fukai 1993). in combination with methods from structural Ramsey theory. The capacity for modern simplicial networks may be on the order of a double-exponential in the number of neurons (since, in the limit of N + oo, there is an exponential relationship in the number of multispin interactions on top of an exponential relationship in the number of mutaa-muliispin interactions, i.c., both pair-spins and multi-spins can have higher degrees of attraction). This capacity, however, will likely scale nonlinearly with the choice of (random) dilution, e.g., there may be a steep drop in performance around critical dilution range, likely where some important dynamical guarantees are lost due an intolerably small number of connections of a particular order.
to why this occurs and proven the
Even higher orders and diluted mixtures of setwise connections may also be studied. Such networks, per Lemma 2..3 will likely improve their performance as higher-degree connections are added (as shown in Appendix A.8). However, and as implicd in Section 2.3] the number and distribution of these connections may need to be careful chosen in highly diluted settings.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To reproduce our results in the main text and appendices, we provide our Python code as supplemen- tary material at https: ://g:/hbbb wtw...cco/ttt.
We have also provided a small worked example in Appendix A.4 to help clarify computational steps in the model construction. Assumptions made in our theoretical results are stated in Section 2.3 and Appendix A.6
ACKNOWLEDGEMENTS
The first author thanks Milena Menezes Carvalho for graphic design assistance with Figure 1 as well as Robert Tang. Tom George, and members the Neural Coding and Brain Computing Unit at OIST for helpful discussions. We thank anonymous reviewers for their feedback and suggestions. The second author acknowledges support from KAKENHI grants JP19H04994 and JP18H05213.
REFERENCES
Elena Agliari, Adriano Barra, Andrea De Antoni, and Andrea Galluzzi. Parallel retrieval of correlated patterns: From Hopfield networks to Boltzmann machines. Neural Networks, 38:52-63, 2013.
ISSN 0893-6080. doi: http://oooooo///..ooo///ooceeo22
URL https: /www //ww/
sciencedi com htt.a//www....ocooooooa.coo/o//a
S Amari, Kurata, and H Nagaoka. Information geometry of Boltzmann machines. IEEE Trans Neural Netw, 3(2):260-271, 1992.
S.-l. Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on Computers, C-21(11):1197-1220) 1972. doi: 10. 10.1199T-C.1172.223447.
Daniel J. Amit, Hanoch Gutfreund, and H. Sompolinsky. Storing infinite numbers of patterns in spin-glass model of neural networks. Phys. Rev. Lett., 55:1530-1533, Sep 1985. doi: 10. 100/ PhytRevLett.55.155 15330 URL httt::/:w..a.aes...roo//////////
1530
Alfonso Araque, Vladimir Parpura, Rita P Sanzgiri, and Philip G Haydon. Tripartite synapses: glia, the unacknowledged partner. Trends in Neurosciences, 22(5):208-215, 1999.
Gordon W. Arbuthnott and Jeff Wickens. Space, time and dopamine. Trends Neurosciences, 30 10.00.........22000020......
12.003. h (2):62 69, 2007. ISSN 0166-2236. doi: http:///wwc.eeeeerereee///////// w/www. www.sccienceiree.. wttp:rerr..ccccccrireeeeee
com
Dmitriy Aronov, Rhino Nevers, and David W. Tank. Mapping non-spatial dimension by the huppocampal-entorfee entorhinal circuit. Nature, 543(7647):719-722 Mar 2017. ISSN 1476-4687. doi: 10.11088/aturr221697
Pierre Baldi and Santosh S. Venkatesh. Number of stable points for spin-glasses and neural networks of higher orders. Phys. Rev. Lett. 58:913-916, Mar 1987. doi: 10. R0.1100/hyyselleettt ett.58..11. URL
https: www.kkappp...rg.oeoooooe 6o 1010
58. 18.9113 913
Xiaojun Bao, Eva Gjorgieva, Laura K. Shanahan, James D. Howard, Thorsten Kahnt, and Jay A. Gottfried. Grid-like neural representations support olfactory navigation of a two-dimensional odor space. Neuron,
Jun 2019. ISSN 0896-6273, doi: neuron..uu.
034. URL htt.ttttss://d...ooo
1016/1 neuron. 2019 03. 034
Alison L. Barth and James F A Poulet. Experimental evidence for sparse firing in the neocortex.
Trends in Neurosciences, 35(6):345-3555 2012. ISSN 01662236. doi: 10. 10.1006/j.tins.
Jacob L. S. Bellmund, Peter Gärdenfors, Edvard I. Moser, and Christian Doeller. Nav- igating cognition: Spatial codes for human thinking. Science, 362(6415) 2018. doi: 10. 1e2.///iince.aat6777.. 1 URL htRL/ttps::// //ttpp:://ww... www.nce..ccc.....o... org doi www.//////////ooooo//...//aaaae/ 1126/
sgieence 6766
10

---

Published as a conference paper at ICLR 2023
Alex Bellos.
He ate all
the pi
Japanese
man memorises to 111,700
digits.
The
Guardian, 2015.
URL https:/ /tpp:://www. thequardian www.guuaraa.aa.aaiiaa
Com
pi hrt.44000000aa.ay.....m/aoooooor
Mats Bengtsson. Higher Order Neural Newworks. Diane Publishing Company, 1990. ISBN 9780941375924, /ttpa///////////o.........o...a.
T. V P Bliss and A. R. Gardner-Medwin, Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation the perforant path. The Journal of Physiology, 232(2):357-374, 1973. ISSN 14697793, doi: 10.11 1/.p11//jhysiol.1999 19t..sp01...0777
Erik B Bloss, Mark Cembrowski, Bill Karsh, Jennifer Colonell, Richard D Fetter, and Nelson Spruston. Single excitatory axons form clustered synapses onto CAI pyramidal cell dendrites. Nature Neuroscience, 21(3):353-365 -3353,33,,-0 March 2018.
Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Liò, Guido Montu- far, and Michael Bronstein. Weisfeiler and Lehman Go Cellular: CW Networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 2625-2640. Curran Asso- ciates, Inc., 2021. URL https: https::/ www.ungggs.euuipp.r http://pr......aaaaeee///.// cccpppper/222///1e.....
Melina Paula Bordone, Mootaz M. Salman, Haley E. Titus, Blham Amini, Jens V. Andersen, Barnali Chakraborti, Artem V. Diuba, Tatsiana G. Dubouskaya, Eric Ehrke, Andiara Espindola de Freitas, Guilherme Braga de Freitas, Rafaella A. Gonçalves, Deepali Gupta, Richa Gupta, Sharon R. Ha, Isabel A. Hemming, Minal Jaggar, Emil Jakobsen. Punita Kumari, Navya Lakkappa. Ash- ley I.. Marsh, Jessica Mitlöhner, Yuki Ogawa, Ramesh Kumar Paidi, Felipe C. Ribeiro, Ah- mad Salamian, Suraiya Saleem, Sorabh Sharma, Joana M. Silva, Shripriya Singh, Kunjbihari Sulakhiya, Tesfaye Wolde Tefera, Behnam Vafadari, Anuradha Yadav, Reiji Yamazaki, and Constanze I. Seidenbecher. The energetic brain review from students to students. Jour- nal of Neurochemisstry, 151(2):139-16) 2019. doi: http:://doi.org/10.1111 1/jnc. 14829. URL do wwwp/bs////../oooooo 1ttp//10..11111ppe 14829
/tttp::/cnniieelibrrre
Anton Bovier and Véronique Gayrard. Lower bounds on the memory capacity of the dilute Hop- field model. In Nino Boccara, Eric Goles, Servet Martinez, and Pierre Picco (eds.), Cellular Automata and Cooperative Systems, pp. 55-66 Springer Netherlands, Dordrecht, 1993a. ISBN 978- URL
94-011-1691-6. doi: 10.
978 94 011 1691-11855
Anton Bovier and Véronique Gayrard. Rigorous results on the thermodynamics of the dilute Hopfield model. Journal of Statistical Physics, 720))79-112, Jul 1993b. ISSN 1572-9613, doi: 10. 10.1007/BF01044044. htlL:htts:://dooo orgpiiorggli 1tt.://F1.0000F11011 0480441
Timothy F Brady, Talia Konkle, George A Alvarez, and Aude Oliva. Visual long-term memory has massive storage capacity for object details. Proceedings of the National Academy of Sciences, 105 2008
A. Brombas, S. Kalita-de Croft, F. J. Cooper-Williams and S. R. Williams. Dendro-dendritic cholinergic excitation controls dendritic spike initiation in retinal ganglion cells. Nature Com- munications, 8(1):15683, Jun 2017. ISSN 2041-1723, doi: 10. 10.1003/ncomms15560 URL
https://doi www.boo..rrg1001 wwt////ommm///nmm/m.mo.. 15633
Jehoshua Bruck and Vwani P. Roychowdhury. On the Number of Spurious Memories in the Hopfield Model. IEEE Transactions on Information Theory, 36(2):393-397, 1990. ISSN 15579654, doi: 10.1 10.1109/18.5246
Stephen G. Brush. History of the lenz-ising model. Rev. Mod. Phys., 39:883 Oct 1967. doi: 10. N0.1103/Re URL httpl/ttssss/...eeee RevModPhys. 883
http::///nnnkii.... wps.ikkp.ps..... org/ wttp:///////.....///0/0

---

Published as a conference paper at ICLR 2023
Mare Brysbaert, Michaël Stevens, Pawel Mandera, and Emmanuel Keuleers. How many words do we know? Practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant's age. Frontiers in Psychology, 7, 2016. ISSN 1664-1078 doi: 10.3389/fpsy
URL wtt.http:// htsp://www... Frontierrssseoo http:///www........arooooooooo/r
#389/fpaye 25 01116
Thomas F Burns and Robert Tang, Detecting danger in gridworlds using Gromov's link condition. arXiv, 2022. doi: 10.4850/ARXIV.220..0000 06274
URL https:
Thomas Burns, Tatsuya Haga, and Tomoki Fukai, Multiscale and extended retrieval of associative memory structures in cortical model of local-global inhibition balance. eNeuro, 9(3), 2022.
doi: 10.1523/NEEURO.OO W00.EEEERROO0022--- 200.000002222220000000202 URL htt:::tttt://t www.we.eer....mm http://rr///ooo//////////
ENEURO 002.-00033-22 2022
György Buzsáki. Neural syntax: Cell assemblies, synapsembles, and readers. Neuron, 68(3)3622- 385, 2010. ISSN 0896-6273. doi: http://ww..ag///..oood..annoooo0 URL https
/www. www.sccienediie... rect com wttp:///wc.........../a////////.
Rishidev Chaudhuri and Ila Fiete. Computational principles of memory. Nature Neuroscience, 19(3): 394-403, March 2016.
Rishidev Chaudhuri and lla Fiete. Bipartite expander Hopfield networks as self-decoding high- capacity error correcting codes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32, Cur- Associates, Inc., 2019. URL ht.th htttttpp:s//prroceeeiingg. neur ps wwtppppppp/.........
Spyridon Chavlis and Panayiota Poirazi. Drawing inspiration from biological dendrites empower artificial neural networks. Current Opinion in Neurobiology, 70:1-10, 2021. ISSN 0959-4388. doi:
https://doi.org/10... http:////0....11....ooooo...0000 0000 URL https::/ wwweeeeeieiiieeeeeeoeooeoeeeeeee rect. com/ Computational Neuroscience.
Yuhan Chen, Shengjun Wang, Claus C Hilgetag, and Changsong Zhou. Trade-off' between multiple constraints enables simultaneous formation of modules and hubs in neural systems. PLoS Comput. Biol., 10022700 March 2013.
Peter H Chipman, Chi Chung Alan Fung, Alejandra Pazo Fernandez, Abhilash Sawant, Angelo Tedoldi, Atsushi Kawai, Sunita Ghimire Gautam, Mizuki Kurosawa, Manabu Abe, Kenji Sakimura, Tomoki Fukai, and Yukiko Goda. Astrocyte GluN2C NMDA receptors control basal synaptic strengths of hippocampal CAI pyramidal neurons in the stratum radiatum. Elife, 10, October 2021.
Kimberly M. Christian and Richard Thompson. Neural Substrates of Eyeblink Conditioning: Acquisition and Retention. Learning and Memory, 1066):47-455, 2003. ISSN 10720502. doi: 10.1101/lm.59603.
James Clift, Dmitry Doryn, Daniel Murfet, and James Wallbridge. Logic and the 2-simplicial transformer. In International Conference on Learning Representations, 2020. URL https
//openreview.co net. wtt.::
Claudia Clopath, Tobias Bonhoeffer, Mark Hübener, and Tobias Rose. Variance and invariance of neuronal long-term representations. Philos Trans Soc Lond B Biol Sci, 372(1715), March 2017.
Alexandra O. Constantinescu, Jill X. O'Reilly, and Timothy E. J. Behrens. Organizing con- ceptual knowledge humans with a gridlike code. Science, 352(6292); 1468, 2016. doi: 10.11126/ciencceaaaf0000 URL https: httpps://wwrrr /wwww www.sccieeccc..m.co org doi http///i//aabbbbooae iscience.
Ana Covelo and Alfonso Araque Neuronal activity determines distinct gliotransmitter release from single astrocyte. eLife, 7:e3233,, Jan 2018. ISSN 2050-084X. doi: 10.7554/cLife.322377 URL
htt.ps://df..org/ 7554 /1.54/eeifff 32237
12

---

Published as a conference paper at ICLR 2023
Christopher J. Darwin, Michael T. Turvey, and Robert G. Crowder. An auditory analogue of the Sperling partial report procedure: Evidence for brief auditory storage. Cognitive Psychology, 3(2): 255-267, 1972. ISSN 00100285. doi:
Nathaniel D. Daw and Kenji Doya. The computational neurobiology of learning and reward. Current Opinion in Neurobiology, 16(2):199 199-204, 2006. ISSN 09594388. doi: 10. 10.1016/jj coni.. 2006.03.000
Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On model of associative memory with huge storage capacity, Journal of Statistical Physics, 168(2):288-22)) 299, Jul 2017. ISSN 1572-9613. doi: 10. 10.11077/100555-07-18880
URL https: /ttK:/tttt////...o//////// org/10
Anne Didier, Alan Carleton, Jan G. Bjaalie, Jean-Didier Vincent, Ole Petter Ottersen, Jon Storm-
Mathisen, and Pierre-Marie Lledo. dendrodendritic reciprocal synapse provides recurrent
excitatory connection in the olfactory bulb. Proceedings of the National Academy of Sciences, 98 (11):6441-6466, 2001. doi: 11.0033ppaa.101122663
URLhttps www., pnas org/doi/
hoss/10.1073/pm80 101126398
Adi Doron, Alon Rubin, Aviya Benmelech-Chone Chovav, Netai Benaim, Tom Carmi, Ron Refaeli, Nechama Novick, Tirzah Kreisel, Yaniv Ziv, and Inbal Goshen. Hippocampal astrocytes encode reward location. Nature, Aug 2022. ISSN 1476-4687. doi: 10. 1038//4418660022222
URL
https://dop .rrgggrgoo.... 0088/38800 / 1586 022 111222-55111 4
Gérard Dreyfus, Isabelle Guyon, Jeer-e Nadal, and Léon Personnaz. High order neural networks for efficient associative memory design. D. Anderson (ed.), Neural Information Processing Systems. American Institute of Physics, 1987. URL URL.ttpps:/pppcceeiiinos htw.:occeeeinnn....i.......ppppp cc/
Stefania Ebli, Michaël Defferrard, and Gard Spreemann. Simplicial neural networks. In NeurIPS 2020 Workshop on Topological Data Analysis and Beyond, 2020. URL htRp:tttps////eeeeeee net/ pett/forum2iddmPCt3330
Mária Eresey-Ravasz, Nikola T. Markov, Camille Lamy, David C. Van Essen, Kenneth Knoblauch, Zoltán Toroczkai, and Henry Kennedy. A predictive network model of cerebral cortical connectivity based on distance rule. Neuron, 80(1):184-197, 2013. ISSN 0896-6273. doi: https://doi.
http::ts: htttpp::/ww..sceeneeeee www.ncediiiccc...com..m....cc... rect. com/ science/
Drricle//11/5088223333300
Sarah J Etherington, Susan Atkinson, Greg J Stuart, and Stephen R Williams. Synaptic In- tegration, chapter 1, pp. 1-15. Wiley, 2010. ISBN 9780470015902.
doi: http::/doi.org/ wiley.com
00000000777004000019022000000000 pub2. hol/itttts::/onnneeeeeee /ttp///00....10002/7770000001111
pub2
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks.
In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3558-3563 2019.
Viola Folli, Marco Leonetti, and Giancarlo Ruocco. On the maximum storage capacity of the Hopfield model. Frontiers Computational Neuroscience, 10, 2017. ISSN 1662-5188. doi: 00139//fncom.2066.00111 10. 33389/fncom...... 00144
URL httpp://www..ttttteesss....ooem
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193-00, Apr 1980. ISSN 1432-0770.
doi: 10. 00.1007/BF00344251. URL JtLL/h/cpps://6b..r///000.000
BF BF00344251
Yuri Geinisman, Robert W. Berry, John F. Disterhoft, John M. Power, and Eddy A. Van der 7ee. Associative learning elicits the formation of multiple-synapse boutons. Journal Neuroscience, 21(15)55568-5573 2001. ISSN 0270-6474. doi: H0.522//NUUROC112--5555558888
URL
https://www www. btt.:p//wwww...rrrrrr.ooooorrrrs
Joydeep Ghosh and Yoan Shun. Efficient higher-order neural networks for classification and function approximation. International Journal of Neural Systems, 3(4): 1992. doi: 1422
$012906579200055. wtl.ttte....oooo.oooo/////////20
13

---

Published as a conference paper at ICLR 2023
Paul E. Gilbert and Raymond P. Kesner. Memory for objects and their locations: The role of the hippocampus in retention of object-place associations. Neurobiology of Learning and Memory, 81 (1):39-45, 2004. ISSN 10747427. doi: 10.1016//11074-72270000000000000
Nace L. Golding, Nathan P. Staff, and Nelson Spruston. Dendritic spikes mechanism for cooperative long-term potentiation. Nature, 416 Jul 2002. ISSN 1476-4687, doi: 10.1038/naturc000554 URL
A. M. Gordon, G. Westling, K. J. Cole, and R. S. Johansson. Memory representations underlying mo- tor commands used during manipulation of common and novel objects. Journal of Neurophysiology. 69(6):179-1177,, 1993. ISSN 00223077. doi: 10.1152/jn. 1993,69 69.6.11.... 1789.
Giorgio Gosti, Viola Folli, Marco Leonetti, and Giancarlo Ruocco. Beyond the maximum storage capacity limit in Hopfield recurrent neural networks. Entmopy, (8), 2019. ISSN 1099-4300. doi: 1099 10.3390/e21080726. UtLhttpss:/ www /ttp:///m.0000333//////////
I. S. Gradshteyn and I. M. Ryzhik. Table of integrals. series, and products. Elsevier/Academii Press, Amsterdam, seventh edition, 2007. ISBN 978-0-12-3736333 the Russian. Translation edited and with a preface by Alan Jeffrey and Daniel Zwillinger.
12--733337-4 Translated from
Eva-Maria Griesbauer, Ed Manley, Jan M. Wiener, and Hugo J. Spiers. London taxi drivers: A review of neurocognitive studies and an exploration of how they build their cognitive map of London. Hippocampus, 32(1):3 20, 2022. doi: http::///ooioor//111/ wttp://0o.............//........ www///00ohh//339999/ 233995. URL https
//online iibrar.... wi ley com wtt.c/ccccoooaeeccceee
10. www.//ipo...p..... 23395
M. Griniasty, M. V. Tsodyks, and Daniel J. Amit. Conversion of Temporal Correlations Between
Stimuli to Spatial Correlations Between Attractors. Neural Computation, 5(1):1-17, 01 1993. ISSN 0869-7667. doi: 10.1 11l166///o..19.......1 1993.5.1.1. URL httpt::tt////oo org 11.2:///e..6622ecc 1993.5.1.1
Vincent Gripon and Claude Berrou. Sparse neural networks with large learning diversity. IEEE Transactions on Neural Networks, 2277):1087-1096 2011. ISSN 10459227. doi: 10.1 109.TNN.
2011.2146789.
Lukas N. Groschner, Jonatan G. Malis, Birte Zuidinga, and Alexander Borst. A biophysical account of multiplication by single neuron. Nature, 603(7899):1199-123 Mar 2022. ISSN 1476-4687, doi: 10. 1038/s41586-022 028-6-0222-442---3 /ttp:////////.......000000////1.
Zengcai V. Guo, Hidehiko K. Inagaki, Kayvon Daie, Shaul Druckmann, Charles R. Gerfen, and Karel Svoboda. Maintenance of persistent activity in frontal-hhalmmocortiaaa loop. Nature,
545(7653) 181-186.11111.... May 2017. ISSN 1476-4687. doi: 10. 10.1038/maturre2234.. URL https 1000o/.00/1111108////1..
Bengt Gustafsson and Holger Wigström. Physiological mechanisms underlying long-term potentiation. Trends in Neurosciences, 11(4):156-162, 1988. ISSN 01662236. doi: 10.1016/0166-2236688) 90142-7.
Tatsuya Haga and Tomoki Fukai. Extended temporal association memory by modulations of inhibitory circuits. Phys. Rev. Lett., 123:078101, Aug 2019. doi: 10. 10/1 123.078101. URL
https:///imm aps org/ httg:/co/110/
123. 127.7111111111111111111
Mustafa Hajij, Kyle Istvan, and Ghada Zamzmi. Cell complex neural networks. In NeurIPS 2020
Workshop on Topological Data Analysis and Beyond, 2020. URL htttp:://pperevee
net//oor wtt.//ormmmmmddd1e. (tt.
Eric Hart and Alexander C. Huk. Recurrent circuit dynamics underlie persistent activity in the macaque frontoparictal network. eLife, 9xe52460, May 2020. ISSN 2050-084X. doi: 10.7554/ el.ife.52460. URL.tttp.a.. //tpal//////o.eeee www.oii.ooo//...ooooooo/// 2tt.//10..555eee.. 75.4/eLi 524460 32379044[pmidj.
Nathan G. Hedrick, Zhongmin Lu, Eric Bushong, Surbhi Singhi, Peter Nguyen, Yessenia Magaña, Sayyed Jilani, Byung Kook Lim, Mark Ellisman, and Takaki Komiyama. Learning binds new inputs into functional synaptic clusters via spinogenesis. Nature Neuroscience, 2566)726-777. Jun 2022. ISSN 1546- 1726. doi: 10. 000333/4411330222-0100000 1038/ 1038/s41593-022--1111
URL
14

---

Published as a conference paper at ICLR 2023
Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-Kep Key normalization for Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020. pp- 4246-4253 Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18663/v11 1/20/00222222 wtw.a/o.iinninngeennn....
URL bamcolo ogy www.aant..oooooooo//222o00/2
Findings-emn 379.
Suzana Herculano-Houzel The human brain in numbers: linearly scaled-up primate brain. Frontiers in Human Newroscience, 3, 2009. ISSN 1662-5161. doi: 10.338%/meuro.00.031.222 URL https: www. .wwowfrroma Frontiers wtt.:///t/o//teee/..ooo....
Christopher J. Hillar and Ngoc M. Tran. Robust exponential memory in Hopfield networks. The Journal of Mathematical Neuroscience, 8(1):1, Jan 2018. ISSN 2190-8567. doi: 10.1186/
33408-017-0006-22
Heiko Hoffmann. Sparse associative memory. Neural Computation, 31(5))998-1011 2019. ISSN 1530888X. doi: 10.1162/neco 01181.
Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, and Dmitry Krotov. A universal abstraction for hierarchical Hopfield networks. In The Symbiosis of Deep Learning and Differential Equations II, 2022. /tt//ccccas///eeeeeeeee..eeeeeee
J J Hopfieid. Searching for memories. Sudoku, implicit check bits, and the iterative use of not-always- correct rapid neural computation. Neural Compulation, 2065):111--1164 2008. ISSN 08997667. doi: 10. 1.221//cocco000.. 00000.0990--355.
J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2534 2558, 1982. doi: 10. 10.1007//ps.7777 1073/pnas
8.2554. URL htl.stt::: wwwp///wwww/..//// pnas. http://www.paa.../////////////// 10.. 79, 8 2554
J Hopfield. Neurons with graded response have collective computational properties like those of two- state neurons. Proceedings of the National Academy of Sciences, (101)))3888 1984, doi: 10. 1073/pnas.81. 81. URL
10. 3088
Hor, D. and Usher, M. Capacities of multiconnected memory models. J. Phys. France, 49))c, 395, 1988. doi: 10. 10.105 10011///ppyyx1111884400333333030 jphys: 018118888044038800
URL https:://dl..or///0000
Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rac, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020. URL
https:://peereviiww net/ foruf
Jenkins, AJ Dowsett, and AM Burton. How many faces do people know? Proceedings of the Royal Society B, (155(88))20118 111011, 2018.
EG Jones and TPS Powell. Morphological variations in the dendritic spines of the neocortex. Journal of cell science, 5(2):509-529, 1969.
Mareus Kaiser and Claus C Hilgetag. Nonoptimal component placement, but short processing paths, due to long-distance projections in neural systems. PLOS Comput. Biol., 2(7):e95, July 2006.
I. Kanter. Asymmetric neural networks with multispin interactions. Phys. Rev. A, 38:5972-5975, Dec 1988. doi: 10.1 1..1100// URL httpps//iiikaaas.coooo http:////......op..ooooooo/o///. PhysRevA. 38. 5972
Ayaka Kato, Kazumi Ohta, Kazuo Okanoya, and Hokto Kazama. Dopaminergic neurons dy-
namically update sensory values during navigation. bioRxiv, 2022. doi: 10.1100/202.00. 17.504092. UtL wwtp::wwwww.w......... htt.://wwwwww/aaaaooo//oo///////
08. 17. 504092
15

---

Published as a conference paper at ICLR 2023
Ege T. Kavalali, Jürgen Klingauf, and Richard W. Tsien. Activity-dependen regulation of synaptic clustering in a hippocampal culture system. Proceedings of the National Academy of Sciences, 96 doi: 10.1073/pnas.96.... 2tt...........9...... URL https: /www pnws org/
(2):12893-129000 1999.
doi/abs/10.. 1073/ pnas. . 96 22. 12893
Do Hyun Kim, Jinha Park, and Byungnam Kahng. Enhanced storage capacity with errors in scale-free Hopfield neural networks: An analytical study. PLoS ONE, 12(10), 2017. ISSN 19326203. doi:
10.137 woo.o/aalllllooe.....o 0anee8188....
Leo Kozachkov, Ksenia V. Kastanenka, and Dmitry Krotov. Building Transformers from neurons and astrocytes. bioRxiv, 2022. doi: 2101110/2222001..51111000 URL https: www.wxxwwww.ivvv 2tt///otteen//aaal11/22////////0
10. 12. 511991
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009.
Dmitry Krotov and John Hopfield. Dense associative memory for pattern recognition. In Proceed- ings of the 30th International Conference Neural Information Processing Systems, NIPS' 16, pp. 1180-1188, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Dmitry Krotov and John J. Hopfield. Large associative memory problem neurobiology and machine learning. In International Conference Learning Representations, 2021. URL https:
//openrevies www.wwwwww....................ew net. http://wwww.eeeeette/eeeeeeeem.e /tt////ddmmmmyyym
Anders Lansner. Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations. Trends Neurosciences, 1783)1178-186, 2009. ISSN 01662236. doi: 10.1016/j tin...ins.008... 12.002.
Ya Le and Xuan Yang. Tiny ImageNet visual recognition challenge. CS 231N, 7(7):3, 2015.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2277 22224, 1998. doi: 10.1 (0.1109/5.72679)
Yann LeCun, Corinna Cortes, and Christopher Burges. MNIST handwritten digit database, 2010.
URL http:://ytnnnn..... lecun. www./oom////e//ee htmm/excbmmmrc st
S. J. Lederman and R. L. Klatzky. Haptic perception: A tutorial. Attention, Perception, and Psychophysics, 71 1137):149-1145, 2009. ISSN 19433921. doi: 10.3758/APP..1.7.U
Kea Joo Lee, In Sung Park, Hyun Kim, William T. Greenough, Daniel S. Pak, and Im Joo Rhyu. Motor skill training induces coordinated strengthening and weakening between neighboring synapses. Journal of Neuroscience, 33(23):9794 2013. ISSN 0270-6474. doi: 10.1523/
INEUROSCI.088-- 12.2013. URL https: www. wwwp:wwwwwww..................ww jneurosci org/ http://oonentt3/33e
9794
Y. C. Lee, Gary D. Doolen, H. H. Chen, G. Z. Sun, T. Maxwell, H. Y. Lee, and C. Lee Giles. Machine learning using higher order correlation networks. In Proceedings of the "Evolution, Games, and Learning" Conference, 1986.
Martijn A. R. Leisink and Hilbert J. Kappen. Learning in higher order Boltzmann machines using linear response. Neural Networks, 13:2000, 1999.
Francesca Elisa Leonelli. Elena Agliari. Linda Albanese, and Adriano Barra. On the effective initiali- sation for restricted Boltzmann machines via duality with Hopfield model. Neural Networks, 143: 314-326, 2021. ISSN 0893-6080. doi: http:///o.........aaaaaaa///////
06.0077. URL WRR.ttts.
www rect com
M. Leonetti, E. Hörmann, L. Leuzzi, G. Parisi, and G. Ruocco. Optical computation of spin glass dynamics with tunable complexity, Proceedings of the National Academy of Sciences, 118(21): c2015207118, 2021. doi: abs/ 0tts/10..173//a0 20100000.....021111111
118. URL htthttts::///t pnws www.:paaaa.aaaoa.ooo org/doi/
Lek-Heng Lim. Hodge Laplacians on graphs. SIAM Review, 62(3):685-715, 2020. doi: 10.1137/
18M1223101. URL https: //doi .org / 8M1 223101
16

---

Published as a conference paper at ICLR 2023
W.A. Little. The existence of persistent states in the brain. Mathemattiall Biosciences, 19(1):011-100
1974. ISSN htt.p///w.....oeeoooo//////////a
doi: http:///.aooy///0000000025554441 URL https
wwww:.wcieeeee www..sclencedirecttt com/
Matthias Löwe and Franck Vermet. The Hopfield model on sparse Erdös-Renyi graph. Journal of
Statistical Physics, 143(1):005-214, Apr 2011. ISSN 1572-9613, doi: 10. 10.10077:10995--11-01066
Matthias Löwe. On the storage capacity of Hopfield models with correlated patterns. The Annals of Applied Probability, 8(4):1216 1250, 1998, doi: 10.1214/aap//1028800
URLhttpa
///oo.orr//0/ httpp//..1111////111/00000000000
N. J. Mackintosh. Conditioning and associative learning. Oxford University Press, Oxford, 1983.
Stephen Maren. Neurobiology of Paviovian Fear Conditioning. Annual Review of Neuroxcience, 24 (1):897-931, 2001. ISSN 0147-006X. doi: 00.114///mmurev.eurr.2222
D Marr. Simple memory: theory for archicortex. Philos Trans R Soc Lond B Biol Sci, 262(841): 23-81, July 1971.
Chiara Marullo and Elena Agliari. Boltzmann machines as generalized Hopfield networks: A review of recent results and outlooks. Entrpyy 23(1), 2021. ISSN 1099-4300. doi: 10.3390/6230100334 URL (ttts:tp: // wwww http://wwwwc..//////.3///.//////
William Mau, Michael Hasselmo, and Denise Cai. The brain in motion: How ensemble fluidity
drives memory-updating and flexibility, eLife, 9.e63550, Dec 2020. ISSN 2050-084X. doi:
10.7554/eLife.63555 URL http::tt/./o...og///////////
7554 watas/eLife... 63550
R. McEliece, E. Posner, E. Rodemich, and S. Venkatesh. The capacity of the Hopfield associative memory. IEEE Transactions on Information Theory, 33(4):461-48) 1987. doi: 10.1 1057328.
Frances McSweeney and Eric S. Murphy (eds.). The Wiley Blackwell Handbook of Oper- ant and Classical Conditioning. John Wiley & Sons, Ltd, Oxford, UK, may 2014. ISBN 9781118468135 doi: 10.1002/97811184681133 URL BRLhhtp::// /attp://doi... wiley. 978111846811
Roland Memisevic and Geoffrey E. Hinton. Learning to Represent Spatial Transformations with Factored Higher-Ordes Order Boltzmann Machines. Neural Computation, 227))117331142 06 2010. ISSN 0899-7667, doi: 10. 10.1162/mcc.2200001-0--9999 neco 2010 0 01-09-
URL /RL/:tttps//dd...o///0... /tt.:////....000111///133
Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Remi King. Toward realistic model of speech processing in the brain with self-supervised learning, 2022. URL
01685
Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Univer- sal Hopfield networks: A general framework for single-shot associative memory models, 2022.
URL wttphhtts///xxi...aato hwtp:////////oo//o//o//oo/ss//// 22w.. 04557
B. Milner. Amnesia following operation the temporal lobes. In C. Whitty and O. Zangwill (eds.), Ammesia, pp. 109-133. Butterworth, London, 1966.
G.A. Milner. The Magical Number Seven, Plus or Minus Two: Some Limits Our Capacity for Processing Information. Psychological Review, 101(2)3343335) 352, 1955. URL http: /ttt::/ssee /spider / apa. org/
html
Rogier Min, Mirko Santello, and Thomas Nevian. The computational power of astrocyte medi-
ated synaptic plasticity. Frontiers in Computational Neurosccenee, 6, 2012. ISSN 1662-5188. doi: 10.3389/incom. 2tt.ccmmm.12200099... URL http::://ww....tiiiiiietttse/eee
10. 3389/ fncom. 2012. 00093
17

---

Published as a conference paper at ICLR 2023
Ali A. Minai and William B. Levy. The dynamics of sparse random networks. Biological Cybernetics, 77 1993. ISSN 03401200. doi: 10.1007/BF00200831.
A. A. Mofrad and M.G Parker. Nested-clique network model of neural associative memory. Neural Computation, 29: 19,11-1869, 2017.
Gianluigi Mongillo, Omri Barak, and Misha Tsodyks. Synaptic Theory of Working Memory. Science, 31965869) 2008. ISSN 10959203. doi: 10.1126/sceence. 15tii
Michael G Müller, Christos Papadimitriou, Wolfgang Maass, and Robert Legenstein. A model for structured information representation in neural networks of the brain. eNeuro, 7()).,, 7(3):EEEUO.0053: 19. 19.2020, May 2020.
Kaoru Nakano. Associatron-a model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, SMC-2 1972. doi: 10.1109/TSMCC.1972.59999
Charles M Newman Memory capacity in neural network models: Rigorous lower
bounds. Neural Networks, (3):223-2368 1988. ISSN 0893-6080. doi: https://do..or//
00.1006/0093-6680008899003333330
(ttthhtts:://// http::/wwww..cceenc..eee.....e rect. com/ science/
Christos H. Papadimitriou, Santosh S. Vempala, Daniel Mitropolsky, Michael Collins, and Wolfgang Maass. Brain computation by assemblies of neurons. Proceedings of the National Academy of Sciences, 117(25):14444144777 2020. doi: 10. (073//pas.200189333 pnas . wttp:///gg//oooooottttto /ttps/////////a//o/ooooo httpa//bs....111..ae........o 1073/ppaas.20.1131111 2001893117
URL htLhtts:: /www
Seongmin A. Park, Douglas S. Miller, and Erie D. Boorman. Inferences multidimen-
sional social hierarchy use a grid-like code. Nature Neuroscience, 24(9):1292-1301, Sep 2021. ISSN 1546- 1726.
doi:
URL http:: http::s:///oii.og///////
s41593
Gertrudis Perea, Marta Navarrete, and Alfonso Araque. Tripartite synapses: astrocytes process and control synaptic information. Trends in Neurosciences, 32(8):421-431, 2009.
Peretto and J Niez. Long memory storage capacity of multiconnected neural networks.
Biol. Cybern., 54(1):53 64, may 1986. ISSN 0340-1200. doi: 10. 10.1007/BF00337715. URL
https::/doi 1tt.:////00111000000100111111111
Giovanni Petri and Alain Barrat. Simplicial activity driven model. Phys. Rev. Lett. 121:228301,
Nov 2018. doi: 10.1 10.100//hyeeeeLett1221..
URL
10. 10.1133/tyyReevettt.2.... 121.22230...11
Brad E. Pfeiffer and David Foster. Autoassociative dynamics in the generation of sequences of hippocampal place cells. Science, 349(6244):180-883 2015. ISSN 10959203. doi: 10.1126/
science.aaa9633.
Didier Pinault, Yoland Smith, and Martin Deschênes. Dendrodendritic and axoaxonic synapses the thalamic reticular nucleus of the adult rat. Jourmal of Neuroscience, 17(9)3215-32333 1997. ISSN 0270-6474. doi: 10. 10.11//IEEUOOSIIIII.... 17-00000....-.001.........1
Utt.ttps::/// www. jneurosci
oxc:/conntent/177933333
Fernando Pineda. Generalization of back propagation to recurrent and higher order neural net- works. In D. Anderson (ed.), Neurul Information Processing Systems. American Institute of Physics, 1987. URL pat.cceefecccs11eeeeeeeeeeeeepdd
Panayiota Poirazi and Athanasia Papoutsi. Illuminating dendritic function with computational models. Nature Reviews Neuroscience, 301 Jun 2020. ISSN 1471-0048. doi: 1038/ 0001553300---0 0301-7
541583-020-0301-7 URL http::ttts//cc..tm http:////ooooo///oo///ooooo 2200000077.....pp./////...///...
Panayiota Poirazi, Terrence Brannon, and Bartlett W. Mel. Pyramidal neuron as two-layer neural network. Neuron, 9766)989-999, 2003. ISSN 0896-6273. doi: https://oi.org/10
10166908866662233330044449
URL http:///ww///////e//e////eeeeeee http:///wwwwwccceeeeee...ttt com/ wwwpsciccsccc////cccccccccc
ntticclee/i1130086277000000
18

---

Published as a conference paper at ICLR 2023
Alon Poleg- Polsky and Jeffrey S Diamond. NMDA receptors multiplicatively scale visual signals and enhance directional motion discrimination in retinal ganglion cells. Neuron, 896))1277-1290,, 2016.
Melissa A. Preziosi and Jennifer H. Coane. Remembering that big things sound big: Sound symbolism and associative memory. Cognitive Research: Principles and Implications, 2(1), 2017. ISSN 2364.7464. doi: 10.1 0211166611235-06--00000
Vinu Varghese Pulikkottil, Bhanu Priya Somashekar, and Upinder S. Bhalla. Computation wiring, and plasticity in synaptic clusters. Current Opinion in Neurobiology, 70:101-112, 2021. ISSN 0959-4388. doi: https//oik.ogg///.001/....o.2.2. 001. URL https:// www. www.scienccce rect. putational Neuroscience.
Com-
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks all you need. In International Confer- ence Learning Representations, 2021. URL hatt::::/ppenrreiee..nttt/// fim
EIg9RnzIico
Nelson Rebola, Mario Carta, and Christophe Mulle. Operation and plasticity of hippocampal CA3 circuits: Implications for memory encoding. Nature Reviews Neuroscience, 18(4):209-221, 2017.
ISSN 14710048.
doi: 10.1008/mm..017.10
Max B. Reid, Lilly Spirkovska, and Ellen Ochoa. Rapid training of higher-order neural networks for invariant pattern recognition. In In Proc. International Joint Conference on Neural Nets, pp. 689-692, 1989.
R.A. Rescorla and A.R. Wagner. A theory of Pavlovian conditioning: Variations in the effective- ness of reinforcement and nonreinforcement. In A.H. Black and W.F. Prokasy (eds.), Classical Conditioning II: Current Research and Theory, pp. 64-99. Appleton Century Crofts, New York, 1972.
Iris Reuveni, Sourav Ghosh, and Edi Barkai. Real time multiplicative memory amplification mediated by whole-cell scaling of synaptic response in key neurons. PLOS Computational Biology, 13 (1):1-31, 01 2017. doi: 10.1371/joumall.ceh pcb....cii.0066..... 1005306. URLhttpss itts://:///aoi http://a....../.111.331101 Sournal.ped 1005306
Mark Rigby, Federico Grillo, Benjamin Compans, Guilherme Neves, Julia Gallinaro, Sophie Nashashibi, Gema Vizcay-Bareena, Florian Levet, Jean-Baptiste Sibarita, Angus Kirkland, Roland A. Fleck, Claudia Clopath, and Juan Burrone. Multi-synaptic boutons are fea- ture of cal hippocampal connections that may underlie network synchrony. bioRxiv, 2022. doi: 10. 10100022.00...... 0...........00666 htttss/ts::///ttt http://wwwwbxx.xxoo/oo// org/ content/earlyl 2022 00/2/055/0 30/ 2022.22222222 05 30. 493836
Daniel S. Rizzuto and Michael J. Kahana. An autoassociative neural network model of paired- associate learning. Neural Computation, 13(9):2055-2092 2001. ISSN 08997667. doi: 10. 116/
0889966001503999117
Jacopo Rocchi, David Saad, and Daniele Tantari. High storage capacity in the Hopfield model with auto-interactions-stthee stability analysis. Journal of Physics A: Mathematical and Theoretical, 50(46): 465001, 2017.
T. Mitchell Roddenberry, Nicholas Glaze, and Santiago Segarra. Principled simplicial neural networks for trajectory prediction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9020-9029, PMLR. 18-24 Jul 2021. URL https: https::/pzoceeeingss. mlr
html
Fernando E. Rosas, Pedro A. M. Mediano, Andrea I. Luppi. Thomas F, Varley, Joseph T. Lizier, Sebastiano Stramaglia, Henrik J. Jensen, and Daniele Marinazzo. Disentangling high-order mechanisms and high-order behaviours in complex systems. Nature Physics, Mar 2022. ISSN 1755-2481, doi: 10. 10.1033/411577-02--155833 s41 041567-022-01588-
URL htt ott////.oo////.11100
19

---

Published as a conference paper at ICLR 2023
Dmitri A. Rusakov and Dimitri M. Kullmann. Extrasynaptic glutamate diffusion in the hippocampus: Ultrastructural constraints, uptake, and receptor activation. Journal of Neuroscience, 18(9):
3158-3170, 1998. ISSN 0270-6474. doi: (0.552/NNERRSSCI18-00-1111.1 https:: www. jneurosci mrgg org/ coottntt//////3111/1
URL
Andrea Santoro, Federico Battiston, Giovanni Petri, and Enrico Amico. Unveiling the higher-
order organization of multivariate time series, 2022. URL htttts::///aatiti http://///axxaooo//a////////////
10702
Saratha Sathasivam and Wan Ahmad Tajuddin Wan Abdullah. Logic Learning in Hopfield Networks. Modern Applied Science, 2(3), 2008. ISSN 1913-1844. doi: 10.5539/aas.2n33p577
Michael T Schaub, Austin R Benson, Paul Horn, Gabor Lippner, and Ali Jadbabaie. Random walks on simplicial complexes and the normalized Hodge I-Laplacian. SIAM Review, 62(2):353-391,
2020.
Elad Schneidman, Michael J. Berry, Ronen Segev, and William Bialek. Weak pairwise correlations imply strongly correlated network states in neural population. Nature, 440170887:10077-101 Apr 2006. ISSN 1476-4687. doi: 10. 10.00//tturre0470... URL htl.ttp::: httpttsss//ooooor////0000 nature04701
W. B. Scoville and B. Milner. Loss of recent memory after bilateral hippocampal lesions. Journal of Neurolgy, Neurosurgery, and Psychiatry, 20(1):1 1957.
Philipp Seidl, Philipp Renz, Natalia Dyubankova, Paulo Neves, Jonas Verhoeven, Jörg K. Wegner, Marwin Segler, Sepp Hochreiter, and Günter Klambauer. Improving few- and zero-shot reaction template prediction using modern Hopfield networks. Journal of Chemical Information and Modeling, (229))2111-22221 2022. doi: http///cccci..../000
URL httpt:://poi.....tttttttptpttttt
10. 10.1022/acs jeim. 1c01065
PMID: 35034452.
Terrence J. Sejnowski. Higher-order Boltzmann machines. AIP Conference Proceedings, 151(1): 398-403, 1986. doi: 10.1063/1.36246. URL
10 10.3166/1 36246
Sugandha Sharma, Sarthak Chandra, and Ila Fiete. Content addressable memory without catastrophic forgetting by heteroassociation with fixed scaffold. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Inter- national Conference Machine Learning. volume 162 of Proceedings of Machine Learning mlr
Research, pp. 19658-19682, PMLR, 17-23 Jul 2022. URL press/vl622 wt/p//l12222/2 20222ahaama2222 html
Masatoshi Shiino and Tomoki Fukai. Self-consistent signal-to-noiss analysis the statistical behavior of analog neural networks and enhancement of the storage capacity. Phys. Rev. E, 48:867-897, Aug 1993. doi: 10. 100310yssse....4....o PhysRevE. 48.48888 867
URL htttps://iind wttp:pppooo//o///0.1110..
Matthew Smart and Anton Zilman. On the mapping between Hopfield networks and restricted Boltzmann machines. In International Conference on Learning Representations, 2021. URL
htttps::/openreviiee.
net teeee/oooummiiidRoeerrrrr
KE Sorra and KM Harris. Occurrence and three-dimensional structure of multiple synapses between individual radiatum axons and their target pyramidal cells in hippocampal area cal. Journal of Neuroscience, (19):3736-3744 1993. ISSN 0270-6474, doi: 10. 10.1523/NEUROSC.33.000000
1993. URL URL https::////...........ltt www. jneurosci. org/ wttp///tt/ttttt////////
George Sperling. A Model for Visual Memory Tasks. Human Factors: The Journal of Human Factors and Ergonomics Society, 5(1):19-31, 1963. ISSN 15478181. doi: 10.1 017770011228863000000
Reisa Sperling, Elizabeth Chua, Andrew Cocchiarella, Erin Sand-Giovannetti, Russell Poldrack, Daniel L. Schacter, and Marilyn Albert. Putting names faces: Successful encoding of associative memories activates the anterior hippocampal formation. Neurolmage, (020:1400-11111 2003.
ISSN 105381 19. doi: 10.
20

---

Published as a conference paper at ICLR 2023
David Spivak. Higher-dimensiont. models of networks. arXiv:0909.4314, 2009.
Larry R. Squire. Memory and the Hippocampuss: A Synthesis From Findings With Rats, Monkeys, and Humans, Psychological Review, 1952)19 1992. ISSN 0033295X. doi: 10.1037/
1133-95XX.99221199
Larry R. Squire, Arthur P. Shimamura, and David G. Amaral. Memory and the Hippocampus. Neural Models of Plasticity, pp. 208-239, 1989. doi: 10.1016/b978-00 1 22-48956-4.5001--44
Vivek H. Sridhar, Liang Li, Dan Gorbonos, Máté Nagy, Bianca R. Schell, Timothy Sorochkin, Nir S. Gov, and lain D. Couzin. The geometry of decision-making in individuals and collec- tives. Proceedings of the National Academy of Sciences, 116650))222001151111,, 2021. doi:
10.1073//pna.2202157111
htllhttps:/ htap:://www....
www.wwwww.....mm../. pwwp://wwwppp...p............... pnas org. http:///w...a....//////////a////
/paas
21021
Lionel Standing. Learning 10000 pictures. The Quarterly journal of experimental psychology, 25(2): 207-222, 1973.
J. C. Stanley. Simulation studies of temporal sequence memory model. Biological Cybernetics, 121 Sep 1976. ISSN 1432-0770. doi: lad::10.10000 10.1007/BF00334115 155. URL https://doi
org/10.
Amos Storkey. Increasing the capacity of Hopfield network without sacrificing functionality. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 1327:451-456 1997. ISSN 16113349, doi: 10.1007/b1b00201996
R. Juthhrrld and J. W. Rudy. Configural association theory: The role of the hippocampal formation in learning, memory, and amnesia. Psychobiology, 17(2):129-144 1989. ISSN 08896313, doi: 10.3755/BFF3337822.
A Treves and D Amit. Metastable states in asymmetrically diluted Hopfield networks. Journal of Physics A: Mathematical and General, 21(14):315--366, jul 1988. doi: 10. 00.10/03055-44002111 14/016. httpattps://oo..ooo wtt.://///....100000000000000000
Danil Tyulmankov, Ching Fang, Annapurna Vadaparty, and Guangyu Robert Yang. Biological learning in key-value memory networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34. pp. 22247-22255 Curran Associates, Inc., 2021. / CC/ httppaper/22211 http2///iiee/accccc2222ee6633332 oot.aaacce2eeaaee777777716666122 2ttpa///01.00000..66660000000000
ps
pdff
Michael T. Ullman. Contributions of memory circuits language: The declaratiie//poocedural model. Cognition, 92(1-2)2231-270 2004. ISSN 00100277. doi: c0006//.cognittion.000.... 10.008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need.
Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httpss///prccecdgggs iewrrerrooeeeeeneeonoe.oop.o...e mtt.ripp.ccc/pppe///0/////E hag.ee//aaacceeeeeeseeaaaaaaaaa-
Edward K. Vogel, Geoffrey Woodman, and Steven Luck. Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 27(1):92 2001. ISSN 00961523. doi: 10.1037/0096- 1523.27.1.9..
Richard A. Watson, C. L. Buckley, and Rob Mills. Optimization in "self-modeling" complex adaptive systems. Complexity, 16(5):17-26, 201 la. ISSN 10990526. doi: 10. 10.1002/cplx.203446
Richard A. Watson, Rob Mills, and C. L. Buckley. Global adaptation in networks of selfish compo- nents: Emergent associative memory at the system scale. Artificial Life, 17(3):147-166, 2011b. ISSN 10645462. doi: 10. 162/artt Latparrll__000000 00009.
Melanie Weber, Pedro D. Maia, and J. Nathan Kutz. Estimating memory deterioration rates follow- ing neurodegeneration and traumatic brain injuries in a Hopfield network model. Frontiers in Neuroscience, I1(O)), 2017. ISSN 1662453X. doi: 10.3389/fnins. 200.0/mmnn..17..00633....3
21

---

Published as a conference paper at ICLR 2023
Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitmer, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern Hopfield networks and attention for immune repertoire classification. Advances in Neural Information Processing Systems, 33: 18832-1845, 2020.
Tom J. Wills, Colin Lever, Francesca Cacucci, Neil Burgess, and John O'Keefe. Attractor dynamics in the hippocampal representation of the local enviromment. Science, 308(5723) 2005.
ISSN 00368075. doi: 10. 100//c//cceec 100iii..nee.1005...... 108905.
István Winkler and Nelson Cowan. From sensory to long-term memory: Evidence from auditory memory reactivation studies. Experimental Psychology, 52(1):3-20, 2005. ISSN 16183169, doi:
10.10277/1118-3199.5....
Alexander Woodward, Tom Froese, and Takashi Ikegami. Neural coordination can be enhanced by occasional interruption of normal firing patterns: A self-optimizing spiking neural network model.
Neural Networks, 62:39-46, 2015, ISSN 18792782. doi: 10.101 00.1016f..eeunet2211..888
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning, 2022.
oxg/abbs/224.0877
Ming Zhang. Artificial higher order neural networks for modeling and simulation. IGI Global, 2012. ISBN 978 1781466611756. doi: 10.4018/978- 17788-1466666 1-4666-217556
22

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PURPOSE OF STUDYING AND EXTENDING HOPFIELD NETWORKS
In the context of memory more broadly, associative memory is a form of long-term memory, memory which can potentially last a lifetime. However, memory also exists on at least two shorter and functionally distinct time-scales: short-term memory and sensory memory. Short-term working memory stores information for on the order of tens of seconds, and performs as a limited, passive
temporary memory reservoir with which to manipulate and use information (Milner 1955 Mongillo
cct al. 2008). Sensory memory operates on even shorter timescales than short-term memory, typically
on the order of only seconds. It is where visual (Sperling) 1963; Vogel et al. 2001). J, auditory (Darwin
iet al. 1972 Winkler & Cowan 2005), and other sensory information (Gordon et al., [993] Lederman
& Klatzky 2009) is first actively 'rmembeeeed all other information is either immediate sensory information or recalled information. A general schema for forming a long-term memory is therefore to: (I) receive external sensory information; (2) store the features of that sensory information in sensory memory; (3) manage and store one or more features of the sensory information (and/or combine it with other sensory information) in short-term memory; and then (4) consolidate this information into long-term memory. Understanding these processes from a theoretical and computational perspective has several real-world implications, including: (i) may allow us to create more intelligent machines, by gaining inspiration and insight from biological strategies to store, retrieve, and use long-term associative memories; and (ii) it may help us understand the ncurobiological mechanisms (and their implications) for biological memory systems, helping to not only understand related psychological and biological phenomena, but to potentially help identify therapeutic targets for related dysfunction.
Psychological abilities attributed associative memory in humans and non-human animals typically said to be any form of long-term memorisation which involves 'pairing' or 'associating' distinct stimuli such that when presented with one stimuli, the subject can recall the other stimuli. Classical examples of this type of associative memory include pairings: name-face pairs (Sperling al. 2003), object-sound pairs (Preziosi & Coane (017), and object-place pairs (Gilbert & Kesner 2004). These types of associative memories are part of explicit or declarative memory (Ullman (004)) i.e., long-term memory that can be explicitly or voluntarily stated or declared. This is in contrast to associative memories which are part of implicit or non non-declarative memory, i.e., long-term memory recalled or used unconsciously or unintentionally. Examples of this type of associative memory are generated by classical conditioning (Maren] 2001 Christian & Thompson 2003) and operant conditioning (Mackintosh]. 1983 McSweeney & 2014).
Computational accounts of implicit associative memory have a long and successful history starting from examples like the Rescorla- Wagner model (Rescorla & Wagner 1972). Today, this research has
grown into the computational field of reinforcement learning (Daw & Doya 2006). In comparison to
implicit associative memory, explicit associative memory seems more computationally sophisticated. as suggested by its complex biological bases (Chaudhuri & Fiete 2016 Clopath et al. 2017 Mau
et al. 2020).
Soon after the proposal of Marr (1971), one of the first and most influential computational models of (explicit) associative memory was the Hopfield model (Hopfield) 1982), which we study and extend here. A nice feature of this model is that in the basic case of embedding a single memory, it is easy to see there exists choice of threshold for every neuron whereby any partial activation of the memory will lead activation of all its other members. This therefore generates an attractor dynamic around the fixed point of the stored memory, which is comparable to the neuronal assembly attractor dynamics seen in the hippocampus (Wills et al. 2005 Pfeifter & Foster 2015] Rebola et al., 2017).
Hippocampus is probably the foremost brain structure involved in memory. It. together with the surrounding entorhinal, perirhinal, and parahippocampal cortices, is especially important for explicit memory (Scoville & Milner 1957; Milner 1966], (Sqiire) 1992). It is a necessary structure for the initial formation and learning of explicit memory, acting as a short-term memory for later long-term consolidation, thought to occur in cortex (Squire et al. 1989 Sutherland & Rudy 1989). Classically, we think these capacities are mainly achieved via Hebbian learning and long-term potentiation (Bliss) & Sardnerr-Medwin [973] Gustafsson & Wigström 1988). However, there is now increasing literature which shows how other mechanisms may help to achieve these memory functions (see Appendix A.2 for examples).
23

---

Published as a conference paper at ICLR 2023
As complete computational account of long-term memory storage, the classical Hopfield model encounters challenges. As discussed in Section [ of the main text, the memory capacity of the classical Hopfield network is linear in the number of neurons N, specifically: approximately 0.14N patterns may be stored before spurious attractors overwhelm workable levels of memory recall (Amit et al. 1985 McEliece et al., 1987 Bruck & Roychowdhury [990), and this capacity diminishes further when the patterns are statistically or spatially correlated (Löwe, 1998). in sparse connectivity
regimes (Treves & Amit 1988 Löwe & Vermet 201T), and in combination (Burns et al. 2022).
Biological networks typically have very sparse connectivity (Minai & Levy, | Lansner 2009
Barth & Poulet 2012) and everyday memory items typical share many statistical features, may have
sophisticated inter-relations, and are spatially or semantically correlated in non-trivial structures
(Constantinescu et al., | Aronov et al. 2017: Bellmund et al. 2018 [Bao et al. 2019 Park et al.,
2021] Griesbbuee et al., 2022). Despite this, humans can remember very high-fidelity information of thousands of statistically similar images (Standing) 1973; Brady et al. 2008) and human faces
(Jenkins et al.] 2018)) tens of thousands of linguistic items (Brysbaert al. 2016), and more than
100,000 digits of the number æ (Bellos) (015) - all, seemingly, without dramatically sacrificing or over-writing other memories.
Although modern Hopfield networks have substantially increased theoretical memory capacity (Krotov& & Hopfield 2016 Demireigil et al.] 2017). the combined biological and psychological evidence mentioned above, along with the finite (if large) number of brain cells (Herculano-Hoon Houzel 2009) and energetic demands of maintaining them and their inter-connections (Bordone et al./ (211)) suggest there may be more to the neurophysiological and computational story. Furthermore, even if we find that the theoretical memory capacity should still be high according to a Hopfield interpretation, capacity can be considered as measure of undesired interferences between memories, and thus may be maximised for cognitive convenience or speed and accuracy of memory recall.
Nevertheless, problems and criticisms don't 't detract from the usefulness and importance of the Hopfield model or its modern variations the study of memory systems (Sathasivam & Wan Abdullah 2008, |Rizzuto & Kahana 2001. Weber et al. 2017). usefulness in machine learning applications (Widrich et al. al.]| |0200 Seidl et al. 2022). contribution to more general machine learning (Sharma et al., 2022; Hoover et al. (2022). or connection between biology and machine learning (Chaudhuri & Fiete 2019] Tyulmankov et al. 2021 Kozachkov et al. 2022). There are even more opportunities to build upon this substantial foundation to create more sophisticated computational models of associative memory. Notably, much work has improved the efficiency and capacity of
the Hopfield network (Storkey 1997: Hopfieid 2008 Krotov & Hopfield 2016); Gripon & Berrou
2011 Mofrad & Parker| 70. Other work has focused on achieving sparse representations ( (Kim 2019) or including other forms of biological realism (Watson et al.) 201 101la| bj
al. 2017|Hooffman
Woodward et al. 20155; Burns et al. 2022). The current work contributes to developments in sparsity,
biological realism, and memory capacity.
A.2 SETWISE CONNECTIONS AND MODUIATIONS ARE BOUNTIFUL. IN BIOLOGY
Setwise connections are not limited to the case, one might expect, of multiple synaptic contacts
between pairs or sets of cells Jones & Powell 1969 Sorra & Harris 1993 Geinisman et al. 2001
Lee al. 2013 Rigby al. (022). which may result in multiplicative interactions Poleg-Polskk
& Diamond 2016; Reuveni et al., 2017 Groschner et al.] 2022) or form of functional synaptic
clusters (Kavalali al. 1999 Bloss al. 2018 Pulikkottil al. [202]] Hedrick et al., (022). Other
examples (some of which are illustrated in Figure 3) include the wide spatial dispersion of certain
neurotransmitters (Rusakov & Kullmann [9988 Arbuthnott & Wickens 2007] Kato et al., (022).
et al. 2017). persistently- connected neuronal assembly structures or 'synapsembles" (Buzsáki) 2010; Papadimitriou et al.
dendro-dendritic synapses (Pinault et al. 1997] et
2020). distributed persistent activity during activities such as motor planning and working memory (Guo et al., 2017 Hart & Huk 2020), neuroglial modulations of neurotransmitter release probabilities
across multiple neurons synapses (Min et al.) 2012 Covelo & Araque| 201 8; Chipman et al., 2021), 'tripartite' astrocyte- estrocyte-neuron
neuron synapses (Araque et al. 1999, Perea et al.) 2009), astrocytic coding (Doron et al.] 2022). and during dendritic integration at the level of individual neurons (Golding (ct al.] | Etherington et al. 2010). Modulations of and interactions between such connections are
illustrated in Figure
24

---

Published as a conference paper at ICLR 2023
It is also possible to 'functionally' construct setwise connections through only pairwise synapses. as shown in Krotov & Hopfield (202I). In one sense, this kind of pairwise-basee bassee reconstruction of setwise connections could also be thought of as similar results from Poirazi et al. (2003), who showed how multi-layer artificial neural networks can approximate more biologically -oogiccallysophhssiiaaeed model neurons with dendrites. Indeed, many of the known and suggested computational features of dendritic integration (Poirazi & Papoutsi, 20200, Chavlis & Poirazi (2021) may be considered as highly specialised and sophisticated forms of convolutions or setwise interactions.
A
B
ü.
(iki
GD
k
(ij.k
1
(ij.k)
C
D
(x, Xy, Mu
(j.k)
B.j
(o.eiiin yEY
xeX
Figure 3: A. Multi-synaptic bouton. B. Dendritic integration. C. Extra-synaptic neurotransmitter
diffusion. D. Functional connections between neural assemblies.
A
B
EiK
G.
<1k
C
4t
K.j.k
C
NMDA
C
D
m
#
(.jk)
G.Lmm (.j.k)
K
j.k.B
(klmmn)
n
Figure 4: A. Neurotransmitter depletion at synapse. B. NMDA-spikes and unequal synaptic strengths in dendritic integration. C. Transmission speed plasticity using myelination (top) and axon diameter (bottom) affect 'temporal' cemporal'/functionn /functiona setwise influence in post-synaptic cell. D. Astrocytic messaging, between both neurons and astrocytes.
25

---

Published as a conference paper at ICLR 2023
A.3
OPTIONS FOR MODELLIGG SETWISE CONNECTIVITY IN NEURAL NETWORKS AND WHY WE CHOOSE SIMPLICIAL COMPLEXES
In geometric and topological artificial intelligence and machine learning, recent advances have been realised by utilising higher dimensional analogues of graphs such as simplicial complexes (Ebli et al.
Roddenberry et al. 202T), cube complexes (Bums & Tang 2022). , cell complexes (Hajij et al. 2020 Bodnar et al. 2021 and hypergraphs (Feng et al. 2019 Xu et al. 2022). Unlike graphs, these
2020:
structures can naturally represent higher-degree, setwise relationships. However, not all structures are
appropriate for all systems (Spivak) 2009; Rosas et al. 2022).
Why do we choose to model our collections of setwise connections as weighted simplicial complexes and not use general cell complexes or hypergraphs? There are three main reasons:
1. Simplicial complexes allow all possible setwise relations to exist. Simplices, by construction, may span any number of vertices. This means any possible combination of neurons may share common, setwise weight. This is also possible in hypergraphs, but not possible in all complexes, e.g., cube complexes may not include triangles. The 'natural' complex for modelling all possible setwise relationships is therefore a simplicial onc. Since we also wish for our setwise weights to be symmetrical (i.e., have the same value when updating the spin with respect to each constituent neuron), it is unnecessary to include any more than one unique object per setwise relationship. This also makes the choice of a simplex suitable, since there can only be one unique simplex for a given set of vertices - which is also the case for edges in undirected hypergraphs (but we find these are less suitable).
2.
The sub-edge problem of hypergraphs makes them less suitable. Hypergraphs are graphs where edges may contain any number of unique vertices from the vertex set. a sense, these are more general structure than simplicial complexes and lack downward closure, e.g., if the edge {1, 2, exists, edges such as (2.3) or {1} do not necessarily exist, whereas {1.2.3] was simplex in a simplicial complex, simplices (2 and {1} exist. However, hypergraphs do not have well-defined 'sub-edges' (described as the 'sub-edge problem' in Remark 3.5 of Spivak )2099)) This has the consequence of defining interactions between 'levels' of hyperedges (setwise relationships) in hypergraphs slightly awkward. In contrast, simplicial complexes have a well-defined hierarchy of setwise relationships, partly due to the downward closure condition.
3. Downward closure of setwise connections is biologically plausible. Another benefit of downward closure in simplicial complexes is that it currently seems better supported from the perspective of biological plausibility (also see Appendix (A.)). For example, although it can happen that setwise connection (anatomical or functional) between neurons could exist without any underlying pairwise connections, the typical machinery used to create such setwise connections is sufficiently local to assume that, because of the functional local modularity of connections in the brain (Kaiser & Hilgetag| 2006) Chen et al., [203)] Müller et al. 2020 Earssey-Ravazz al. 2013), there is high probability of these neurons having a pairwise connection simply due to proximity.
As an additional practical henefit, simplicial complexes are currently more well-studied than structures such as hypergraphs (at least in some areas, e.g., spectral theories or (co)homology, which are of natural interest here), meaning that we can also take advantage of the relative maturity of the field in those areas- admittedly, we use very few advanced methods or properties in an essential way in this study, although we hope do so in future studies, having now introduced an initial interpretation of simplicial Hopfield networks and begun exploring some of their potential benefits. However, it will also be interesting to see what differences can be found between hypergraphic and simplicial Hopfield networks, and perhaps which provides a closer approximation to biology or which shows improved performance on certain tasks.
Among other possibilities, the weight of simplex w(a) could be modulated by the local energy of its spins S,, its coface's spins, or of those of simplices in the same dimension as a which are "neighbouring" (in the Hodge Laplacian sense (Lim, /020| Schaub et al. 20200). These interactions could take many different mathematical forms (Petri & Barrat 2018 Ebli et al. 2020 Roddenberry et al. 2021] Rosas et al.) 2022 Santoro et al., 2022). Neurobiologically. these interactions could represent neural-ghin interactions, glia-glia interactions, nonlinear dendritic integration (especially interactions, or hierarchical as-
dendritic spikes and shunting), saanntransmitter--uurmmmlll
26

---

Published as a conference paper at ICLR 2023
sembly operations and dynamics, to name few (see Appendix A.2 for illustrations and further information).
The downward closure of simplicial complexes could be seen as a disadvantage. For example, when including simplices of high dimension, we are also forced to limit our choices of simplices if we wish maintain the simplicial structure. Again, whenever k-simplex exists in K, all its faces must also exist, e.g., if a triangle exists, so must its surrounding edges and their surrounding vertices. any constituent simplex missing, the structure of the simplicial complex broken and in our case would become an undirected, weighted hypergraph. Instead, in our simulations, we prefer to interpret "missing simplices' as merely functionally missing-in-actionn by setting their weights zero if we do not wish to include them in the model. This has the consequence of having no mathematical effect on our update rules while retaining the convenience of simplicial structure.
A.4 A SMALL WORKED EXAMPLE
Consider small example of just P = 3 memory patterns embedded in simplicial Hopfield network on N neurons. First, let's consider a 3-skeleton, i.e., network without any 'dilution' or 'missing weights' up D 3. Such a network will have functional connections totalling
Σ 2=2
D+1
N d
+
15 + 20 15 = 50.
(10)
2
We typically do not include functional self-connections (autapses) although Hopfield networks with such networks have been studied (Folli al. 2017 Rocchi al. 2017; Gosti al.) (019). In
simplicial Hopfield networks, such self-connections correspond to 0-simplices, i.c., vertices. While these vertices do exist in the underlying simplicial complex K, we set their associated weights to O.
Given N = 6, the 3-skeleton in this example is
K₂ = ({1), (2), (3), (5), {6}},
K1
(2.3), (2, (2. (,,, (2.6),
K₂ = (5,1
(3,5,6),
K3 = [L.23K....2220000222222222221110
(3,4,5,6)) 6)},
K 0 K. UK, UK2
(11)
Set three patterns as
g\ -1,1,,11,1,,1,,,
_² --,,,,,,,,,,,,-
(12)
For all Ko. set w(a) 0. For all higher dimensions, set
P
w(a) = N
Σ eu
(13)
#=1
For example,
w((1,3)) =1/6 = (3,5,6)
([2,4,,5, -1/6- [| (14)
1/2
27

---

Published as a conference paper at ICLR 2023
Given set of spins 8 at time-step t, the network will evolve according to Equation minimising the energy shown in Equation [ with the product of their respective spins, e.g., if ( =
The energy function consists of sum of products of the weights
E
+ w/ ((1,3))
s(t) (1.3)
+ + w (3,5,6)) S(4) (3,5,6) + w/ (22,4,,5))) 500
(2,4,5,6) +...)
(... 1/6
1/6 -1 .... +1/2 1+...).
(15)
In this 3-skeleton case, then, the network's 's energy function can also be written as
(000 5
E
2
5
5
(16)
4=1
Σ S
5/ s(0
sto
Essentially, the energy function is similar to sum of the energy functions of Krotov & Hopfield (2016) with all possible k-neuron connections, but where the weights of those connections are independent of each other for each level of interaction, making each connection and each level more controllable. The memory capacity of this type of simiplicial Hopfield network is discussed in Section 2.3 and Appendix (A.6) However, one of our main contributions is the findings related to the 'diluted' case, i.e., where more than just the O-simplices have their weights set to 0. Indeed, these are the cases we mainly evaluate in Section 2.4
Following on with the same example as above, we can create a diluted simplicial Hopfield network based on K. For example, if we chose to limit ourselves to (o) 15 parameters, we could choose to apportion third of these parameters to each dimension, i.e., set weights only for subset K' C K, e.g.,
(17)
(,4,,,000000000,,,333,,2,,,,0,,, (23,,,,))
="=K² UKJKKK
In our numerical simulations, the choice of which connections to keep is entirely random. By analogy, we can think of this dilution procedure as a naive solution to the following (fairly contrived) communications problem: Imagine body of people, e.g., a very large committee, comes to its decisions. Currently, each committee member has individual channels of communication with every other member. This is good for high-fidelity, accurate, and nuanced conversations between members, but not so good for efficiency or speed of decision-making For example, if certain block of members consistently vote similarly, would perhaps be quicker for those members to communicate as a group to check what their majority opinion rather than all members individually communicating with every other member one at a time. Conversely, when two members consistently vote differently or are active members within distinct voting blocks (and especially if their votes are often nie-breakers). perhaps those two members ought to regularly discuss matters privately and in detail. Our naïve solution is to randomly replace some individual channels of communication with small group communication channels. Possibly by performing a survey of members or observing patterns in their voting or communications, we could come up with a better strategy. However, in this analogy, it appears the naive solution works reasonably well (see results in Section of the main text). We think deserving next step will be determine better strategies, perhaps based on or accounting for the overall memory structure and correlations between memory items.
tasked with increasing the speed at which deliberative
28

---

Published as a conference paper at ICLR 2023
A.5 SIMPLICIAL HOMOLOGY
Simplicial homology allows us to precisely count the number of 'holes' in each dimension of a simplicial complex by calculating the d-diinesionnal Betti number, Ag. related topological property which is particularly useful when studying low-dimensional objects (c.g., classification of surfaces) is the Euler characteristic, which for simplicial complex can be calculated by x(K) = which 'balances' out the number of holes in odd and
( K& i.e., it is an alternating even dimensions. is related to the Betti numbers insofar as the Euler characteristic is also given by X(K) i -) Sr. Note, however, that K Bg. As such, although the Euler characteristic can be used for comparing the topologies of two simplicial complexes, it is not as informative (with respect holes) homology (although the latter is more costly to compute, as will see).
k-chains and boundaries.
The group of k-chains is free Abelian group with the basis of Ka.
C₁ Cx(K) = kK₁
Σ ago as EZ E
The boundary (difference between the 'end points") of a face a in dimension k is
0x(o) = Σ ((
jEO
where sign(j. = (-1)(=1, where j is the i-th element of (ordered) and o' := o(f)).
Example Consider the simplicial complex
K ((1, (1, (1. 3), (2. (1), (3),
(18)
For k 2, we have K₂ (f1,2,3)) and C2 ([1 31)), Let us calculate the boundary of σ = {1, 2,3. First, we calculate the respective sign functions:
sign (-1)
(-1) 1-1))=1((11)---- (-1)
sign(2, = (-1)== =-1
(ign(,,)=((11))))))
= (-1)==1=(-1)² (-1)²
Using these values, we may calculate the boundary by
02(0) sign(1,
+ sign(2. (-1)(())2)
+ (igg(3,o)(a))) +1)(0)33
(1)(a)1)
(1)((233)
((1))((1,3)
+ (1)((1,))
[2,33
- [1,3]
+ (1.2).
The boundary of (1,2.3) is (2.3) {1,3) + (1,2). Notice, this is cycle.
3
1)>1
+(3> >
1
2
+(>>1)
Figure Geometric realisation of the simplicial complex in equation 18 and the boundary of {1, 2,3).
29

---

Published as a conference paper at ICLR 2023
Chain complex. The k-th boundary mapping, Os is the map (k) Ck-1(K)..... If > m - or & < -1, then (k(K) = 0 and 0g == O, where m is the number of 0 simplicess. Based on this, the chain complex of K is
0 (K) On-1
d + (i(K) + Co(K) do C-1(K) -
a
We define 0² = ô θ = 0. For example, 0;-1 0 = 0. This has the consequence of making the boundary of, say, solid tetrahedron (3-simplex) set of oriented triangles (2-simplices) with 'net flow' of O (similar to Stokes' curl theorem in calculus).
Example Consider the following simplicial complex and its chain complex:
K {11, 2), (1), (2), (4),04
8, - Co(K) & + C-1(K) 0.
0 - C1 (1())
The boundary map 0 is {1,2] (2) {1) and all faces in Kú mapped to the empty du-
Homology. We can now see that our k-cycles are 74 = ker 0, (k-chains a where Og (a) = 0). Whereas, the k-boundaries are Bg im 0k+1 (k-chains in the image of θx+1). Notice, BB Zn.
The (reduced) k-homology of K is the Abelian group
(k( Zx/Ba. Bk.
and we define (m-I(K) ker 0m-1 for k and (K) 0 for A 0.
The k-th Betti number (number of topological holes) is
B* = dim Hx
dim (Zx) - dim (B&)
nullity (0x) rank (0++1)-
Note, nullity On dim (Co).
Example 3. Consider the simplicial complex
== ((1,2), (1,3), (2,3), (1), (2), (3).0),
which is the same as that depicted in Figure 5 but without the illed-in triangle. The chain complex is
(1,2)
(1,3)
(2,3)
(1)
-1
-1
[2]
1
0
-1
(3)
0
(0 map)
0 +
2
0.
8,
8,
We may compute its Betri numbers by
G nullity (06) rank (8o+1)
- 2 =
nullity (0)) rank (01+1)
1 - -,
and, by definition, 831 = in this example.
A.6 PROOFS
The following are our proofs for statements in the main text.
Corollary A.1 (Memory capacity is proportional to the number of network connections). the connection weights in a Hopfield network are symmetric, then the order of the network's 's memory capacity is proportional the number of its connections.
30

---

Published as a conference paper at ICLR 2023
Proof. Let d be the degree of connections in Hopfield network with N neurons. The explicit implicit number of connections in such a network is N By a simple counting argument, the number of repeated connections between any set of d neurons is interpreted d!. By Newman (1988); Demircigil et al. (2017), the order of such a network's memory capacity Nd-1 So the following relationship holds:
Nd-1 N Na/d! d
d!
N dc
Lemma A.2 (Fully-connected mixed Hopfield networks). A fully-connected mixed Hopfield network based on a D-skeleton with N neurons and P patterns has, when N - and P finite: fixed point altractors the memory patterns and dynamic convergence towards the fixed point attractors within a finite Hamming distance 6. When P + oo with - oo, the network has capacity to store up to (Ea= / 2 in N) memory patterns (with small retrieval errors) or (E / (4 N) (without retrieval errors).
Proof. Let K be a D-skeleton. Let 5 the spin of neuron at time-step f, and let the spin correspond to stored pattern, S(() E, To be in a fixed point, the local field h2 applied to must satisfy the inequality 5( h4 > O, meaning the local field being applied to the neuron must be of the same sign as the present spin.
In the case of the mixed network, based on the simplicial Hopfield Equations the local field is
Σ
eu s(t-1) Sore oii
Σ
cll cl ((t-1) +... So o/i
Kg
K2
o
#=1
oEK2
p=1
(19)
Kal
Sore s(1-11)
[Ka]
oEKa
Because K is D skeleton, each dimension Ka will have (7) elements. Using Stirling's Approx- imation for the binomial coefficient, (*) zs /d! (for N > d which holds if dim (K) is small, which we argue should normally be for both computational and biological reasons), we can simplify Equation 19 slightly
D
P
hイ150
h
Σ d=1
d!
Σ
Σ
ell s(t-1)) Si Sa :
(20)
Nd oEK p=1
To analyse the stability of pattern, we set S(()) E. where the choice of 1 is arbitrary (since the weights are symmetric). Substituting g for S and Equation 20 for h, the inequality we must satisfy for becomes
h
Σ d=1
d! N m)
Σ aEK u=
P Σ
el (o > O.
(21)
Nd
Nd
Notice in Equation 21 we decomposed the summation over patterns into signal terms (for the pattern we are analysing) and noise terms (for the contribution of all other patterns). In the limit of -> oc, the signal terms are fixed numbers (of order 1) and, by the Central Limit Theorem, since the noise terms are sums of random numbers (essentially, random walks), they will have means 0 and standard deviations of
d!
(P-
II (N m):
(22)
Nd
31

---

Published as a conference paper at ICLR 2023
which approximate as / P/N We that d increases, the noise terms reduce in variance. However, if P remains fixed and N is sufficiently large, the noise terms become negligible compared to the signal terms. This therefore guarantees that every pattern will be fixed point.
Furthermore, these fixed points will remain highly stable against random noise. Suppose we randomly flip finite number of spins away from pattern ç S (a fixed point). The signal terms' strengths are reduced by 20 but still of order 1, whereas the noise terms remain of order N-1/2, Therefore, states within Hamming distance away from & will converge to the fixed point.
Now let P with N. The total variance of the noise terms is
P/N4
(23)
The probability of stability of neuron in Equation 21 the probability that the noise terms larger than -1; at v -1 they will overcome the signal terms. This probability is
h1 >0) = V2T02 [ dr exp
(- x² (=20
=
1 + erf
V 20²
(24)
2
where
2 V dte -t
/
erf(x)
(25)
For small (which this is, especially as d increases and relative the signal), the value of the error function in Equation 24 will be large and can therefore be approximated (Gradshteyn & Ryzhik 2007 as
erf(z) ~
(26)
We can now approximate Equation
24 as
1
Pr(g 000
122 2T
exp
2z
(27)
where
Σ
P/Nd
(28)
d=1
We can now calculate the probability of stable pattern, i.c., that the inequality 5 h2 > 0 is satisfied for all i, with
1
Pr(stable pattern) 2
exp
2m
2z
(29)
Z
exp
2m
2z
Since N + co, Equation 29 will be close to 1 as the second term will be negligible. This will always be true if
32

---

Published as a conference paper at ICLR 2023
2 2InN
(30)
Therefore, since we have N neurons with Ed= Na connections between them, the maximum number of patterns may store (while accepting small errors) is
Σ 2d=1 Nd
Pe
(31)
2
Or, if we cannot accept errors,
Σb d== N'
Pe
(32)
4
An additional informal perspective to consider regarding memory capacity is to notice that if P scales with N, the ratio between the signal and noise terms will be constant. And, since P x No the theoretical memory capacity scales polynomially with N and linearly with D, and so the theoretical capacity is approximately Eb Cd Ná, where cd constant which depends on
A.7 NUMERICAL IMPLEMENTATION OF SIMILARITY MEASURES
As in Millidge et al. 2022). in order to fairly compare similarity functions, we: (i) normalised simi- larity scores (separately for each similarity function) so their sum would be equal to (since different measures had intrinsically different scales); and (ii) for distance measures, used the normalised reciprocal (since distance measures return low values for similar inputs, but the model relies on high values being returned for similar inputs, in the dot product).
A.8 SUPPLEMENTARY FIGURES AND TABLES
Table 3: Pearson correlation coefficients (r) between overlap and B3 for mixed diluted networks from Table Bolded values are significant at a 0.05 (without multiple comparisons adjustment). With multiple comparisons adjustment, there are no significant correlations. Given their construction, all networks have 8u 1, and although there is a small chance of dimensional holes in some networks, we found that 822 = 0 for all simulated networks.
No. patterns
0.05N
0.1N
0.15N
0.2N
0.3N
R12
0.02
0.09
0.21
-0.15
-0.01
R1Z
N/A
0..5
001
0.01
-0.08
Table 4: Extended list of network condition keys (top row), their number of non-zero weights for 1-, 2-, and 3-simplices (second, third, and fourth rows). N is the number of neurons. For simulation, the number of simplices each dimension are rounded the nearest integer.
R3
RT23
R123
R123
RT23
1 simplicces
0.50
N)
0.25
0.25 (N)
1/3(
2-simplices
0.25 .
0.50
0.25
1/3
3 simplices
N
0.25
0.25 (N)
0.50
1/3
33

---

Published as a conference paper at ICLR 2023
0.05N
P 0.10N
0.15N
1.0
1.0
1.0
8
8
0.8
0.8
0.8
Overiap
0.6
0.6
0.6
0.4
0.4
0.4
0.0
P 0.20N
P 0.25N
P 0.30N
1.0
1.0
[
1.0
1000 COKD
0.8
8
0.8
OOOKED 6
0.8
Overlap
0.6
0.6
0.6
0.4
0.4
0.4
K1
R12 (25%)
RI2 (50%)
R12 (75%)
R2
K1
R12 (25%)
R12 (50%)
R12 (75%)
R2
R12 (25%)
R12 (50%)
R12 (75%)
R2
Figure 6: Box and whisker plots of final overlap distributions from traditional simplicial Hopfield
networks with varying numbers of embedded patterns. Orange lines indicate the median. The red dashed line indicates an overlap of 0.5, chance.
Table 5: Mean + standard deviation of overlap distributions (n 100) from traditional simplicial Hopfield networks with varying numbers (top row) of random binary patterns. Keys per Table At all pattern loadings, one-way ANOVA showed significant variance between the networks
10-12, F > 13.25).
No. patterns
0.05N
0.1N
0.15NN
0.2NV
0.3.N
R123
1 1 0
0.99 + 0.08
0.97 + 0.17
0.93 + 0.22
0.89 + 0.15
R123
1 + 0
1 1 0
0.98 + 0.05
0.95 + 0.17
0.91 + 0.18
R123
1 0
1 1 0
1 1 0
0.96 + 13
0.93 + 0.13
RI23
1
1 0
1 1 0
1 0
+ 0
R3
0.94 0.06
0.78 + 0.14
0.52 0.15
0.51 + 0.13
0.51 + 0.14
Table 6: Mean + standard deviation (n 10) of fraction of correctly recalled MNIST memory pat- terns in simplicial Hopfield networks at a memory loading of 1000 memories. Network performance varied significantly.
K1
RI2
R12
R123
Euclidean
1 +
+ 0
1 0
+ 0
Manhattan
1 1
+
1 0
1 1 0
Dot Product
0.93 + 0.03
0.93 + 0.02
0.94 0.02
1 + 0
ced
0.90 + 0.02
0.91 0.03
1 1 0
cmd
0.95 + 0.03
0.97 + 0.03
1 0
34

---

Published as a conference paper at ICLR 2023
T= 1
T= 2
T= 10
Low
PC2 (25. 50.
K1
E
High
PC2 (25. (5.0%)
R12(25%)
PC2 (25.0 0%))
1
R12(50%)
PC2 (25. (25.0% .0%)
R12(75%)
PC2 (25. .0%)
R2
0 PC1 (34.3%)
2
PC1 (34.3%)
PC1 (34.3%)
Figure 7: Relative energies of continuous modern networks in 10 x 10 grid plane of the first two dimensions PCA space, as computed using the memory patterns. Each network has N 10 and the same P 10 memory patterns are embedded. Network conditions vary by row and inverse temperatures vary by column. Black dots are projections of the 10 embedded patterns in the PCA space. The combined explained variance of the first two principle components is 59.3% of the memory patterns.
35

---

Published as a conference paper at ICLR 2023
Table 7: Same Table 6 but for CIFAR-10.
K1
R12
R12
R123
Euclidean
0.31 + 0.08
0.39 + 0.08
0.51 + 0.07
0.64 + 0.08
Manhattan
0.70 + 0.06
0.77 + 0.06
0.90 + 0.05
0.97 + 0.04
Dot Product
0.50 + 0.07
0.56 0.07
0.68 + 0.06
0.72 + 0.08
ced
0.61 0.06
0.74 0.06
0.81 + 0.07
emd
0.65 0.07
0.91 0.06
0.99 + 0.02
Table 8: Same as Table 6 but for Tiny ImageNet.
K1
RT2
R12
R123
Euclidean
0.31 + 0.15
0.34 + 0.14
0.55 + 0.12
0.61 + 0.15
Manhattan
0.63 + 0.10
0.70 + 0.08
0.84 + 0.08
0.91 + 0.09
Dot Product
0 + 0
0 + 0
0 0 0
0 -
ced
0.51 + 0.14
0.65 + 0.13
0.70 + 0.11
cmd
0.71 +: 0.08
0.92 + 0.06
0.95 + 0.06
36


Reviews:
[{"review_id": "hVHJn9jOYf", "reviewer": "Reviewer_fyDU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduced simplical complexes into various Hopfield networks and showed that the memory capacity of the modified Hopfield nets are increased. They shoed that such improvement comes from the topology rather tha the additional information in the form of parameters. They also whoed that how distance measures of a more \"geometric flavour\" can further improve performance in these networks.", "summary_of_the_review": "The study is comprehensive, and both theoretical analysis and simulation results are carried out to prove their ideas. ", "strength_and_weaknesses": "The paper is clearly written. Both theoretical analysis and empirical simulations are carried out to prove their ideas. My main concern is that, why the capacity in Hopfield nets is a problem, especially in the biological brain where there are a large number of neurons? As long as there are enough neurons, the memory capacity is not a problem (we don't need to memorize many items at the same time). Is it more about a mathmetical \"game\" or has some real-world implications?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dcK-aASDFRd", "reviewer": "Reviewer_qaFD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce Simplicial Hopfield Networks — an architecture that extends the original Hopfield Network by allowing setwise connections in addition to pairwise connections between all neurons in the network. This reformulation allows higher memory capacity and performance and significantly increases the memory capacity of the traditional Hopfield Network.", "summary_of_the_review": "The paper offers a novel perspective into how to interpret Hopfield Networks, though reproducibility of this paper is difficult in its current state (see weaknesses). I recommend this as borderline accept, with room for improvement if my comments can be addressed.\n\nI did not read the appendix in depth.", "strength_and_weaknesses": "******************Strengths******************\n\n- (+ +) The idea of treating Hopfield Networks as simplicial complexes is enticing and indeed novel.\n- (+) The experimental results are minimal and simple, but still convincing. The memory capacity of a Hopfield Net, in the binary case, is noticeably improved by including higher degree connections.\n\n********************Weaknesses********************\n\n- (- -) The experimental results are minimal and simple. Only K=1 and K=2 simplices are considered (see below). In particular, the continuous case chooses only one “simplicial model” and applies it on MNIST, CIFAR, and Tiny ImageNet while comparing different similarity metrics. Comments:\n    - It would be useful to see behavior across simplicial models in the continuous case. In particular, perhaps, simplicial networks of higher order than just 2.\n    - Figure 2 describing these results is laid out such that it is very difficult to compare the baseline to the proposed model… This is fine, but perhaps a corresponding table like Table 2 for recall on the different datasets for one of the distance metrics would be preferable.\n- (-) The re-formulation of the Hopfield Network using simplices is convenient for mathematical proofs, but practically it can be computationally prohibitive to consider all the subsets that higher degree simplicial complexes create. This is shown by the experiments which exclusively considered pairwise and K=2 degree complexes. To sell the practical application of this idea, I encourage the authors to include experiments with higher order simplicial complexes, or to reconsider use of the hypergraph formulations as discussed in the Appendix.\n- (-) I found Figure 1 completely unhelpful in understanding the paper. 1B has no meaning to me, only that the weight distributions “look” the same at different simplices (there’s not even a y-axis). 1C is not sufficiently explained and is equally meaningless to me.\n\n**Additional comments**\n\n- (-) Please revise the statement “Ramsauer et al. (2021) extended modern Hopfield networks to allow neurons to take continuous values – where previously they took only binary values”. Ramsauer et al. claimed to have allowed Hopfield networks to take continuous states, but in fact continuous Hopfield networks were introduced by [Hopfield himself in 1984](https://www.pnas.org/doi/10.1073/pnas.81.10.3088). We want to correct this incorrect claim as much as possible in papers moving forward.\n- (-) There is no mention of code to be released for reproducibility.\n- (-) Notation in equation (5) is strange. Why input “$\\ldots \\Xi^T S \\ldots$” to the logsumexp when the RHS uses “$\\ldots \\xi_\\sigma^\\mu S_\\sigma \\ldots$”?\n- (-) Where does $M^2$ come from in equation (6)? This is seemingly copied directly from Ramsauer et al. and not defined here.\n\n**Questions**\n\n- The original HN uses a dot product, which is clearly a similarity measure where more similar inputs have a higher value. However, taking the Manhattan or Euclidean distance is a **********difference********** measure where more similar inputs have a smaller value. Is the inverse of the euclidean distance taken in the experiments? Please describe this for reproducibility.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QCh5_Sty9zc", "reviewer": "Reviewer_R7jA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to add set-wise connections to pair-wise connections in the Hopfield networks. The main contribution of the paper is the claim that adding such connection improves the memory storage properties of these networks. Interesting biological mechanisms of building many-body synapses are also mentioned.   ", "summary_of_the_review": "Potentially interesting paper, but requires clarifications regarding the core mathematical definitions and more accurate description of the relationship to previous work. For now, I will score this paper in the middle range. I am keeping an open mind about this work, and depending on the revisions/discussions with the authors can either increase or decrease the scores.", "strength_and_weaknesses": "This paper connects many interesting ideas, such as Hopfield networks, simplicial complexes, many-body synapses in biology, etc. These are great topics, and it is nice to see them discussed in the paper. However, I am struggling understanding the core mathematical definitions of the proposed model. Specifically, \n\n1. I do not understand the definition and notations of the model (equation 1). Could the authors please write down (possibly in the appendix) the complete expression for the energy function in some toy model example (say 2 memory patterns, upto 3-simplex connections). I want to see the complete expression for the energy function written in terms of $S_i$ and $\\xi^\\mu_i$. \n\n2. Is the energy function (equation 1) a sum of the energy functions of Krotov & Hopfield 2016 with say 3-neuron connections, 2-neuron connections, 1-neuron connections? If so, are the weights of those connections independent of each other for 3-, 2- and 1- body interactions? In the model of Krotov & Hopfield, once the patterns $\\xi^\\mu_i$ are fixed (consider a power model for simplicity with n=3), the connectivity - symmetric tensor of the 3rd rank $w_{ijk}$ - is completely determined. This tensor has $N(N-1)(N-2)/6$ elements in it, but those elements are not independent. Are the authors proposing to declare some of the elements of this tensor to be zero, so that only those terms present in the simplicial complex are non-zero?  \n\n3. Also, I don’t fully understand the definitions of the models considered in Table 1. For the conventional 2-body Hopfield network if the number of patterns is odd, then the connectivity matrix $w_{ij}$ has all non-zero entries. There is not any flexibility to delete any entries in it. How can the authors decrease the number of parameters from $N(N-1)/2$ to $0.75N(N-1)/2$? Which elements of the matrix $w_{ij}$ are deleted? Similarly, how do authors add 3-body connections? \n\nAdditionally, I do not understand the histograms in Figure 1B. If the memories in the Hopfield network are binary ($\\pm 1$) patterns and there are P of those, the elements of the weight matrix belong to the range $-P$ to $+P$ and are all integers. How do the authors get numbers smaller than 1 on the x-axis? Also, these three histograms look very similar to me. Is this what the authors are trying to illustrate by showing them? I somehow could not find a clear conclusion from these histograms in the text. \n \nContinuous modern Hopfield networks were introduced in Krotov & Hopfield 2016, not in Ramsauer et al. 2021. Ramsauer et al. 2021 generalized the model of Krotov and Hopfield (which used any rapidly growing neuron-wise activation function, e.g. power function) to the case of softmax activation function, and made the connection with transformers. However, the theoretical framework for continuous modern Hopfield networks has already been introduced in Krotov & Hopfield 2016, see for example equation (10) there, and numerical experiments with MNIST (all use continuous variables). \n\nWhile I greatly appreciate that the authors discuss legitimate biophysical mechanisms for building set-wise connections in the brain (Appendix A1), these set-wise connections can also be built from simple pair-wise connections by introducing additional hidden neurons, see for example https://openreview.net/forum?id=X4y_10OX-hX. This possibility should at least be mentioned somewhere in the paper. Even if one does not use more rarely discussed biophysical mechanisms mentioned by the authors in Appendix A1, there is nothing unbiological (except for shared weights) in many-body set-wise connections. \n\nTypo: the constant factor $T$ should be replaced by $1/T$ in equation (7). \n\nA small suggestions to the authors for making their formulas more compact (equations 5, 6): $\\frac{1}{T^{-1}} = T$. Also, the last two terms in equation (6) are just constants, thus can be dropped.\n\nThe notation $\\dot{S}$ in equations 2, 4, etc., is somewhat confusing, since it is often used for denoting the time derivative. While here, the authors use it for denoting the updated state of the network. I suggest that the authors either replace it by something else, or add a note the first time it appears in the paper about its meaning. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster