Kernel Memory Networks: A Unifying Framework for Memory Modeling
Authors: Georgios Iatropoulos
Johanni Brea
Wulfram Gerstner
Venue: NeurIPS 2022 Accept (2022)
Keywords: attractor networks
associative memory
kernel methods
Hopfield networks

Abstract: We consider the problem of training a neural network to store a set of patterns with maximal noise robustness. A solution, in terms of optimal weights and state update rules, is derived by training each individual neuron to perform either kernel classification or interpolation with a minimum weight norm. By applying this method to feed-forward and recurrent networks, we derive optimal models, termed kernel memory networks, that include, as special cases, many of the hetero- and auto-associative memory models that have been proposed over the past years, such as modern Hopfield networks and Kanerva's sparse distributed memory. We modify Kanerva's model and demonstrate a simple way to design a kernel memory network that can store an exponential number of continuous-valued patterns with a finite basin of attraction. The framework of kernel memory networks offers a simple and intuitive way to understand the storage capacity of previous memory models, and allows for new biological interpretations in terms of dendritic non-linearities and synaptic cross-talk.

Kernel Memory Networks: A Unifying Framework for Memory Modeling
Georgios Iatropoulos 1.2
Johanni Brea! l.
Wulfram Gerstner
(firstname. 1astname)@@pfl..
Laboratory of Computational Neuroscience, Blue Brain Project École Polytechnique Fédérale de Lausanne Switzerland
Abstract
We consider the problem of training a neural network to store set of patterns with maximal noise robustness. A solution, in terms of optimal weights and state update rules, is derived by training each individual neuron to perform either kernel classification or interpolation with minimum weight norm. By applying this method to feed-forward and recurrent networks, we derive optimal models, termed kernel memory networks, that include, as special cases, many of the hetero- and auto-associative memory models that have been proposed over the past years, such as modern Hopfield networks and Kanerva's sparse distributed memory. We modify Kanerva's 's model and demonstrate simple way to design kernel memory network that can store an exponential number of continuous-valuee basin of attraction. The framework of kernel memory networks offers simple and intuitive way to understand the storage capacity of previous memory models, and allows for new biological interpretations in terms of dendritic non-linearities and synaptic cross-talk.
patterns with finite
Introduction
Although the classical work attractor neural networks reached its peak in the late 1980's, with the publication of number of seminal works [e.g., 2, 20, 22, 26], recent years have seen a renewed interest in the topic, motivated by the popularity of the attention mechanism [65], external memory- augmented neural networks [24, 66], as well as a new generation
emergy-based attractor networks models, termed modern Hopfield networks (MHNs), capable of vastly increased memory storage [17, 35]. Recent efforts to understand the theoretical foundation of the attention mechanism have, in fact, shown that it can be linked to Hopfield networks [36, 57], but also Kanerva's sparse distributed memory (SDM) [8, 30], and to the field of kernel machines [63, 68]. The last connection is particularly intriguing. in light of the many theoretical commonalities between neural networks and kernel methods [10, 11, 28, 47, 67]. Overall, these results suggest that unified view can offer new insights into memory modeling and new tools for leveraging memory in machine learning.
In this work, we aim to clarify some of the overlap between the fields of memory modeling and statistical learning, by integrating and formalizing a set of theoretical connections between Hopfield networks, the SDM, kernel machines, and neuron models with non-linear dendritic processing.
$ Joint senior authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1.1 Our contribution
We derive set of normative kernel-based models that describe the general mathematical structure of feed-forward (i.e., ) and recurrent (i.e., auto-associative) memory networks that can perform error-free recall of given set of patterns with maximal robustness to noise.
We show that the normative models include, special cases, the classical and modern Hopfield network, as well as the SDM.
We derive simple attractor network model for storing an exponential number of continuous-valueed patterns with finite basin of attraction. We discuss its similarity to attention.
Furthermore, we explain how classifiers with non-linear kernels can be interpreted as general forms of neuron models with non-linear dendritic activation functions and synaptic cross-talk.
1.2 Related work
Our work is primarily related to [8, 9, 35, 36, 45, 57]. While MHNs are extensively analyzed [35, 36, 45, 57], the approach is energy- boargy-based and makes no statements about the relation between MHNs and kernel methods; a brief comment in [57] mentions some similarity to SVMs, but this is not further explained. The work by [8] focuses the SDM and its connection to attention. It observes that the classical Hopfield network is a special case of the SDM, but no further generalization is made, and kernel methods are mentioned. In work, we place MHNs and the SDM in broader theoretical context by showing that both models are special suboptimal cases of a family of memory networks that can be derived with normative kernel-based approach.
2 Background
Consider the following simple model of hetero-associative memory: a single-layer feed-forward network consisting of single output neuron connected Nin inputs with the weights E IR-,, The output Sout {+1} is given by
8 Sont
8g11
w (S(in) -0]
(1)
where Sin is the input vector (also called query), the threshold, and Φ a function that maps the "raw" input to Ng-dimensional feature space, where typically No >> Nin- Suppose that we are given set of M onput-output patterns
1/M in which every entry & is randomly drawn from {+1} with sparseness = PF = In order for the neuron to store the patterns in a way that maximizes the amount of noise it can tolerate while still being able to recall all patterns without errors, one needs to find the weights that produce the output &h in response to the input t Vu, and that maximize the smallest Fuclidean distance between the inputs and the neuron's 's decision boundary.
Using Gardner's formalism [20, 22], this problem can be expressed as
w ((E!)) - 0) 2 k, V4
arg max W
s.t.
(2)
[w]] = v
where w is a constant. This equivalent to solving
min |||l22 S. Sbut (w (T(((t)))))))))))))))))))) -0) 1, V/
which be directly identified as the support vector machine (SVM) problem for separable data [13]. The solution to Eq. 3 can today be found in any textbook on basic machine learning methods, and yields an optimal output rule that can be written in a feature and kernel form
M
M
Sout = sgn
a eh ( ((()))) - 0
sgn
Σ
al ehu K K(Et' (EtH
smm) -
(4)
where we, in the latter expression, have used the "kernel-trick" K K(x;, x3) ((xx)) (xxx)). The solution depends on the Lagrange coefficients a 2 O, many of which are typically zero. Patterns with a > 0 are called support vectors.

---

3 Kernel memory networks for binary patterns
3.1 Hetero-associative memory as a feed-forward SVM network
We begin by considering a hetero-associative memory network with an arbitrary number N output neurons, whose combined state we denote Sout- In order for the network as whole be able to tolerate a maximal level of noise and still successfully recall its stored memories, we solve Eq. for each neuron independently. As each neuron can have different classification boundary along with different set of support vectors, its weights will, in general, be characterized by independent set of M Lagrange coefficients. To simplify the notation, we represent these coefficients af across neurons and patterns μr, as entries in the matrix A, where (A)ir = a." We also combine all thresholds in the vector = (01...., @Naa). and all input and output patterns as columns in the assume that all neurons have the same feature map, so that Φ; = Φ. Vi (see Fig. 1). All functions are applied column- colse The optimal response when the argument is matrix, for example ((Xin) = ((.())).....
matrices X (Li.. ................................ ..)) and Xout (E ..... Eou ). Finally,
of the network can now be compactly summarized as follows.
Property 1 (Robust hetero- assoocaavsee memory network). single-layer hetero-associative memory network trained to recall the patterns Xout in response the inputs Xin with maximal noise robustness, has an optimal output rule that can be written as
Sout = sgn (A 3 Xont ) (Xin)))X)))))))))))))) ((in))
(feature form)
(5)
sgn (A O Xout) ) K (Xinn Sin) 0
(kernel form)
(6)
where 3 denotes the Hadamard product.
3.2 Auto-associative memory as a recurrent SVM network
The -asociaassociative network can be made auto-associative by setting Nout Ni and Xout = Xin- The network now effectively recurrent, as each neuron can serve both as an input and output simultaneously (see Fig. 1). Consider a recurrent network with N neurons, whose state at time point t is denoted (!t) E {=1}", and whose dynamics evolve according to the update rule
((+1) = sgn w (ss(t)) 0.
where w, R: is the weight vector neuron 1..... N. In order to make the patterns { fixed points of the network dynamics, we train each neuron i independently on every pattern to, again, produce the response ş when the rest of the network is initialized ş" Moreover, we maximize the amount of noise that can be tolerated by the network while maintaining error-free recall by maximizing the smallest Fuclidean distance between each neuron's 's decision boundary and its inputs. This maximizes the size of the attractor basins [18, 32]. The problem of training the entire network is, in this way, transformed into the problem training N separate classifiers according to
min ||well2 S. t. e.¹ <W. (VTTo(E") 0.) 2 1, Vu,i. w:
The solution can be obtained by slightly modifying Property 1, and is stated below.
Property 2.1 (Robust auto-associative memory). A recurrent auto-associative memory network trained to recall the patterns X with maximal noise robustness has an optimal synchronous update rule that can be written as
(1+1)
sgn (A () ((s(0) -0]
(feature form)
sgn
(A X)K(X,s ((1)) 0
(kernelform
(10)
Remark. With linear feature map (xx) x, the optimal update is reduced to
((+ sgn (A o x)XT= (t) -0)
-0]
(11)
where (A 3 X)XT can be identified as the general form of the optimal weight matrix.
3

---

The solution described by Property 2.1 does not, in general, prohibit a neuron from having self- connections. Applying this constraint yields the following result.
Property 2.2 (Robust auto-associia aitooassoociative memory without self ceff-connections) A recurrent auto- associative memory network without self self-connections, with the inner proorrproduct kernel K (x, Xj) = k(x, x2). that has been trained to recall the patterns X with maximal noise robustness, has an optimal asynchronous update rule that can be written in the kernel form
M k
N
(t+1) 8/
sgn
Σ Cff
St
Σ el
5
02
(12)
Storage capacity. An intuition for the storage capacity scaling of the hetero- and auto-associative memory networks can be gained by observing that the network as a whole will be able to successfully recall patterns as long each neuron is able to correctly classify its inputs (or is very unlikely to produce an error). The capacity of the network can thereby be derived from the capacity of each individual neuron. It is well-known that linear binary classifier can learn correctly discriminate maximum of mmax 2 2D random patterns, where Dvc is the Vapnik- Chenik-Cherooneniss dimension of the classifier [15, 20, 40, ch. 40J. For neuron with N inputs and linear feature map o(x) = x, this results in Dvc = N and, thus, the capacity Mmax R 2N. Suppose, on the other hand, that the kernel is homogeneous polynomial of degree p. so that K (x,, Xj) (x] xy)P. - In this case, &b will contain all monomials of degree composed of the entries in x. As there are ((NP) unique p-degree monomials (see Appendix the exponential kernel, which we write K (x; x,) exp(x; xj) -p=o(x) x3) /pl, the
I. the input dimensionality and M Maax will be ((P).. For
dimensionality of & will be Ep (7) 2 which yields Mmax O(e.).
Special cases. In the following sections, we will show that many of the models of hetero- and auto-associative memory that have been proposed over the past years are special cases of the solutions in Properties 1, 2. and 2.2, characterized by specific choices of A, Ø, and K.
3.3 Kanerva's sparse distributed memory is a feed-forward SVM network
The sparse distributed memory (SDM), developed by Kanerva [30], is one of the most famous examples of hetero-associative memory model. It has lately received much attention in the context of generative memory models [69] and attention layers in transformers [8].
The SDM consists of a register of N₂ memory slots, each associated with an address 2; E {+ Ni.
i == 1,...,,,, No. All addresses are listed as rows in the matrix = (Z1....,,y,, ,N,)) LT,,)T The content of each slot is represented by an -ime-iimesionnal vector, initialized at zero. Suppose that wish to store the M patterns X = (Eoutr.... edd in the addresses Xin = (fl..... . where all entries are random and bipolar. The basic idea of the SDM is write the data to, and later read it from, multiple memory slots at once (hence the distributed storage); this ensures a degree of noise-robustness. mathematical terms, the read-out of the SDM provided with a query Sin, given by
Sout = sgn Xout OZZX - b) (Zsnnn - 6)]
(13)
where the Heaviside function with bias b Nu. - 2r, and r is parameter that determines the precision of the writing and reading process. Upon comparing Eqs. 13 and 5, the SDM can be directly identified as special case of a suboptimal forwaro SVM network in the feature form, with A = 1, 0 = 0. and the feature map ΦSDM(x) O b). When viewed as a kernel method, the function of the SDM is to store the dense addresses X,, as sparse high-dimensional representations ØSDM, to make it easier to later determine the slots closest to a query Sin. and retrieve the relevant data.
Capacity. As the SDM is linear in ΦSDM, with Dvc Ro No. it follows from the analysis in Sec. 3.2 that one should expect the capacity to scale as Mmax ~ O(N)). Moreover, one should expect a proportionality constant ~0.1, since the SDM is suboptimal relative to the feed-forward SVM network, analogously to how the classical Hopfield network suboptimal relative to the recurrent SVM network (see See. 3.4). This is consistent with earlier proofs [12, 31].

---

A1
A2
A3
P
W
Sourd spn[w/ &. (8..) 0.]
Sost S (8 0)
sgn mu 0 b) ( - R]]
B1
B2
C Sint
Sind
O
W
(211) sgn (s a.]
g(0e3) sgn (g
Figure Graphical representation of (Al-A2) the forwad SVM network, (A3) the SDM, (B (BI-B2) the recurrent SVM network, and (C) an SVM mapped to the anatomy of a pyramidal cell (see Sec. 5). Circles represent neurons, while boxes represent the input transformation by the feature map Φ which can be dependent (A1, B1) or independent (A2, B2) of neuron index i.
Kernel of infinite SDM. In practice, SDM with large number of memory slots N₂. requires calculations involving a large address matrix Z. This can be avoided by applying the kernel-trick to Eq. 13 in the limit No oc, which allows for the output to be computed with
Sout
sgn
Xout KSDM (Xin: Sin)
(14)
where we have defined the kernel as
KsnM(Xi, lim (SDM(x;) ΦSDM(X)) No0
(15)
No
in order to ensure convergence. In this section, we will derive this kernel for two different variants of ionslaatoo-iivarinnt It is interesting to note here that ΦSDM
the SDM and demonstrate that both is equivalent to a single-layer neural network with N₂ neurons, weights Z. and bias b. This means that KSNM equivalent to the kernel of infinitely wide neural network [11, 47, 67].
a binary vector that indicates those memory addresses in Z that differ by at most r bits compared to x. For any two bipolar vectors z and x, the bit-wise difference can be computed as 1/1z - x = = - x x| This means that @sDM(x) indicates all addresses that lie within a sphere centered at x with radius 2 2/r. Consequently, the inner product ΦSDM(X;) ΦSDM(X,) is the number of addresses located in the overlapping volume of two spheres centered x; and xj. Although an exact calculation of this quantity can be found in [8, 30]. its connection to the SDM kernel has, to the best of our knowledge, not previously been made. We therefore modify the previously published expression with normalization factor 1m and state the following property.
We begin by noticing that (x) has a geometrical interpretation [8,31] 31].
Property 3.1 (Kernel of an infinite SDM on the hypercube). In the limit No - Dc, the kernel of an SDM with N₂ memory slots, whose addresses are randomly drawn from {+1) Nin, is given by
A A- (N
(Nin 1an-A. 2)
(A A
KSDM (x) (XDM(X;;
Σ
(16)
2N
N
1=
-r
d+
5

---

where r is the bit-wise error threshold and A is the bit-wise difference between and x,, given by
A = 1/2 x - xy| = | x,112-
The SDM can also be implemented with continuous addresses, randomly placed on a unit hypersphere of (Nin- - 1) dimensions, denoted gNin -1 The vector (SDM(X) now indicates all addresses that lie within hyperspherical cap centered at with angle arccos(b) between its central axis and the rim.
The inner product ΦSDM (x,) ) ΦDDM (x;) ) is the number of addresses located in the overlapping area of two spherical caps centered at x; and Xj. While a calculation of this quantity, again, can be found in [8], it has not previously been connected to the kernel of SDM. We simplify the previously published result and also derive a closed- form approximation, valid for highly sparse ØSDM (see Appendix B for details). The results summarized below.
Property 3.2 (Kernel of an infinite SDM on the hypersphere). In the limit No - xc, the kernel of an SDM with No memory slots, whose addresses are randomly drawn drawnfrom S NN.. is given by
KsDM(X,,xg) Nin 2
tan² (am)) Ni tan' (4)
2 1
sin( Nin -2 22
de
(17)
2т
2
2
where a₂ = 1 arccos(x; (x, x²), ab ar)cos(b), and B is the incomplete Beta function. In the highly sparse regime, when 0.9 S b < and 1
@IDaNAIlo < 1, the kernel can be approximated with
No
f
A
N
1
KsDM(X;; xy)
B
1
(18)
2
2
where A - 1|x x;112 and i sin(arccos(U)).
In conclusion, an infinitely large SDM with sparse internal representations ΦSDM. can be represented as suboptimal case of feed-forward SVM network with translation-invariann kernel.
3.4 The modern Hopfield network is a recurrent SVM network
The Hopfield network [26] is, arguably, the most well-known model of auto-associative memory. In its modern form [35], it is recurrent network of N neurons with the state s(t), whose dynamics are governed by the energy and state update rule
M Σ
M
N
E=
F
Σ
ct S
(1+1)
sgn
cdt
F
Σ jitte
cl
(t)
(19)
where F is a smooth function, typically sigmoid, polynomial, or exponential. This "generalized" Hopfield model has a long history [see, e.g., 21, 25. 37] but has received renewed attention in recent years under the name modern Hopfield network (MHN) or dense associative memory 35]. By comparing Eq. 19 with Eq. 12, the state update of the MHN can identified as a special case of suboptimal recurrent SVM network in the kernel form, with k: = F', A = 1, and A = 0 (since f 0.5). With a linear F" (x) = x, the MHN reduces to the classical Hopfield network, which is a special case of the recurrent SVM network with the linear kernel lxx] x,) x x,.
Capacity. The storage capacity of the MHN has been shown to depend on the shape of F'. In the linear case, the capacity is famously limited to ~0.1N patterns, depending the precision of retrieval [2, 42]. If, on the other hand, F is polynomial with degree p, the capacity scales as Mmax (NN)) [35], while an exponential F² endows the network with capacity Minax O(e [17]. From the perspective of the kernel memory framework, this scaling directly follows from the analysis in Sec. 3.2 with F² In fact, in the regime of low errors, the kernel memory framework can also be used to derive more precise capacity scaling for the classical Hopfield network. We first note that any one-shot learning rule that implies A > 0 is equivalent to an SVM network where every stored pattern is a support vector. Such a heuristic is only likely to be close to the optimal solution and perform well large networks with very few patterns, as high-dimensional linear SVMs trained on few patterns are highly likely to find solutions where all patterns are support vectors; this effect has been termed support vector proliferation [4]. Restricting the network this regime limits the capacity to M O( Tlog ), consistent with the result in [42] (see Appendix A.2).
N

---

Iterative learning rules. The problem of iteratively training MHNs with biologically plausible online learning rules has recently been studied [64], with a resulting storage capacity ranging from ~0.16NN to ~,,, depending on the exact implementation. The aim, in general, of such studies is to find a learning rule capable of producing a capacity close to the theoretical maximum ~2N. For this purpose, the perspective of kernel memory networks can be particularly helpful, as many of the algorithms that have been developed over the past two decades to optimize SVMs can be utilized for MHNs as well. For example, a network formulated in the feature form can be trained with the stochastic batch perceptron rule [14, 34], the passive aggressive rules [16], the minnorm rule [5], as well as with likelihood maximization applied to logistic regression [29, 46, 59]. In the kernel form, two of the most well-known online algorithms for training linear and non-linear SVMs are the Adatron [3] and the Aereel-Aatton [19]. A performance comparison between iterative learning and the modern Hopfield learning rule can be found in Appendix C.
Generalization. Viewing the MHN as recurrent network of SVMs can also facilitate a more intuitive understanding of its ability to generalize, when used as a conventional classifier. In this setting, one designates a subset of the neurons as input units, and the remaining neurons as outputs. Given a set of input-output associations, one optimizes the memory patterns & using, for example, gradient descent. Such an experiment was performed by Krotov and Hopfield [35] on the MNIST data set, using a polynomial non-linearity F(z) = z?. Results showed that the test error first improved as p increased from to 3, but later deteriorated for high degrees, like p = 20. While it may be difficult avergy-based framework, it is entirely expected when viewed from
to explain this behavior within the SVM perspective: a kernel of low polynomial degree has too few degrees of freedom to fit the classification boundary in the training set, causing underfitting, while polynomial of too high degree grants the model too much flexibility, which results in overfitting
The pseudoinverse learning rule. The coefficients A are, in general, computed numerically, and cannot be written in closed form. However, in the special case when Eq. 8 is underdetermined. meaning M < No, a closed-form (but suboptimal) solution can be obtained using the least-squares SVM method [60]. The result is generalized form of the pseudoinverse learning rule [50]. See Appendix D for details.
4 Kernel memory networks for continuous patterns
4.1 Auto-associate memory as a recurrent interpolation network
So far, we have considered memory models designed to store only bipolar patterns. We now relax this constraint and allow patterns to be continuous-valued. We first observe that any set of patterns X E xMM can be made fixed points of the dynamics by training each neuron to interpolate ę when the rest of the network is initialized in ş" for every pattern μr. Assuming that the model is equipped with a kernel that allows for each fixed point to also be attracting, we can ensure that lower bounding estimate of the size of the attractor basin is maximized by finding the interpolation with minimum weight norm (see Appendix E for proof). These results are summarized below.
Property 4 (Robust auto-associative memory with continuous patterns). Suppose that the dynamics of recurrent auto assooiiasse memory network evolve according to the synchronous update rule
(t+1) (X,')
(20)
where K K= K -(x) Φ(X) is the kernel matrix and K its Moore-Penrose pseudoinverse, where Ki-K- ((X) full column rank. Then, the dynamics of the network is guaranteed to have the fixed points X. Moreover, if the points are attracting, Eq. 20 maximizes a lower bound of the attractor basin sizes.
4.2 A recurrent interpolation network with exponential capacity
Memory models for continuous data [e.g., 27, 48] have generally received less attention than their binary counterparts. Recently, however, Ramsauer et al. [57] proposed an energy-based model capable of storing an exponential number of continuous-valun softmax network). While the structure of this model is similar to Eq. 20, it cannot be analyzed within
patterns (we will refer to this model as the
7

---

the framework of Property 4, as it involves a kernel that is neither symmetric nor positive-definit [68].
Nonetheless, we will in this section demonstrate that is possible to use conventional kernel methods to design an attractor network with exponential capacity for continuous patterns. We utilize the properties of the SDM by using a translation -ivvarirann kernel with a fixed spatial scale r. For the sake of simplicity, we choose the exponential power kernel (Expa)
(
1--GII ( x,112)
87
Kexpa (x,, X;) exp
(21)
where 3,r O. These parameters determine the shape of the attractor basin that surrounds each pattern. While roughly sets the radius of attraction, 3 represents an inverse temperature which changes the steepness of the boundary of the attractor basin. Moreover, as long as the patterns are unique, the kernel matrix is invertible and we have Kexp, = K [44].
We will now analyze the noise robustness and storage capacity of this model. To make the analysis tractable, we will operate in the regime of low temperatures, meaning the limit - 0o. We first establish the following three properties.
Property 5.1 (The Exp; network at zero temperature). Given set of unique patterns {5" (5") J== with min 5 5"1l2 r, the state update rule for the Expg network at B - oo reduces to
s(++1) = =xO(r- - | s(1
(22)
where ((-) is the Heaviside function with ((0) e (see Appendix E.2.1).
Remark. In geometrical terms, Property 5.1 states that the boundary of the basin of attraction surrounding each pattern becomes sharp (N - diimenniooa hypersphere with radius in the limit 3 + co. For lower, finite B, the spherical boundary becomes increasingly fuzzy. From the perspective of an energy landscape, each pattern lies in an N N-dimensional energy minimum with infinitely steep walls when / + co. As 3 is lowered, the barriers become progressively smoother.
M Property 5.2 (Convergence in one step). Given set of unique patterns {E") μ=1 with min |15"" - -2'll2 > 2r, the Expe network at B + co, initialized at (0)) = e + AC, will converge to ed in one step A <.
Property 5.3 (No spurious attractors). Given a set of unique patterns {E") with minnezall IlE" 12112 > 2r and žyμ 11 rr((1 e the only attractors of the dynamics of the Exp network at 3 + xo are the points [
Ju=1, together with 0 if 3p ||E"112 ST.
Remark. Properties 5.2 and 5.3 can be shown to be true simply by inserting the expression ( g" + A in Eq. 22. Assuming no overlaps between the basins of attraction, a quick calculation shows that (1)) ş" || < r. If, on the other hand, the network is initialized such that ||s(0) - > r, Vμ,, one always obtains s(2) co where g0) is either 0 or the pattern closest to In other words, the network recalls pattern only if the initialization is close enough to it. If located far from all patterns, the network assumes an "agnostic" state, represented either by the origin or the pattern closest to the origin (if the origin happens to be located within basin of attraction).
In the following two properties, we evaluate how the radius of attraction r determines the maximum input noise tolerance and storage capacity.
Property 6 (Robustness to white noise). Assume that we are given a set of unique patterns I(,,IN) with min,,,yylll"" -2112= > 2r, and that the Exp3 network is initial-
E),
El....,gh EM ized in a distorted pattern (0)) Ed (5*+ E,, where N'O, o2In), Then, at 3 co, the maximum noise variance 2 with which &" can be recovered in at least 50% of trials is
- r
(23)
Property 7 (Exponential storage capacity). At 3 + oc, and for N >> 1, the average maximum number of patterns sampled from IN,IN) that the Exp network can store and recall without errors is lower-bounded according to
N(1
20 maxe
M
V 2 - 22 maxx exp
(24)
8
where 2nne is the maximum white noise variance tolerated by the network.
8

---

Remark, Proofs can be found in Appendices E.2.2 and E.2.3. Note that Property 7 is valid in the range of max - 1/2. While the bounds are fairly tight at the upper end of the range, they become loose when o mmax - O. In this limit, which is equivalent to r - O, the storage capacity tends to infinity, as the risk of interference between patterns vanishes when their radius of attraction becomes infinitesimal.
Comparison to the softmax network. If patterns are randomly placed on a hypersphere in- stead of being normally distributed, the state update rule in Eq. 22 reduces to the form s((+1) = X e T s(() - 0), where is a fixed threshold. While the capacity remains exponential (sce Appendix E.3.1), the basin of attraction surrounding each pattern now forms spherical cap instead of a ball. We can compare this to the softmax network at zero temperature, given by g(++1) = 1im X soflmax(33X ((t)) = X arg max(X s()). This model differs from the Exp; only in replacement of θ with arg max. This changes the shape of the attractor basins from spherical caps to Voronoi cells, which parcellate the entire surface of the hypersphere into a Voronoi diagram (see Fig. 2). The boundary of each basin is now no longer radially symmetric around pattern, but instead extends as far as possible in all directions. Consequently, at 3 - oo, the softmax network has larger attractor basins and always converges one of the stored patterns, regardless of the initialization point (assuming this is not precisely on a boundary). In contrast, the Expa network may converge the origin if initialized far from all patterns. This can be interpreted response, which indicates that the model cannot associate the input query with any of its stored patterns.
agnostic
5 Discussion
Biological interpretation. Kernel memory networks can be mapped to the anatomical properties of biological neurons. Consider an individual neuron in the feature form of the recurrent network (Eq. 9). The state of neighboring neurons is first transformed through ($s)) and thereafter projected to the neuron through the weight matrix (A @ X)O(X)T, When the kernel is polynomial of degree P, so that K (X.,x) cross-terms, up to degree p. The input to each neuron, in other words, consists of the states of all other neurons, as well as all possible combinations of their multiplicative interactions. This neuron model can be viewed as a generalized form of, for example, the multiconnected neuron [49], the clusteron [43], or the sigma-pl pi unit [58, p. 73]. These are all perceptrons that include multiplicative input interactions as a means to model synaptic cross-talk and sensitivvnt on non-linear dendrites [55] (see Fig. 1).
(xF x, + 1)", the transformation (ss) consists of all elements in and their
In the kernel form (Eq. 10), each neuron is, again, implicitly comprised of a two-stage process, whereby the raw input s is first transformed through the function K (X,s) and then projected through the weight matrix A @ X. For any inner-product kernel K = kxx] xj), this representation can be directly identified as a two-layer neural network, where the hidden layer is defined by the weights X and the activation function k. This interpretation of the recurrent network was recently proposed in [35, 36] and discussed in relation hippocampal-cortical interactions involved in memory storage and recall; it is particularly reminiscent of the hippocampal indexing theory [6, 61]. However, the kernel form can also be viewed as a network in which each individual neuron is a generalized form of the two-layered pyramidal cell model [53, 54]. This was originally proposed as an abstract neuron model augmented with non- non-linear dendritic processing [41]. should be noted, however, that the idea of interpreting kernel methods as neural networks has longer history, and has been extensively analyzed in the case of, for example, radial basis functions [51, 52]. For further details, see Appendix F.
Summary. We have shown that conventional kernel methods can be used to derive the weights for hetero- and auto-assoo associatie memory networks storing binary or continuous-valus maximal noise tolerance. The result is family of optimal memory models, which we call kernel memory networks, which includes the SDM and MHN as special cases. This unifying framework facilitates an intuitive understanding of the storage capacity of memory models and offers new ways to biologically interpret these in terms of non-linear dendritic integration. This work formalizes the links between kernel methods, attractor networks, and models of dendritic processing.
patterns with

---

Figure 2: Plot of random patterns S together with attractor basins 3 - 0o. Dots represent patterns (M = 17) while thick and thin red lines correspond to the boundaries of the attractor basins according the Exp, network and the softmax network, respectively. The radius of the circular boundaries has been set to half the minimum pairwise distance between the patterns.
Future work. A unifying theoretical framework for memory modeling can be useful for the development of both improved bio-plausible memory models and for machine learning applications. First, recognizing that there exists algorithms for training optimally noise-robust classifiers and adapting these to biological constraints can aid in the development of normative synaptic three-factor learning rules [23]. Second, the theoretical link between neuron models, kernel functions, and storage capacity enables one to fit kernel memory networks to neurophysiologica data and to analyze the computational properties of biophysically informed memory models. Finally, unifying framework reveals that most memory models differ only in the choice of kernel (model complexity) and Lagrange parameters (model precision). This categorization simplifies the tailoring of memory models their application, and allows for the design of models whose properties fundamentally can depart from kernel memory networks, by, for example, choosing kernels not associated with reproducing kernel Hilbert space.
Acknowledgments and Disclosure of F Funding
This study was supported by funding from the Swiss government's ETH Board of the Swiss Federal Institutes of Technology to the Blue Brain Project, research center of the École Polytechnique Fédérale de Lausanne (EPFL.).
References
[1] Abbott, and Arian, Y. (1987). "Storage capacity of generalized networks". Physical Review 36, pp. 5091-5094.
[2] Amit, D. J., Gutfreund, H., and Sompolinsky. H. (1985). "Storing infinite numbers of patterns in spin glass model of neural networks". Physical Review Letters 55, pp. 1530-1533.
Anlauf, J. and Bichl, M. (1989). "The AdaTron: adaptive perceptron algorithm". Europhysics
Letters 10, pp. 687-692.
[4] Ardeshir, N., Sanford, C., and Hsu, D. (2021). "Support vector machines and linear regression coincide with very high-dimensional features". Advances in Neural Information Processing Systems 35.
Bansal, Y., Advani, M., Cox, D. D., and Saxe, M. (2018). "Minnorm training: algorithm for training over pee-paaraeterizee deep neural networks" Preprint. arXiv: 1806.00730.
[6] Barry, N. and Maguire, A. (2019). "Remote memory and hippocampus: a constructive critique". Trends in Cognitive Sciences 23, pp. 128-142.
[7] Boros. G. and Moll, V. (2004). Irresistible integrals: symbolics, analysis and experiments in the evaluation of integrals. Cambridge University Press.
Bricken, T. and Pchlevan, C. (2021). "Attention approximates sparse distributed memory". Advances Neural Information Processing Systems 34.
10

---

[9] Casali, D.. Costantini, G., Perfetti, R., and Ricci, E. (2006). "Associative memory design using support vector machines". IEEE Trunsactions on Neural Networks 17, pp. 1165-1174.
[10] Chen, and Xu. (2020). "Deep neural tangent kernel and Laplace kernel have the same RKHS". Proceedings of the 202/ International Conference on learning Representations.
[11] Cho, Y. and Saul, L. (2009). "Kerell methods for deep learning". Advances in Neural Information Processing Systems 22.
[12] Chou, P. A. (1989). "The capacity of the Kanerva associative memory". IEEE Transactions on Information
Theory 35, pp. 281-298.
[13] Cortes, C. and Vapnik, V. (1995). "Support- networks". Machine Learning 20, pp. 273-297.
[14] Cotter, A., Shaley-Shwarzz, S., and Srebro, N. (2012). "The kemelized stochastic batch perceptron". Proceedings of the 29th International Conference on Muchine Learning, pp. 739-746.
[15] Cover, T. M. (1965). "Geometrical and statistical properties of systems of linear incqualities with applications pattern recognition".. IEEE Transactions on Electronic Computers 5C-14, pp. 326 334,
[16] Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., and Singer, Y. (2006). "Online passive-aggressive
algorithms". Journal of Machine Learning Research 7, pp. 551-585.
[17] Demircigil, M., Heusel, J., Löwe, M.. Upgang. S., and Vermet, F. (2017). "On model of associative memory with huge storage capacity" Journal of Statistical Physics 168, pp. 288 289.
[18] Forrest, M. (1988).
aablessbiid and learning in neural networks". Journal of Physics A: Mathematical and General 21, pp. 245-255.
[19] Frieß, T.-T., Cristianini, N., and Camphell, C. (1998). "The Kernel-Adatron algorithm: a fast and simple learning procedure for support vector machines", Proceedings of the 15th International Conference on Machine Learning, pp. 188-196.
[20] Gardner, (1987a). "Maximum storage capacity in neural networks". Europhysics Letters 4. pp. 481- 485.
[21] Gardner, F. (1 (1987b). "Multiconnecttd neural network models". Journal of Physics A: Mathematical and General 20, pp. 3453-3464,
[22]
Gardner, E. (1988). "The space of interactions in neural network models". Journal of Physics A: Mathe- matical and General 21. pp. 257-270.
[23] Gerstner, W., Lehmann, M., Liakoni, V., Corneil, D., and Brea, J. (2018). "Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian thrce-factor learning rules". Frontiers in Neural Circuits 12.
[24] Graves, A., Wayne, G., and Danihelka, I. (2014). "Neural Turing machines". Preprint, 1rXiv:1410.5400
[25] Hintzman, D. L. (1984). "MINERVA
simulation model of human memory". Behavior Research Methods, Instruments, & Computers 16, pp. 96-101.
[26] Hopfield, J. (1982). "Neural networks and physical systems with emergent collective computational abilities". Proceedings of the National Academy of Sciences 79, pp. 2554-2558.
[27] Hopfield, J. J. (1984). "Neurons with graded response have collective computational properties like those of two-state neurons".. Proceedings of the National Academy of Sciences 81, pp. 3088-3092.
[28] Jacot, A., Gabriel, F., and Hongler, C. (2018). "Neural tangent kernel: convergence and generalization in neural networks". Advances in Neural Information Processing Systems 31.
[29] Ji, Z., Srebro, N., and Telgarsky, M. (2021). "Fast margin maximization via dual acceleration". Proceed- ings of the 38th International Conference Machine Learning, pp. 4860-4869.
[30] Kanerva, (1988). Sparse distributed memory. MIT Press.
[31] Keeler, J. D. (1988). "Comparison between Kanerva's ' SDM and Hopfield-typs neural networks". Cogni- five Science 12, pp. 299-329.
[32] Kepler, T. B. and Abbott, L. F. (1988). "Domains of attraction in neural networks". Journal de Physique 49, pp. 1657-1662.
[33] Koiran, P. (1994). "Dynamics of discrete time, continuous state Hopfield networks". Neural Computation
6. pp. 4pp-455466.
[34] Krauth, W. and Mezard, M. (1987). "Learning algorithms with optimal stability in neural networks". Journal of Physics A: Mathematical and General 20, pp. L745-L.522.
[35] Krotov, D. and Hopfield. J. (2016). "Dense associative memory for pattern recognition". Advances in Neural Information Processing Systems 29, pp. 1172-1180.
[36] Krotov, D. and Hopfield, J. J. (2020). "Large associative memory problem in ncurobiology and machine learning", Proceedings of the 9th International Conference on Learning Representations.

---

[37] Lee, C., Doolen, G., Chen, H. H. Sun, G. Z.. Maxwell, T.. Lee. H. Y., and Giles, C. L. (1986). "Machine learning using higher order correlation network". Physica D: Nonlinear Phenomena 22, pp. 276-306.
[38] Lee. and Kim, W. C. (2014). Concise formulas for the surface area of the intersection of two hyper- spherical caps. Technical Report. Department of Industrial and Systems Engineering, KAIST.
[39] Li, S. (2011). "Concise formulas for the area and volume of a hyperspherical cap". Asian Journal of Muthematics and Statistics 4, pp. 66-70.
[40] MacKay, D. C. (2003). Information theory, inference and learning algorithmss Cambridge University Press.
[41] Major, G., Larkum, M. E., and Schiller, J. (2013). "Active properties of ncocortical pyramidal neuron dendrites". Annual Review of Neuroscience 36, pp. 1-24.
[42] McEliece. R., Posner, E.,. Rodemich, E., and Venkatesh, S. (1987). "The capacity of the Hopfield associative memory". IEEE Transactions on Information Theory 33, pp. 461-482.
[43] Mel, B. W. (1991). "The clusteron: toward a simple abstraction for a complex neuron", Advances in Neural Information Processing Systems 4, pp. 35-42,
[44] Micchelli, C. A. (1986). "Interpolation of scattered data: distance matrices and conditionally positive definite functions". Constructive Approximation 2. pp. 11-22.
[45] Millidge, B., Salvatori, T., Song. Y.. Lukasiewicz. T., and Bogacz, R. (2022). "Universal Hopfield networks: general framework for single- shot associative memory models". Proceedings of the 39th International Conference on Machine Learning, pp. 15561-15583.
[46] Nacson, M. S., Lec, J., Gunasckar, S., Savarese, P. H. P., Srebro, N., and Soudry, D. (2019). "Convergence of gradient descent on separable data". Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, pp. 3420-3428.
[47] Neal, R. M. (1996). "Priors for infinite networks". Bayesian Learning for Neural Networks. Ed. by Neal, R. M. Vol. 118. Lecture Notes in Statistics. Springer. pp. 29-53.
[48]
Nowicki, D. and Siegelmann, H. (2010). "Flexible kernel memory". PLOS ONE 5, c10955.
[49] Peretto, P. and Niez, J. J. (1986). "Long term memory storage capacity of multiconnected neural networks". Biological Cybernetics 54, pp. 53-63.
[50] Personnaz, L., Guyon, I., and Dreyfus, G. (1986). "Collective computational properties of neural networks:
New learning mechanisms". Physical Review A 34, pp. 4217-4228.
[51] Poggio, T. and Girosi, F. (1990a). "Networks for approximation and learning". Proceedings of the IEEE 78, pp. 1481-1497.
[52] Poggio, T. and Girosi, F. (1990b). "Regularization algorithms for learning that are equivalent to multilayer networks". Science 247. pp. 978-982.
[53] Poirazi, P., Brannon, T., and Mel, B. W. (2003). "Pyramidal neuron as two-layer neural network". Neuron 37, pp. 989-999.
[54] Poirazi, P. and Mel, B. W. (2001). "Impact of active dendrites and structural plasticity on the memory capacity of neural tissue". Neuron 29, pp. 779-796.
[55] Polsky, A., Mel, W., and Schiller. J. (2004). "Computational subunits in thin dendrites of pyramidal cells"". Nature Neuroscience 7. pp. 621-627.
[56] Radhakrishnan, A., Belkin, M., and Uhler, C. (2020). "Overparamoterized neural networks implement associative memory", Proceedings of the National Academy of Sciences 117, pp. 27162-27170
[57] Ramsauer, H., Schäff, B., Lehner, J., Seidl, P., Widrich, M., Gruber, I.., Holzleimnrr, M., Adler, T., Kreil, D., Kopp. M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. (2021). "Hopfield networks is all you need"". Proceedings of the 9th International Conference on Learning Representations.
[58] Rumelhart, D. E. and McClelland, J. L. (1986). Parallel distributed processing: explorations in the microstructure of cognition. Vol. 1. MIT Press.
[59] Soudry, D., Hoffer, F.., Nacson, M. S., Gunasckar, and Srehro, N. (2018). "The implicit bias of gradient descent on separable data". The Journal of Machine Learning Research 19, pp. 2822-2878.
[60] Suykens, J. and Vandewalle, J. (1999). "Least squares support vector machine classifiers". Neural Processing Letters 9, pp. 293-300.
[61] Teyler, J. and Rudy, J. W. (2007). "The hippocampal indexing theory and episodic memory: updating the index". Hippocampus 17, pp. 1158-1169.
[62] Tony Cai, and Jiang, T. (2012). "Phase transition in limiting distributions of coherence of high- dimensional random matrices". Journal of Multivariate Analysis 107, pp. 24-39.
[63] Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.P., and Salakhutdinov, R. (2019). "Transformer dis- section: unified understanding for transformer's 's attention via the lens of kernel". Proceedings of the
12

---

2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-HJCNEL pp- 4344-4353.
[64]
Tyulmankov, Fang. Vadaparty, A., and Yang. G. R. (2021). "Biological learning in key-value memory networks", Advances Neural Information Processing Systems
[65]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). "Attention is all you need".. Advances in Neural Information Processing Systems 30.
[66] Weston, J.. Chopra, S., and Bordes, A. (2015). "Memory networks". Proceedings of the 2015 International Conference on Learning Representations.
[67]
Williams, C. (1996), "Computing with infinite networks". Advances in Neural Information Processing
Systems 9.
[68] Wright, M. and Gonzalez, J. E. (2021). "Transformers are deep kernel machines". Preprint. arXiv: 2106.01506.
non-Mercer binary
[69] Wu, Wayne, G., Graves, and Lillicrap. (2018). "The Kanerva machine: a generative distributed memory". Proceedings of the 2018 International Conference on Learning Representations.
13


Reviews:
[{"review_id": "Yn8nSBY-6A", "reviewer": "Reviewer_ztkh", "summary": "This paper is a theoretical work in which the authors derive a set of normative models that describe the general structure of memory networks that can perform error-free recall of a given set of input patterns. Starting from well-known properties of binary classifiers they show how hetero-associate and auto-associate memories can be formulated as feed-forward and recurrent SVM networks, respectively. They also characterize the storage capacity of these networks in terms of their Vapnik-Chervonenkis dimension. Importantly, they show that previously proposed memory models such as Kanerva's sparse distributed memory and Hopfield networks are special cases of the SVM networks. They also extend their auto-associate memory model to be able to store continuous-valued patterns instead of just binary-valued patterns. Finally, they discuss how the models developed here could potentially be mapped to anatomical properties of biological neurons. ", "questions": "- Are there any ideas on how the theory developed here could give rise to better models of memory? ", "limitations": "The authors discuss the limitations of their approach.\n\n ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is a purely theoretical work that develops general expressions for optimal weights in memory networks storing binary or continuous-valued patterns with maximal noise tolerance. It is interesting to think about how the general theory developed here could give rise to better models of memory, and how they could lead to better AI methods. \n\nAlso, the mapping of kernel attractor networks to anatomical properties of biological neurons is potentially an interesting avenue of research. \n\nThe presentation is also very clear and it is easy to follow the main ideas. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Lzi9fmZLQEx", "reviewer": "Reviewer_5xox", "summary": "This work provides a nice framework to describe hetero- and auto-associative memory models. This framework can be used to describe dynamics and capacity of different existing models, such as SDMs and Hopfield networks. As a background, the authors provide an explanation to kernel methods, that are then needed to understand the contributions of the paper: they show how a kernel trick for \"one hidden layer nets\" is equivalent to finding the best set of weights with minimum norm, and a general property of multilevel feedforward kernels.\nThe first contribution (Section 3), consists in defining auto associative memory models with binary states as recurrent SVMs. Particularly, they state that the larger the attractor basins, the more robust the model. This is then used to draw connections with prowler models in the literature, such as SDMs and Hopfield models. This study is then extended to include recent models, generalized to continuous patterns. \n", "questions": "You state that your \"hope is that a proper understanding of the theoretical overlap of contemporary memory models can facilitate the development\" of new models. Generally, I agree that fundamental works that are able to identify underlying common principles of different models are important in the literature. However, in your specific case, how do you think your formulation will help future progress in the field? How will your work influence future applications of AM models?\n\n\nMinor:\n\nRemove one of the two remarks at lines 82 and 102\n\n\n\n", "limitations": "The very few limitations (it is a theoretical paper) are discussed, and I do not see any negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n Very nice paper, well written and straight to the point: it does exactly what is stated in the title.\n\nCons:\n\nI believe the paper is suffering the 9 pages limit, as the authors use most of the space to derive the results, while having a little more space would allow interesting discussions. Particularly, the introduction and conclusion do not address future works/ possible implication of the work in any way.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FbgfkrB7nYi", "reviewer": "Reviewer_a96M", "summary": "This work pursued a classical yet interesting problem that designing a neural network to store a set of patterns with maximal noise robustness, and presented alternative understandings of this problem by investigating several kernel attractor networks. The claimed contributions can be summarized as follows:\n1. proposed two mathematical structures of feedforward and recurrent memory networks for binary patterns.\n2. show that the normative models include, as special cases, the classical and modern Hopfield network, as well as the SDM.\n3. derived a simple attractor network model for storing an exponential number of continuous-valued patterns with a finite basin of attraction.", "questions": "What is the definition or formulation of the maximal noise robustness. It is difficult to get any noise or robustness information from Property 1, 2.1, and 2.2 unless I missed something.", "limitations": "Nothing mentioned.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is organized clearly, and the authors provided many plots for illustrating the structure of the proposed networks.\n2. The derivation procedure of the feedforward and recurrent memory networks are easy to follow.\n3. The result about storing an exponential number of continuous-valued patterns with a finite basin of attraction seems to be a laudable performance.\n\nWeaknesses:\nLimited to my knowledge, this paper is too hard to follow and comment. Here, I list the most concerned points, which may hinder the acceptance.\n1. Is there something over-claimed? For example, the discussions about its similarity to attention and new biological interpretations seem only to give some understandings from the perspectives of attention and other fields with slight analysis. It would be better to provide detailed and solid support for this claim.\n2. There lacks some clear-out and comparative results between the proposed networks and the previous ones. For example, what are the advantages of this framework that the authors proposed here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept