Title: Dense Associative Memory Through the Lens of Random Features
Venue: NeurIPS 2024
Accepted: yes
Keywords: Associative Memory

Kernels

Random Features

Hopfield Network
Abstract: Dense Associative Memories are high storage capacity variants of the Hopfield networks that are capable of storing a large number of memory patterns in the weights of the network of a given size. Their common formulations typically require storing each pattern in a separate set of synaptic weights, which leads to the increase of the number of synaptic weights when new patterns are introduced. In this work we propose an alternative formulation of this class of models using random features, commonly used in kernel methods. In this formulation the number of network's parameters remains fixed. At the same time, new memories can be added to the network by modifying existing weights. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories and shares their desirable computational properties.

Dense Associative Memory Through the Lens of Random Features

Abstract

Dense Associative Memories are high storage capacity variants of the Hopfield networks that are capable of storing a large number of memory patterns in the weights of the network of a given size. Their common formulations typically require storing each pattern in a separate set of synaptic weights, which leads to the increase of the number of synaptic weights when new patterns are introduced. In this work we propose an alternative formulation of this class of models using random features, commonly used in kernel methods. In this formulation the number of network's parameters remains fixed. At the same time, new memories can be added to the network by modifying existing weights. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories and shares their desirable computational properties.

1 Introduction

Hopfield network of associative memory is an elegant mathematical model that makes it possible to store a set of memory patterns in the synaptic weights of the neural network [1]. For a given prompt σ i ( t = 0) , which serves as the initial state of that network, the neural update equations drive the dynamical flow towards one of the stored memories. For a system of K memory patterns in the D -dimensional binary space the network's dynamics can be described by the temporal trajectory σ i ( t ) , which descends the energy function

Here ξ µ i (index µ = 1 ...K , and index i = 1 ...D ) represent memory vectors. The neural dynamical equations describe the energy descent on this landscape. In this formulation, which we call the memory representation , the geometry of the energy landscape is encoded in the weights of the network ξ µ i , which coincide with the memorised patterns. Thus, in situations when the set of the memories needs to be expanded by introducing new patterns one must introduce additional weights.

Alternatively, one could rewrite the above energy in a different form, which is more commonly used in the literature. Specifically, the sum over the memories can be computed upfront and the energy can be written as

Duen Horng Chau

Hendrik Strobelt

A)

DistributedRepresentation(DrDAM)

B)DrDAMReconstructionsCloselyMatchMrDAM's

FeaturizedQuery

FeaturizedMemory

0.05

DrDAMEnergies

2

Energy

MrDAMEnergies

0.04

Sum

Energy

0.03

0.02

0.01

Concat

0.00

Query

Memory Matrix

NewPattern

MemoryRepresentation(MrDAM)

Query

DrDAMMrDAM

Each line is the energy descent of one image completion

Time

In this form one can think about weights of the network being the symmetric tensor T ij instead of ξ µ i . One advantage of formulating the model this way is that the tensor T ij does not require adding additional parameters when new memories are introduced. Additional memories are stored in the already existing set of weights by redistributing the information about new memories across the already existing network parameters. We refer to this formulation as distributed representation .

A known problem of the network (eqs. (1) and (2)) is that it has a small memory storage capacity, which scales at best linearly as the size of the network D is increased [1]. This limitation has been resolved with the introduction of Dense Associative Memories (DenseAMs), also known as Modern Hopfield Networks [2]. This is achieved by strengthening the non-linearities (interpreted as neural activation functions) in eq. (1), which can lead to the super-linear and even exponentially large memory storage capacity [2, 3]. Using continuous variables x ∈ R D , the energy is defined as 1

where the function g : R D → R D is a vector function (e.g., a sigmoid, a linear function, or a layernorm), the function F ( · ) is a rapidly growing separation function (e.g., power F ( · ) = ( · ) n or exponent), S [ x , x ′ ] is a similarity function (e.g., a dot product or a Euclidean distance), and Q is a scalar monotone function (e.g., linear or logarithm). For instance, in order to describe the classical Hopfield network with binary variables (eq. (1)) one could take: linear Q , quadratic F ( · ) = ( · ) 2 , dot product S , and a sign function for g i = sign ( x i ) = σ i . There are many possible combinations of various functions g , F ( · ) , S ( · , · ) that lead to different models from the DenseAM family [2-7]; many of the resulting models have proven useful for various problems in AI and neuroscience [8]. Diffusion models have been linked to even more sophisticated forms of the energy landscape [9, 10].

From the perspective of the information storage capacity DenseAMs are significantly superior compared to the classical Hopfield networks. At the same time, most 2 of the models from the DenseAM family are typically formulated using the memory representation, and for this reason

Query

DrDAM

require introduction of new weights when additional memory patterns are added to the network. The main question that we ask in our paper is: how can we combine superior memory storage properties of DenseAMs with the distributed (across synaptic weights) formulation of these models in the spirit of classical Hopfield networks (eq. (2))? If such a formulation is found, it would allow us to add memories to the existing network by simply recomputing already existing synaptic weights, without adding new parameters.

A possible answer to this question is offered by the theory of random features and kernel machines. Given an input domain X , kernel machines leverage a positive definite Mercer kernel function k : X × X → R + that measures the similarity between pairs of inputs. The renowned 'kernel trick' allows one to compute the inner-product

between two inputs x , x ′ ∈ X in a rich feature space defined by the feature map φ ( x ) without ever explicitly realizing the feature map φ ( x ) . Various machine learning models (such as support vector machines [12], logistic regression, and various others [13, 14]) can be learned with just access to pairwise inner-products, and thus, the kernel trick allows one to learn such models in an extremely expressive feature space. Kernel functions have been developed for various input domains beyond the Euclidean space such as images, documents, strings (such as protein sequences [15]), graphs (molecules [16], brain neuron activation paths) and time series (music, financial data) [17]. Common kernels for Euclidean data are the radial basis function or RBF kernel k ( x , x ′ ) = exp( -γ ∥ x -x ′ ∥ 2 2 ) and the polynomial kernel k ( x , x ′ ) = ( ⟨ x , x ′ ⟩ + b ) p . To appreciate the expressivity of these kernel machines, note that, for input domain R D , the RBF kernel corresponds to an infinite dimensional feature space ( Y = ∞ ) and the polynomial kernel to a O ( D p ) dimensional feature space.

Interpreting the composition of the separation and similarity functions in eq. (3) as the left hand side of the kernel trick eq. (4) we can map the energy into the feature space, using appropriately chosen feature maps. Subsequently, the order of the summations over memories and features can be swapped, and the sum over memories can be computed explicitly. This makes it possible to encode all the memories in a tensor T α , which we introduce in section § 3, that contains all the necessary information about the memories. The energy function then becomes defined in terms of this tensor only, as opposed to individual memories. This functionality is summarized in fig. 1. Additionally, we show examples of retrieved Tiny ImageNet images that have been memorised using the original DenseAM model, which we call MrDAM , and the 'featurized' version of the same model, which we call DrDAM (please see the explanations of these names in the caption to fig. 1). These examples visually illustrate that mapping the problem into the feature space preserves most of the desirable computational properties of DenseAMs, which normally are defined in the 'kernel space'.

Contributions:

We propose a novel approximation of a DenseAM network utilizing random features commonly used in kernel machines. This novel architecture does not require the storage of the original memories, and can incorporate new memories without increasing the size of the network.

We precisely characterize the approximation introduced in the energy descent dynamics by this architecture, highlighting the different critical factors that drive the difference between the exact energy descent and the proposed approximate one.

We validate our theoretical guarantee with empirical evaluations.

In the past, kernel trick has been used for optimizing complexity of the attention mechanism in Transformers [18], and those results have been recently applied to associative memory [19], given the various connections between Transformers and DenseAMs [4, 20]. Existing studies [18, 19] focus on settings when attention operation or associative memory retrieval is done in a single step update. This is different from our goals here, which is to study the recurrent dynamics of the associative memory updates and convergence of that dynamics to the attractor fixed points. Iatropoulos et al. [21] propose kernel memory networks which are a recurrent form of a kernel support vector machine, and highlight that DenseAM networks are special cases of these kernel memory networks. Making a connection between nonparametric kernel regression and associative memory, Hu et al. [22] propose a family of provably efficient sparse Hopfield networks [23, 24], where the dynamics of any given input are explicitly driven by a subset of the memories due to various entropic regularizations on the energy. DenseAMs have been also used for sequences [25, 26, 24]. To reduce the complexity of computing all the pairs of F ( S [ ξ , x ]) for a given set of memories and queries, Hu et al. [27] leverage a low-rank approximation of this separation-similarity matrix using polynomial expansions. The kernel trick has also recently been used to increase separation between memories (with an additional learning stage to learn the kernel), thereby improving memory capacity [28]. There are also very recent theoretical analysis of the random feature Hopfield networks [29, 30], where their focus in on the construction of memories using random features. Kernels are also related to density estimation [31], and recent works have leveraged a connection between mixtures of Gaussians and DenseAMs for clustering [32, 33]. Lastly, random features have been used for biological implementations of both Transformers and DenseAMs [34, 35].

To the best of our knowledge there is no rigorous theoretical and empirical comparison of DenseAMs and their distributed (featurized) variants in recurrent memory storage and retrieval settings, as well as results pertaining to the recovery of the fixed points of the energy descent dynamics. This is the main focus of our work.

2 Technical background

Given the energy function in eq. (3), a variable x is updated in the forward pass through the 'layers' of this recurrent model such that its energy decreases with each update. If the energy is bounded from below, this ensures that the input will (approximately) converge to a local minimum. This can be achieved by performing a 'gradient descent' in the energy landscape. Considering the continuous dynamics, updating the input x over time with d x /dt , we need to ensure that dE/dt < 0 . This can be achieved by setting d x /dt ∝ -∇ x E .

Discretizing the above dynamics, the update of an input x at the t -th recurrent layer is given by:

where η ( t ) is a (step dependent) step-size for the energy gradient descent, E ( t ) is the energy of the input after the t -th layer, and the input to the first layer x (0) ← x . The final output of the associative memory network after L layers is x ( L ) .

DenseAMs significantly improve the memory capacity of the associative memory network by utilizing rapidly growing nonlinearity-based separation-similarity compositions such as F ( S [ x , ξ µ ]) = exp( β ⟨ x , ξ µ ⟩ ) or F ( S [ x , ξ µ ]) = exp( -β / 2 ∥ x -ξ µ ∥ 2 ) or F ( S [ x , ξ µ ]) = ( ⟨ x , ξ µ ⟩ ) p , p > 2 , among other choices, with β > 0 corresponding to the inverse temperature that controls how rapidly the separation-similarity function grows. However, these separation-similarity compositions do not allow for the straightforward simplifications as in eq. (2), except for the power composition. For a general similarity function, the update based on gradient descent over the energy in eq. (3) is given by:

For example, with Q ( · ) = (1 /β ) log( · ) , F ( · ) = exp( β · ) , S [ ξ µ , x ] = ⟨ ξ µ , x ⟩ and g ( x ) = x / ∥ x ∥ 2 , the energy function and the corresponding update 3 are:

This form does not directly admit itself to a distributed storage of memories as in eq. (2), and thus, in order to perform the gradient descent on the energy, it is necessary to keep all the memories in their original form. We will try to address this issue by taking inspiration from the area of kernel machines [36].

2.1 Random Features for Kernel Machines

The expressivity of kernel learning usually comes with increased computational complexity both during training and inference, taking time quadratic and linear in the size of the training set respectively. The groundbreaking work of Rahimi and Recht [37] introduced random features to generate explicit feature maps φ : R D → R Y for the RBF and other shift-invariant kernels 4 that approximate the true kernel function - that is ⟨ φ ( x ) , φ ( x ′ ) ⟩ ≈ k ( x , x ′ ) . Various such random maps have been developed for shift-invariant kernels [18, 19, 38] and polynomials kernels [39-41].

For the RBF kernel and the exponentiated dot-product or EDP kernel k ( x , x ′ ) = exp( ⟨ x , x ′ ⟩ ) , there are usually two classes of random features - trigonometric features and exponential features. For the RBF kernel k ( x , x ′ ) = exp( -∥ x -x ′ ∥ 2 2 / 2) , the trigonometric features [37] are given on the left and the exponential features [18] are on the right:

where ω α ∼ N (0 , I D ) ∀ α ∈ { 1 , . . . , Y } are the random projection vectors. 5 A random feature map φ for the RBF kernel can be used for the EDP kernel by scaling φ ( x ) with exp( ∥ x ∥ 2 2 / 2) . While the trigonometric features ensure that k ( x , x ) = ⟨ φ ( x ) , φ ( x ) ⟩ = 1 , the exponential features ensure that φ ( x ) ∈ R 2 Y + , which is essential in certain applications as in transformers [18, 19]. Furthermore, while the random samples ω α ∼ N (0 , I D ) are supposed to be independent, Choromanski et al. [42] show that the { ω 1 , . . . , ω Y } can be entangled to be exactly orthogonal to further reduce the variance of the approximation while maintaining unbiasedness. In general, the approximation of the random feature map is O ( √ D/Y ) , implying that a feature space with Y ∼ O ( D/ϵ 2 ) random features will ensure, with high probability, for any x , x ′ ∈ R D , | k ( x , x ′ ) - ⟨ φ ( x ) , φ ( x ′ ) ⟩ | ≤ ϵ . Scaling in the kernel functions such as exp( -β ∥ x -x ′ ∥ 2 2 / 2) or exp( β ⟨ x , x ′ ⟩ ) can be handled with the aforementioned random feature maps φ by applying them to √ β x with 〈 φ ( √ β x ) , φ ( √ β x ′ ) 〉 ≈ exp( -β ∥ x -x ′ ∥ 2 2 / 2) .

3 DrDAM with Random Features

Revisting the general energy function in eq. (3), if we have available an explicit mapping φ : R D → R Y such that ⟨ φ ( ξ µ ) , φ ( x ) ⟩ ≈ F ( S [ ξ µ , x ]) , then we can simplify the general energy function in eq. (3) to

Denoting T = ∑ µ φ ( ξ µ ) , we can write a simplified general update step for any input x as:

where d φ ( x ) /d x ∈ R Y × D is the gradient of the feature map with respect to its input. In the presence of such an explicit map φ , we can distribute the memory in a MrDAM into the single Y -dimensional vector T , and be able to apply the update in eq. (10). We can then use the random feature based energy gradient ∇ x ˆ E ( x ) instead of the true energy gradient ∇ x E ( x ) in the energy gradient descent step in eq. (5). 6 We name this scheme ' D istributed r epresentation for D ense A ssociative M emory' or DrDAM , and we compare the computational costs of DrDAM with the ' M emory r epresentation of D ense A ssociative M emory' or MrDAM in the following:

Proposition 1. With access to the K memories { ξ µ ∈ R D , µ ∈ J K K } , MrDAM takes O ( LKD ) time and O ( KD ) peak memory for L energy gradient descent steps (or layers) as defined in eq. (5) with the true energy gradient ∇ x E ( x ) .

Naively, the random feature based DrDAM would require O ( DY ) memory to store the random vectors and the ∇ x φ ( x ) matrix. However, we can show that we can generate the random vectors on demand to reduce the overall peak memory to just O ( Y ) . The various procedures in DrDAM are detailed in Algorithm 1. The RF subroutine generates the random feature for any memory or input. The ProcMems subroutine consolidates all the memories into a single T ∈ R Y vector. The GradComp subroutine compute the gradient ∇ x ˆ E . The following are the computational complexities of these procedures:

Proposition 2. The RF subroutine in Algorithm 1 takes O ( DY ) time and O ( D + Y ) peak memory.

Proposition 3. ProcMems in Algorithm 1 takes O ( DYK ) time and O ( D + Y ) peak memory.

Proposition 4. GradComp in Algorithm 1 takes O ( D ( Y + D )) time and O ( D + Y ) peak memory.

Thus, the computational complexities of DrDAM neural dynamics are (see appendix F.1 for proof and discussions):

Theorem 1. With a random feature map φ utilizing Y random projections { φ α , α ∈ { 1 , . . . , Y }}

and K memories { ξ µ ∈ R D , µ ∈ { 1 , . . . , K }} , the random-feature based DrDAM takes O ( D ( Y K + L ( Y + D ))) time and O ( Y + D ) peak memory for L energy gradient descent steps (or layers) as defined in eq. (5) with the random feature based approximation gradient ∇ x ˆ E ( x ) defined in eq. (10) .

However, note that the memory encoding only needs to be done once, while the same T can be utilized for L steps of energy gradient steps for multiple input, and the cost of ProcMems is amortized over these multiple inputs. We also show that the computational costs of the inclusion of a new memories ξ :

Proposition 5. The inclusion of a new memory ξ ∈ R D to a DrDAM with K memories distributed in T ∈ R Y takes O ( DY ) time and O ( D + Y ) peak memory.

The above result shows that inclusion of new memories correspond to constant time and memory irrespective of the number of memories in the current DenseAM. Next, we study the divergence between the output of a L -layered MrDAM using the energy descent in eq. (5) with the true gradient in eq. (6) and that of DrDAM using the random feature based gradient in eq. (10).

Theorem 2. Consider the following energy function with K memories { ξ µ ∈ R D , µ ∈ { 1 , . . . , K }} and inverse temperature β > 0 :

We further make the following assumptions: (A1) All memories ξ µ and inputs x lie in X = [0 , 1 / √ D ] D . (A2) Using a random feature map φ : R D → R Y using Y random feature maps, for any x , x ′ ∈ R d there is a constant C 1 > 0 such that ∣ ∣ exp( ∥ x -x ′ ∥ 2 2 / 2) -⟨ φ ( x ) , φ ( x ′ ) ⟩ ∣ ∣ ≤ C 1 √ D / Y . Given an input x ∈ X , let x ( L ) be the output of the MrDAM defined by the energy function in eq. (11) using the true energy gradient in eq. (6) and ˆ x ( L ) be the output of DrDAM with approximate gradient in eq. (10) using the random feature map φ using a constant step-size of η > 0 in (5) . Then

Assumption (A1) just ensures that all the memories and inputs have bounded norm, and can be achieved via translating and scaling the memories and inputs. Assumption (A2) pertains to the approximation introduced in the kernel function evaluation with the random feature map, and is satisfied (with high probability) based on results such as Rahimi and Recht [37, Claim 1] and Choromanski et al. [18, Theorem 4]. The above result precisely characterizes the effect on the divergence ∥ x ( L ) -ˆ x ( L ) ∥ of the (i) initial energy of the input E ( x ) - lower is better, (ii) the inverse temperature β - lower is better, (iii) the number of memories K - lower is better, (iv) the ambient data dimensionality D - lower is better, (v) the number of random features Y - higher is better, and (vi) the number of layers L - lower is better. The proof and further discussion are provided in appendix F.2. Note that theorem 2 analyzes the discretized system, but as the step-size η → 0 , we approach the fully contracting continuous model. An appropriate choice for the energy descent stepsize η simplifies the above result, bounding the divergence to O ( √ D/Y ) :

Corollary 1. Under the conditions and definitions of theorem 2, if we set the step size η = C 2 L (1+2 Kβe β/ 2 ) with C 2 < 1 , the divergence is bounded as:

These above results can be extended to the EDP based energy function E ( x ) = -1 / β log ∑ µ exp( β ⟨ ξ µ , x ⟩ ) + 1 / 2 ∥ x ∥ 2 2 using the same proof technique.

4 Empirical evaluation

To be an accurate approximation of the traditional MrDAM , DrDAM must empirically satisfy the following desiderata for all possible queries and at all configurations for inverse temperature β and pattern dimension D :

(D1) for the same query, DrDAM must predict similar energies and energy gradients as MrDAM ; and

(D2) for the same initial query, DrDAM must retrieve similar fixed points as MrDAM .

However, in our experiments we observed that the approximation quality of DrDAM is strongly affected by the choice of β and that the approximation quality decreases the further the query patterns are from the stored memory patterns, as predicted by theorem 2. We characterize this behavior in the following experiments using the trigonometric 'SinCos' basis function, which performed best in our ablation experiments (see appendix C), but note that the choice of the random features do play a significant role in the interpretations of these results.

4.1 (D1) How accurate are the energies and gradients of DrDAM ?

Figure 3 evaluates how well DrDAM , configured at different feature sizes Y , approximates the energy and energy gradients of MrDAM configured with different inverse temperatures β and storing random binary patterns of dimension D . The experimental setup is described below.

Approximation error of energy when queries are...

Approx error of energy gradients whenqueries are..

atstoredpatterns

nearstoredpatterns

random

at stored patterns

nearstoredpatterns

random

Error if

10

10-

randomly

>

guessed

β=50

100

β=50

10-2

MAE

D=30

10-1

β=40

103

β=30

β=30

β=10

102

β=10

101

101

10-2

100

D=100

MAE

10-1

10-3

102

10²

101

800

10

10-2

MAE

D=1000

100

10-

101

o0

Y

Y

Y

Y

Wegenerated 2 K = 1000 unique, binary patterns (where each value is normalized to be { 0 , 1 √ D } ) and stored K = 500 of them into the memory matrix Ξ of MrDAM . We denote these stored patterns as ξ µ ∈ { 0 , 1 √ D } D , µ ∈ J K K , where D is a hyperparameter controlled by the experiment. For a given β , the memory matrix is converted into the featurized memory vector T α := ∑ µ φ α ( ξ µ ) from eq. (9), where α ∈ J 2 Y K . The remaining patterns are treated as the 'random queries' x b far , b ∈ J K K (i.e., queries that are far from the stored patterns). Finally, in addition to evaluating the energy at these random queries and at the stored patterns, we also want to evaluate the energy at queries x b near that are 'near' the stored patterns; thus, we take each stored pattern ξ µ and perform bit-flips on 0 . 1 D of its entries.

For each set of queries x b ∈ { ξ b , x b near , x b far } , b ∈ J K K , and choice of β , Y , and D , we compute the M ean A pproximation E rror (MAE) between MrDAM 's energy E b := E ( x b ; β, Ξ) (whose gradient matrix is denoted ∇ x E b ) and DrDAM 's energy ˆ E b := ˆ E ( x b ; β, T ) (whose gradient is denoted ∇ x ˆ E b ).

We found it useful to visualize the results using log-scale and to compare the errors against the expected error of a 'random guess' of the energy/gradients (horizontal red dashed line in each plot of fig. 3). The 'random guess error' was empirically computed by sampling a new set of random queries x b guess , b ∈ J K K (independent of the reference queries) and computing the MAE between the standard energy on the reference queries vs. the approximate energies on the random queries. This error was averaged across Y for each β ; the highest average error across all β s is plotted.

Observation 1: DrDAM approximations are best for queries near stored patterns DrDAM approximations for both the energy and energy gradients are better the closer the query patterns are to the stored patterns. In this regime, approximation accuracy predictably improves when increasing

Hamming Error*of retrievals when queries are...

B)

Corrupt

LargerY->BetterApproximation

Ground

Retrieving

atstoredpatterns

nearstoredpatterns

random

Query

Truth

randomnoise

@error = 0.5

D=30

E°0

β=40

Perfect

0.1

@error =0

retrievals

0.0

0.5

β=60

Largerβ

needs,

D=100

0.3

largerY

β=70

β=75

D=1000

β=80

Y= 10k

40k

90k

160k

360k

*Hamming error is squared L2 error on binary data

the value for Y within 'reasonable' values (i.e., values corresponding into sizes of featurized queries and memories that can operate within 46GB of GPU memory).

Observation 2: DrDAM approximations worsen as inverse temperature β increases Across nearly all experiments, DrDAM approximations worsen as β increases. At queries near the stored patterns, β = 50 has an energy error approximately 10 × that of β = 30 and 100 × that of β = 10 across all Y . At high D and when queries are far from the patterns, the error of β = 50 approaches 1000 × the error of β = 10 . This observation similarly holds for the errors of corresponding gradients, corroborating the statement of theorem 2.

Observation 3: DrDAM approximations break at sufficiently high values of D and β In general, DrDAM 's approximation errors remain the same across choices for D , especially when the queries are near the stored patterns. However, when both β and D are sufficiently large (e.g., β ≥ 40 and D ≥ 100 in fig. 3), increasing the value of Y does not improve the approximation quality: DrDAM continues to return almost random gradients and energies. We explore this phenomenon more in § 4.2 in the context of the retrievability of stored patterns.

4.2 (D2) How accurate are the memory retrievals using DrDAM ?

Memory retrieval is the process by which an initial query x (0) descends the energy function and is transformed into a fixed point of the energy dynamics. This process can be described by the discrete update rule in eq. (5), where E can represent either MrDAM 's energy or the approximate energy of DrDAM . A memory is said to be 'retrieved' when | E ( x ( L ) ) -E ( x ( L -1) ) | < ε for some small ε > 0 , at which point x ( L -1) ≈ x ( L ) =: x ⋆ is declared to be the retrieved memory after L iterations because x ⋆ lives at a local minimum of the energy function E .

Quantifying retrieval error Given the same initial queries x (0) ∈ { 0 , 1 √ D } D , we want to quantify the difference between the fixed points ˆ x ⋆ retrieved by descending DrDAM 's approximate energy and the fixed points x ⋆ retrieved by descending the energy of MrDAM . We follow the experimental setup of § 4.1, only this time we run full memory retrieval dynamics until convergence.

Note that since energy uses an L2-similarity kernel, memory retrieval is not guaranteed to return binary values. Thus, we binarize x ⋆ by assigning each entry to its nearest binary value before computing the normalized Hamming approximation error ∆ H , i.e.,

The choice of normalized Hamming approximation error ∆ H on our binary data is equivalent to the squared L2 error on the left side of our bound in eq. (13) (up to a linear scaling of 1 √ D ).

Figure 4A shows the results of this experiment. Many observations from § 4.1 translate to these experiments: we notice that retrieval is random at high β and D , and that retrievals are of generally higher accuracy nearer the stored patterns. However, we notice that high β values can retrieve better approximations than lower values of β when the queries are at or near stored patterns. Additionally, for sufficiently high β (e.g., see D = 1000 , β = 50 near stored patterns), this accompanies an interesting 'thresholding' behavior for Y where retrieval error starts to improve rapidly once Y reaches a minimal threshold. This behavior is corroborated in the high D regime in fig. 4B.

Visualizing retrieval error Figure 4B shows what retrieval errors look like qualitatively. We stored K = 10 random images from CIFAR10 [43] into the memory matrix of MrDAM , resulting in patterns of size D = 3 × 32 × 32 = 3072 , and compared retrievals using β s that produced meaningful image results with MrDAM . To keep β values consistent with our previous experiments, each pixel was normalized to the continuous range between 0 and 1 √ D s.t. ξ µ i ∈ [0 , 1 √ D ] , with µ ∈ J K K and i ∈ J D K .

From § 4.1 and fig. 4A, we know that approximate retrievals are inaccurate at high β and high D if the query is far from the stored patterns. However, this is exactly the regime we test when retrieving images in fig. 4B. The visible pixels (top half of the image) are clamped while running the dynamics until convergence. Retrieved memories at different configurations for DrDAM are plotted against their corresponding MrDAM retrievals in fig. 4B.

As β increases, insufficiently large values of Y fail to retrieve meaningful approximations to the dynamics of MrDAM . We observe that image completions generally become less noisy as Y increases, but with diminishing improvement in perceptible quality after some threshold where DrDAM goes from predicting noise to predicting meaningful image completions.

5 Conclusion

Our study is explicitly designed to characterize where DrDAM is a good approximation to the energies and dynamics of MrDAM . In pushing the limits of the distributed representation, we discovered that DrDAM is most accurate when: (1) query patterns are nearer to the stored patterns; (2) β is lower; and (3) Y is large. Error bounds for these situations are explicitly derived in theorem 2 and empirically tested in § 4.

We have explored the use of distributed representations via random feature maps in DenseAMs. We have demonstrated how this can be done efficiently, and we precisely characterized how it performs the neural dynamics relative to the memory representation DenseAMs. Our theoretical results highlight the factors playing a role in the approximation introduced by the distributed representations, and our experiments validate these theoretical insights. As future work, we intend to explore how such distributed representations can be leveraged in hierarchical associative memory networks [44, 45], which can have useful inductive biases (e.g., convolutions, attention), and allow extensions with multiple hidden layers.

References

John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences , 79(8):2554-2558, 1982.

Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems , 29, 2016.

Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics , 168(2):288299, 2017.

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum? id=tL89RnzIiCd .

Dmitry Krotov and John J Hopfield. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations , 2021.

Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning , pages 15561-15583. PMLR, 2022.

Thomas F Burns and Tomoki Fukai. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations , 2022.

Dmitry Krotov. A new frontier for hopfield networks. Nature Reviews Physics , 5(7):366-367, 2023.

Benjamin Hoover, Hendrik Strobelt, Dmitry Krotov, Judy Hoffman, Zsolt Kira, and Duen Horng Chau. Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models, 2023. URL https://arxiv.org/abs/2309.16750 .

Luca Ambrogioni. In Search of Dispersed Memories: Generative Diffusion Models Are Associative Memory Networks. Entropy , 26(5), 2024. ISSN 1099-4300. doi: 10.3390/ e26050381. URL https://www.mdpi.com/1099-4300/26/5/381 .

Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https: //api.semanticscholar.org/CorpusID:16664790 .

Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning , 20:273-297, 1995. URL https://link.springer.com/content/pdf/10.1007/bf00994018.pdf .

Ryan R Curtin, Parikshit Ram, and Alexander G Gray. Fast exact max-kernel search. In Proceedings of the 2013 SIAM International Conference on Data Mining , pages 1-9. SIAM, 2013. URL https://doi.org/10.1137/1.9781611972832.1 .

Ryan R Curtin and Parikshit Ram. Dual-tree fast exact max-kernel search. Statistical Analysis and Data Mining: The ASA Data Science Journal , 7(4):229-253, 2014. URL https://doi. org/10.1002/sam.11218 .

Christina Leslie, Eleazar Eskin, and William Stafford Noble. The spectrum kernel: A string kernel for svm protein classification. In Biocomputing 2002 , pages 564-575. World Scientific, 2001. URL https://www.worldscientific.com/doi/abs/10.1142/9789812799623_ 0053 .

Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics , 21(suppl_1):i47-i56, 2005. URL https://dl.acm.org/doi/abs/10.1093/ bioinformatics/bti1007 .

K-R Müller, Alexander J Smola, Gunnar Rätsch, Bernhard Schölkopf, Jens Kohlmorgen, and Vladimir Vapnik. Predicting time series with support vector machines. In International conference on artificial neural networks , pages 999-1004. Springer, 1997. URL https: //link.springer.com/chapter/10.1007/BFb0020283 .

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. Proceedings of ICLR , 2020. URL https://arxiv.org/pdf/2009. 14794.pdf .

Deepali Jain, Krzysztof Marcin Choromanski, Kumar Avinava Dubey, Sumeet Singh, Vikas Sindhwani, Tingnan Zhang, and Jie Tan. Mnemosyne: Learning to train transformers with transformers. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=Fdfyga5i0A .

Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy transformer. Advances in Neural Information Processing Systems , 36, 2024. URL https://proceedings.neurips.cc/paper_ files/paper/2023/file/57a9b97477b67936298489e3c1417b0a-Paper-Conference. pdf .

Georgios Iatropoulos, Johanni Brea, and Wulfram Gerstner. Kernel memory networks: A unifying framework for memory modeling. Advances in neural information processing systems , 35:35326-35338, 2022. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/e55d081280e79e714debf2902e18eb69-Paper-Conference.pdf .

Jerry Yao-Chieh Hu, Bo-Yu Chen, Dennis Wu, Feng Ruan, and Han Liu. Nonparametric modern hopfield models. arXiv preprint arXiv:2404.03900 , 2024. URL https://arxiv.org/pdf/ 2404.03900.pdf .

Jerry Yao-Chieh Hu, Donglin Yang, Dennis Wu, Chenwei Xu, Bo-Yu Chen, and Han Liu. On sparse modern hopfield model. Advances in Neural Information Processing Systems , 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 57bc0a850255e2041341bf74c7e2b9fa-Paper-Conference.pdf .

Saul José Rodrigues Dos Santos, Vlad Niculae, Daniel C Mcnamee, and Andre Martins. Sparse and structured hopfield networks. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 43368-43388. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/ santos24a.html .

Hamza Chaudhry, Jacob Zavatone-Veth, Dmitry Krotov, and Cengiz Pehlevan. Long sequence hopfield memory. Advances in Neural Information Processing Systems , 36, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ aa32ebcdd2ce1bed4ef7f456fc8fa5c1-Paper-Conference.pdf .

Dennis Wu, Jerry Yao-Chieh Hu, Weijian Li, Bo-Yu Chen, and Han Liu. STanhop: Sparse tandem hopfield model for memory-enhanced time series prediction. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=6iwg437CZs .

Jerry Yao-Chieh Hu, Thomas Lin, Zhao Song, and Han Liu. On computational limits of modern hopfield models: A fine-grained complexity analysis. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 19327-19343. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/ v235/hu24j.html .

Dennis Wu, Jerry Yao-Chieh Hu, Teng-Yun Hsiao, and Han Liu. Uniform memory retrieval with larger capacity for modern hopfield models. arXiv preprint arXiv:2404.03900 , 2024. URL https://arxiv.org/pdf/2404.03827.pdf .

Matteo Negri, Clarissa Lauditi, Gabriele Perugini, Carlo Lucibello, and Enrico Maria Malatesta. Random feature hopfield networks generalize retrieval to previously unseen examples. In Associative Memory & Hopfield Networks in 2023 , 2023.

Matteo Negri, Clarissa Lauditi, Gabriele Perugini, Carlo Lucibello, and Enrico Malatesta. Storage and learning phase transitions in the random-features hopfield model. Physical Review Letters , 131(25):257301, 2023.

Bernard W Silverman. Density estimation for statistics and data analysis . Chapman & Hall/CRC, 1998.

Bishwajit Saha, Dmitry Krotov, Mohammed J Zaki, and Parikshit Ram. End-to-end differentiable clustering with associative memories. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 29649-29670. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/ saha23a.html .

Rylan Schaeffer, Nika Zahedi, Mikail Khona, Dhruv Pai, Sang Truong, Yilun Du, Mitchell Ostrow, Sarthak Chandra, Andres Carranza, Ila Rani Fiete, Andrey Gromov, and Sanmi Koyejo. Bridging associative memory and probabilistic modeling, 2024. URL https://arxiv.org/ abs/2402.10202 .

Leo Kozachkov, Ksenia V Kastanenka, and Dmitry Krotov. Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences , 120(34):e2219150120, 2023.

Leo Kozachkov, Jean-Jacques Slotine, and Dmitry Krotov. Neuron-astrocyte associative memory. arXiv preprint arXiv:2311.08135 , 2023.

Léon Bottou. Large-scale kernel machines . MIT press, 2007.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 2007. URL https://proceedings.neurips.cc/ paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf .

Valerii Likhosherstov, Krzysztof Marcin Choromanski, Kumar Avinava Dubey, Frederick Liu, Tamas Sarlos, and Adrian Weller. Dense-exponential random features: Sharp positive estimators of the gaussian kernel. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=S0xrBMFihS .

Purushottam Kar and Harish Karnick. Random feature maps for dot product kernels. In Artificial intelligence and statistics , pages 583-591. PMLR, 2012. URL https://proceedings.mlr. press/v22/kar12/kar12.pdf .

Ninh Pham and Rasmus Pagh. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 239-247, 2013. URL https://dl.acm.org/doi/pdf/10.1145/ 2487575.2487591 .

Raffay Hamid, Ying Xiao, Alex Gittens, and Dennis DeCoste. Compact random feature maps. In International conference on machine learning , pages 19-27. PMLR, 2014. URL https://proceedings.mlr.press/v32/hamid14.pdf .

Krzysztof M Choromanski, Mark Rowland, and Adrian Weller. The unreasonable effectiveness of structured random orthogonal embeddings. Advances in neural information processing systems , 30, 2017. URL https://proceedings.neurips.cc/paper/2017/file/ bf8229696f7a3bb4700cfddef19fa23f-Paper.pdf .

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Dmitry Krotov. Hierarchical associative memory, 2021. URL https://arxiv.org/abs/ 2107.06446 .

Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, and Dmitry Krotov. A universal abstraction for hierarchical hopfield networks. In The Symbiosis of Deep Learning and Differential Equations II , 2022. URL https://openreview.net/forum?id=SAv3nhzNWhw .

Peter W Frey and David J Slate. Letter recognition using holland-style adaptive classifiers. Machine learning , 6:161-182, 1991.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 19565-19594. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/li23l.html .

A Limitations

In this paper, we have explored the use of distributed representations via random feature maps in DenseAMs. However, we are only scratching the surface of opportunities that such distributed representations bring to DenseAMs. There are various aspects we do not cover: (i) We do not cover the ability of these distributed representations to provide (probably lossy) compression. (ii) We do not study the properties of DrDAM relative to MrDAM when DrDAM is allowed to have different step sizes and number of layers than MrDAM . A further limitation of our work is the limited number of datasets on which we have characterized the performance of DrDAM .

B Approximation error when increasing the number of stored patterns in DrDAM

§ 4.1 validated eq. (13), confirming that approximation error decreases as the number of random features Y increases under constant number of stored patterns K . We can also consider a related but different question: under constant number of random features Y , how does approximation error behave when increasing the number of stored patterns K ? Intuitively, DrDAM 's approximation should be good when a small number of patterns are stored in the network, and this approximation should worsen as we increase the number of stored patterns.

Figure 5 validates this intuition empirically, with the caveat that random queries generally improve in accuracy because the probability of being near a stored patterns (a regime that generally leads to higher accuracy of retrievals, see § 4) increases as we store more patterns into the network. For this experiment, Y = 2 e 5 was held constant across all experiments and each plotted approximation error is averaged over a number of queries equal to the number of stored patterns K . The experimental design otherwise exactly replicates that of fig. 3.

Approximation error of energy when queries are...

Approx error ofenergy gradientswhenqueries are..

atstoredpatterns

nearstoredpatterns

random

atstoredpatterns

nearstoredpatterns

random

Error if

10

100

β=50

randomly

guessed

β=50

10°

MAE

101

β=40

β=40

10-2

β=30°

D=30

10-1

β=30

β=10

10-2

β=10°

10-3

101

100

10°

MAE

101

D=100

10-

102

103

10~2

100

101

MAE

10-1

10°

102

D=1000

10-1

10-1

1000

0000

o

00

00

o

100

0

o

1000

K

K

K

K

K

K

C Ablation study: Comparing choices for basis function

Different basis functions can be used to approximate the RBF kernel used in the energy of the memory representation of the DAM in eq. (7). We considered the following kernels ('Cos', 'SinCos', 'Exp', 'ExpExp'), rewritten here as

where ω α ∼ N (0 , I D ) , α ∈ J Y K are the random projection vectors and b α ∼ U (0 , 2 π ) are random 'biases' or shifts in the basis function.

Figure 6 shows how well the above basis functions approximated the true energy and energy gradient at different values for β and size of feature dimension Y . Specifically, given the Letter dataset [46] which consists of 16-dimensional continuous vectors whose values were normalized to be between [0 , 1 √ D ] , we randomly selected 900 unique data points, storing 500 patterns into the memory and choosing the remaining 400 to serve as new patterns. We then compared how well the energy and energy gradients of the chosen basis function approximates the predictions of the original DAM.

We observe that the trigonometric basis functions (i.e., either Cos or SinCos) provide the most accurate approximations for the energy and gradients of the standard MrDAM , especially in the regime of high β which is required for the high memory storage capacity of DenseAMs. Surprisingly, the Positive Random Features (PRFs) of [18] do not perform well in the dense (high β ) regime; in general, trigonometric features always provide better approximations than the PRFs.

We conclude that the SinCos basis function is the best approximation for use in the experiments reported in the main paper, as this choice consistently produces the best approximations for the energy gradients across all values of β .

D TinyImagenet Experimental Details

In performing the qualitative reconstructions shown in fig. 1, we used a standard MrDAM energy (eq. (7)) configured with inverse temperature β = 60 . We approximated this energy in a DrDAM using the trigonometric 'SinCos' basis function shown in eq. (8) configured with feature dimension Y = 1 . 8 e 5 . The four images shown were selected from the Tiny Imagenet [11] dataset, rasterized into a vector, and stored in the memory matrix a MrDAM , resulting in a memory of shape (4 , 12288) . Energy descent for both MrDAM and DrDAM used standard gradient descent at a step size of 0 . 1 until the dynamics of all images converged (for fig. 1 after 300 steps, see energy traces). Visible pixels are 'clamped' throughout the duration of the dynamics by zeroing out the energy gradients on the visible top one-third of the image.

Error of Energy

Error of Gradients

at storedpatterns

atnewdata

at stored patterns

at new data

10-1

101

10-1

10-1

10-2

10-2

MAE

MAE

10-2

10-2

β=1

10-3

10-3

CosL2DAM

10-3

10-3

10-4

SinCosL2DAM

10-4

ExpL2DAM

ExpExpL2DAM

10-

10

101

10-1

10-

10-

10-2

102

10-2

102

MAE

β=10

MAE

10-3

10-3

10-3

10-3

10-4

104

10-4

10-

10-1

10-

10-

10

10-2

10-2

10-2

10-2

MAE

β=30

MAE

10-3

10-3

103

10-4

10

10-4

10-

10-1

10-

10

10-2

10-2

10-2

10-2

MAE

β= 60

MAE

10-3

10-3

10-3

10-3

10-4

10-4

10-4

10

400k

800k

翼

160k

400k

800

400k

800k

400k

800k

Y

Y

Y

Y

In MrDAM , the memory matrix necessarily grows linearly when storing new patterns ξ µ . However, the distributed memory tensor T of DrDAM does not grow when new patterns are stored. This means it is possible to compress the memories into a smaller tensor T where Y < NumPixels, provided that we operate in a regime that allows larger approximation errors in the retrieval and smaller initial occlusions. Figure 2 shows a variation of the setting of fig. 1 where stored patterns are actually compressed into DrDAM 's memory tensor, successfully storing 20 ∗ 12288 pixels from a distributed tensor of size Y = 2 e 5 and retrieving the memories with 40 % initial occlusion of the queries, a ∼ 20 %reduction in the number of parameters compared to MrDAM . All other hyperparameters are the same as was used to generate fig. 1, and convergence on all images occurs after 1000 steps.

E Details on Computational Environment for the Experiments

All experiments are performed on a single L40s GPU equipped with 46GB VRAM. Experiments were written and performed using the JAX [47] library for tensor manipulations. Unless otherwise noted, gradient computation was performed using JAX's powerful autograd mechanism. Experimental code with instructions to replicate the results in this paper are made available at this GitHub repository ( https://github.com/bhoov/distributed_DAM ), complete with instructions to setup the coding environment and run all experiments.

F Detailed Proofs and Discussions

F.1 Details for theorem 1

F.1.1 Proof of theorem 1

Proof of theorem 1. The proof above involves noting that, first we need to encode all the memories with ProcMems, which takes O ( DYK ) time and O ( Y + D ) peak memory using proposition 3.

Then we compute L gradients with GradComp for L iterations of energy gradient descent, taking O ( LD ( Y + D )) time and O ( D + Y ) peak memory using proposition 4.

Putting the runtimes together, and using the maximum of the peak memories gives us the statement of the theorem.

F.1.2 Comparing computational complexities of MrDAM and DrDAM

Note that, comparing the computational complexities of MrDAM in proposition 1 to that of DrDAM in theorem 1 does not directly provide any computational improvements as it would depend on the choices of D,K,L,Y . The main point of these results is to highlight, that once the memories are processed via ProcMems, the energy descent with DrDAM requires computation and memory that only depends on D and Y . And together with theorem 2 and corollary 1, we characterize situations where the energy descent divergence between MrDAM and DrDAM can be bounded with a choice of Y that only depends on D (and other parameters in the energy function) but not K .

While we do not claim or highlight computational gains over MrDAM , note that the peak memory complexity of MrDAM is O ( KD ) compared to O ( Y + D ) for DrDAM . Given that in the interesting regime of Y ∼ O ( D/ϵ 2 ) which upperbounds the energy descent divergence between DrDAM and MrDAM in corollary 1 to at most some ϵ > 0 , DrDAM is more memory efficient than MrDAM if the number of memories K > C/ϵ 2 for some sufficiently large positive constant C . Ignoring the time required to encode the memories into the distributed representation in DrDAM using ProcMems, the runtime complexities are O ( LKD ) for MrDAM compared to O ( LD ( Y + D )) for DrDAM . Again, considering the interesting regime of Y ∼ O ( D/ϵ 2 ) , DrDAM will be computationally more efficient than MrDAM if the number of memories K > ˜ CD/ϵ 2 for some sufficiently large positive constant ˜ C .

F.2 Details for theorem 2

F.2.1 Proof of theorem 2

Here we will make use of the following result from Li et al. [48]:

Lemma 1 (adapted from Li et al. [48] Lemma B.1) . For x , z ∈ R K with max i,j ∈ J K K ( x i -x j ) ≤ δ and max i,j ∈ J K K ( z i -z j ) ≤ δ , we have the following:

We now develop the following results:

Lemma 2. Under the conditions and notation of theorem 2, for x , z ∈ X , we have

Proof. Given the energy function in eq. (11), we can write the energy gradient ∇ x E ( x ) as:

where we applied lemma 1 to the softmax term in the right hand side of eq. (22) with δ = β/ 2 since all pairwise distances in X are in [0 , 1] .

Now we have

since ∥ ξ µ ∥ ≤ 1 and ∥ x + z ∥ ≤ ∥ x ∥ + ∥ z ∥ ≤ 2 . Putting eq. (26) in eq. (23), and using the fact that ∥ Ξ ∥ 2 ≤ K , we have

giving is eq. (16) in the statement of the lemma.

Given the structure of the energy gradient in eq. (18) of the energy function in eq. (11), we consider a specific energy gradient for this specific energy function instead of the generic energy gradient in eq. (10). We can rewrite the exact energy gradient as

Using random feature maps, we can write the approximate gradient as

where we again just need to store T and R as defined above and do not need to maintain the original memory matrix Ξ .

Lemma 3. Under the conditions and notation of theorem 2, and assuming that ⟨ φ ( x ) , φ ( x ′ ) ⟩ ≥ 0 ∀ x , x ′ ∈ X , for the approximate gradient ∇ x ˆ E ( x ) in eq. (32) , we have

Proof. We can expand out the left-hand side of eq. (33) as follows:

by reversing the simplifying steps made above to arrive at eq. (32).

Let use denote ϵ = C 1 √ D/Y the approximation in the kernel value induced by the random feature map φ . Then considering the terms in the denominator above, we have

Considering the terms in the numerators, we have

Let us define the following terms for convenience:

a = 1 / K ∑ µ exp( -β / 2 ∥ x -ξ µ ∥ 2 ) ξ µ

ˆ a = 1 / K ∑ µ 〈 φ ( √ β x ) , φ ( √ β ξ µ ) 〉 ξ µ

b = 1 / K ∑ µ exp( -β / 2 ∥ x -ξ µ ∥ 2 )

ˆ b = 1 / K ∑ µ 〈 φ ( √ β x ) , φ ( √ β ξ µ ) 〉

Then, based on our previous bounds, we know that

.

Note that (ˆ a/ ˆ b ) is in the convex hull of the memories since this is a weighted sum of the memories where the weights are positive and add up to 1. Thus within (ˆ a/ ˆ b ) ∈ [0 , 1 / √ d ] d , thus ∥ ˆ a/ ˆ b ∥ ≤ 1 . Now b = (1 /K ) exp( -βE ( x )) . Thus

giving us the right-hand side of eq. (33).

Proof of theorem 2. Expanding out the divergence D ( L ) after L energy descent steps, and using the fact that x (0) = ˆ x (0) = x , we have

Let us denote the individual terms above as d ( t ) with D ( L ) = ∑ t ∈ J L K d ( t ) . Also, let us denote by A = 2 C 1 Ke βE ( x ) √ D/Y and by B = (1 + 2 Kβe β/ 2 ) . Then writing out the t -th term

where the first term is bounded using lemma 2 and the definition of B , and the second term is bounded using lemma 3 and the definition of A .

Note that this gives us the recursion d ( t ) ≤ ηA + ηBD ( t -1) , and thus, D ( L ) ≤ ∑ t ∈ J L K η ( A + BD ( t -1) ) .

Writing out the recursion using induction, we can show that

Replacing the values of A and B above gives us the statement of the theorem.

F.2.2 Dependence on the initial energy E ( x ) of the input.

The divergence upper bound in eq. (12) (in theorem 2) depends on the term exp( βE ( x )) . However, note that, for the energy function defined in eq. (11), assuming that all memories and the initial queries are in a ball of diameter 1 (which is the assumption A1 in theorem 2), E ( x ) ≤ 1 2 -log K β , implying that exp( βE ( x )) ≤ exp( β/ 2) /K , and we can replace this in the upper bound and remove the dependence on E ( x ) .

However, an important aspect of our analysis is that the bound is input specific, and depends on the initial energy E ( x ) . As discussed above, this can be upper bounded uniformly, but our bound is more adaptive to the input x .

For example, if the input is initialized near one of the memories, while being sufficiently far from the remaining ( K -1) memories, then exp( βE ( x )) term can be relatively small. More precisely, with all memories and queries lying in a ball of diameter 1 , let the query be at a distance r < 1 to its closest memories, and as far as possible from the remaining ( K -1) memories. In this case, the initial energy E ( x ) ≈ -(1 /β ) log(exp( -βr/ 2) + ( K -1) exp( -β/ 2)) , implying that

For sufficiently small r < 1 , the above quantity can be relatively small. If, for example, r ∼ O (log K ) , then exp( βE ( x )) ∼ O ( K β ) , while r → 0 gives us exp( βE ( x )) → O (1) . This highlights the adaptive input-dependent nature of our analysis.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We lay out clearly our contribution of a novel approximation of the dynamical process for Dense Associative Memory that uses random features.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are characterized theoretically in § 3, empirically in § 4, and summarized in appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions, propositions, and theorems are given in § 3, with additional proofs in appendix F.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Extensive descriptions of the main experimental results are provided in § 4; experimental details that were not appropriate for the main paper are explained in appendix D and appendix E. Our proposed method itself is explained in detail in Algorithm 1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code and instructions to run all experiments in this paper is provided at this anonymous GitHub repository ( https://anonymous.4open.science/r/drdam ), which is included in appendix E of our paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are provided alongside the results reported in § 4. Computational environment for the experiments is described in appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars representing the 'standard error of the mean' (SEM) are shown on all quantitative plots (i.e., figs. 3, 4 and 6). We evaluated on large sample sizes s.t. the SEM is small, but still visible when the predictions have high variability.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We use the same computational environment for all experiments, which is described in appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work does not require human subjects. Datasets used are public and the nature of the evaluation is not dependent on any inherent biases. The work does not have unexpected anticipated societal harms or other harmful consequences requiring mitigation.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We do not anticipate any additional societal harms or other harmful consequences requiring mitigation beyond the well-known risks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No proprietary datasets were used. The models developed are not anticipated to have a high risk potential for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All owners of assets used in this paper are properly cited. No licensed work was used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper mainly contributes a new algorithm for approximating Dense Associative Memories. This is documented throughout the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research makes no use of crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research makes no use of crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce an iterative random feature Dense Associative Memory Model (DenseAM).\nIt is a kernel approximation of the standard DenseAMs using random feature decomposition and approximation. \nThe authors term this approach to DenseAMs as distributed representation/formulation of memory (DrDAM).\n\nTheoretically, they analyze the time complexity of the proposed method (Thm1) and the approximation error (Thm2), both benchmarked against the standard DenseAM (MrDAM). \nExperimentally, they evaluate the approximation accuracy of DrDAM's energies and gradients, defined as the MAE difference between DrDAM and MrDAM, and the retrieval accuracy of DrDAM. \nImportantly, their analysis of DrDAM considers its iterative nature, distinguishing their contribution from prior works.\n\nIn sum, they address important research questions and their approach is natural and intuitive based on recent advances in the random feature kernel trick.\n\nSTRENGTHS:\n* **Originality**: This paper expands our understanding of Dense Associative Memory Models (DenseAMs) by presenting a random feature DenseAM, termed DrDAM. The idea of DrDAM is novel and beneficial to the ML and associative memory community.\n\n* **Significance**: While the idea of random feature DenseAM is not entirely new, this work delves deeper than most prior studies. For example, to the best of my knowledge, there is nothing as concrete as the iterative Algorithm 1 presented in prior works. This represents a significant and solid step forward for the community, and I appreciate the effort.\n\n* **Theoretical Contribution**: This paper provides an efficiency and approximation analysis of DrDAM.\n\n* **Experimental Contribution**: They explore some aspects of DrDAM, including the approximation of energies and gradients and the approximation ability of DrDAM. Reproducible code is also provided.\n\nWEAKNESSES:\n## Summary of Weaknesses\n\nOverall, this paper presents a good idea, but it seems rushed, which affects the contribution of an otherwise very interesting work. Regardless of the final decision, I hope the authors find my comments helpful in further refining their draft.\n\nThere are a few areas that could benefit from further improvement and clarity:\n\n* **Experiments:** The current experimental results would be strengthened by including comparisons with established baselines and more ablation studies.\n* **Clarity:** Some of the theoretical results are difficult to interpret due to minor typos and a need for increased mathematical clarity.\n* **Motivation:** The motivation and problem statement could be articulated more convincingly. Even after multiple readings of the intro, it remains unclear if distributed representation is truly parameter-efficient beyond the nonparametric sense.\n* **Related Works:** Additionally, while not critical, the idea of Random Feature DenseAM (and Kernel DenseAM) has been discussed in prior works. The related discussion in this paper could be expanded to better position the contribution within the existing literature. For examples:\n  * ```Random Feature DenseAM``` Random Feature Hopfield Networks generalize retrieval to previously unseen examples https://openreview.net/forum?id=bv2szxARh2)\n  * ```Random Feature DenseAM``` Nonparametric Modern Hopfield Models https://arxiv.org/abs/2404.03900\n  * ```Kernel DenseAM``` Kernel Memory Networks: A Unifying Framework for Memory Modeling  https://arxiv.org/abs/2208.09416\n  * ```Kernel DenseAM``` Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models  https://arxiv.org/abs/2404.03827\n  * ```Kernel DenseAM``` Bridging Associative Memory and Probabilistic Modeling https://arxiv.org/abs/2402.10202\n\n\n\n**Note:** I have tried my best to evaluate this paper. I spent much more time on this review than on my previous ones because I like this paper. Still, I might have missed something. Please correct me if I am wrong with fair justifications. I am open to changing my opinion and raising the score.\n\n---\n\n## Major: My main concern with this paper is its soundness and inadequate experimental evidence\n\n### General\n\n\n* ```line23:``` \"Memory representation\" is a strange term for me. If you are referring to the way of encoding memory patterns onto the energy landscape, isn't it \"memory encoding\"? Representation generally involves learning, and to my understanding, there is no learning here.\n\n  >Thus, in situations when the set of the memories needs to be expanded by introducing new patterns one must introduce additional weights.\n\n  A side comment: this statement implies nonparametric, as suggested in [17].\n\n* ```distributed representation```: I find the description of distributed representation vague. With more memories, $K$ increases, resulting in more degrees of freedom in $\\mathbf{T}$. You can say the total number of entries in $\\mathbf{T}$ is fixed, but not the number of parameters, unless you clearly define \"parameters\" differently. The current draft makes it difficult to parse the soundness.  I believe the idea of distributed representation deserves more elaboration. The current draft seems nonparametric to me.\n\n  In my current review, I assume what is stated in lines 29-30 is correct.\n\n* ```Sec3```: Section 3 is hard to follow due to typos and the lack of clarity.\n\n* ```Missing Broader Impacts```: The authors didn't discuss both potential positive and negative societal impacts except in the checklist. My understanding is that this discussion needs to be included in the main text or the appendix, as the checklist is not considered part of the paper.\n\n---\n\n### Theory\n\n\n* ```line92```: Eqn. (6) is hard to follow without clearly defined $S,s$. \n\n  I suppose $S:\\mathbb{R}^d\\times \\mathbb{R}^d\\to \\mathbb{R}$ which makes $s\\in\\mathbb{R}$ a scalar realization of $S$?\n\n\n* ```No fixed point convergence results```: Per my understanding, DenseAMs are energy models with memories stored in their local minima. Memory retrieval is conducted through energy minimization. This setting requires convergence between local minima and fixed points of retrieval updates, such as the convergence results of [Ramsauer2020,Hu2023]. Otherwise, there may be unbounded retrieval error due to the mismatch between local minima and fixed points of retrieval updates. Specifically, I believe lines 106-110 are at best ambiguous.\n\n  * [Ramsauer2020] Hopfield Networks is All You Need. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G.K., and Greiff, V., 2020. arXiv preprint arXiv:2008.02217.\n  \n  * [Hu2023] On Sparse Modern Hopfield Model. Hu, J.Y.C., Yang, D., Wu, D., Xu, C., Chen, B.Y., and Liu, H., 2023. Advances in Neural Information Processing Systems, 36.\n\n* ```Theorem 1```: I feel Thm1 may be incorrect. Could you elaborate more on this?\n\n  * **My understanding:** Let $L$ be the number of updates, $K$ be the number of stored memories, $D$ be the memory dimension, and $Y$ be the feature space dimension. DrDAM improves time complexity from $O(LKD)$ to $O(D(Y K + L(Y + D))$, and improves peak memory from $O(KD)$ to $O(Y+D)$.\n  * **Comment1:** $O(D(Y K + L(Y + D))$ is clearly a typo and makes the result hard to parse.\n  * **Comment2:** Without specifying the relationships between $L$, $K$, $D$, and $Y$, it's hard to tell if this is truly an improvement. I can easily think of counterexamples.\n\n  Given these, can you kindly elaborate more on the significance of thm1?\n\n\n* ```Theorem 2```: Eqn. (12) is strange. Is there any guarantee ensuring $\\alpha L(1+2K\\beta e^{\\beta/2}) < 1$? If not, it doesn't make sense that more updates would result in larger error. Perhaps this is related to the missing fixed point convergence?\n\n---\n\n### Experiments\n\n* ```No Comparison to existing works:``` there is no baseline compared. \n\n* ```No efficiency evaluation1:``` Please include ablation studies on changing $L,K,D,Y$ to verify Thm1. \n\n* ```No efficiency evaluation2:``` Please also compare the efficiency with other methods, including linear, top-K, random feature, and random masked from [17], as well as dense [Ramsauer2020], sparse [Hu2023], and generalized sparse [Wu2023] MHNs. For example, the current draft needs some figures similar to Fig. 4 of [17] to justify the efficiency. Please do whatever makes sense to you. For instance, you can pick just two out of [Ramsauer2020, Hu2023, Wu2023] since they share similar structures.\n\n  * [Ramsauer2020] Hopfield Networks is All You Need. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G.K., and Greiff, V., 2020. arXiv preprint arXiv:2008.02217.\n  \n  * [Millidge2022] Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models. Millidge, B., Salvatori, T., Song, Y., Lukasiewicz, T., and Bogacz, R., 2022, June. In International Conference on Machine Learning (pp. 15561-15583). PMLR.\n  \n  * [Wu2023] Stanhop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction. Wu, D., Hu, J.Y.C., Li, W., Chen, B.Y., and Liu, H., 2023. arXiv preprint arXiv:2312.17346.\n  \n  * [Hu2023] On Sparse Modern Hopfield Model. Hu, J.Y.C., Yang, D., Wu, D., Xu, C., Chen, B.Y., and Liu, H., 2023. Advances in Neural Information Processing Systems, 36.\n\n\n\n* ```Compare with existing kernel DenseAMs.``` Also, a direct comparison with existing kernel methods is also necessary. For example,\n  * Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models. Wu, D., Hu, J.Y.C., Hsiao, T.Y., and Liu, H., 2024. arXiv preprint arXiv:2404.03827.\n  \n  * Kernel Memory Networks: A Unifying Framework for Memory Modeling. Iatropoulos, G., Brea, J., & Gerstner, W., 2022. Advances in Neural Information Processing Systems, 35, 35326-35338.\n  \n  * Bridging Associative Memory and Probabilistic Modeling. Schaeffer, R., Zahedi, N., Khona, M., Pai, D., Truong, S., Du, Y., Ostrow, M., Chandra, S., Carranza, A., Fiete, I.R., and Gromov, A., 2024. arXiv preprint arXiv:2402.10202.\n\n---\n\n## Minor\n\n* ```Inadequate Related Work Discussion```: A paragraph discussing related works is given at the end of the introduction. However, I feel it could be made more comprehensive to provide more background on the evolution of ideas and existing works in this field. At least, I feel the current draft is hard for a non-expert in DenseAMs to follow. Please see the following points.\n\n* ```line38:``` I believe here can be benefited by including more comprehensive references. There are other recent works showing exponentially large memory storage capacity in various settings and dense associative memory (or modern Hopfield) networks/models. For examples: (I am not sure of their exact relevance, so I will leave it to the authors to decide whether they should be included and discussed.)\n\n   * Hopfield Networks is All You Need. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G.K., and Greiff, V., 2020. arXiv preprint arXiv:2008.02217.\n  * Kernel Memory Networks: A Unifying Framework for Memory Modeling. Iatropoulos, G., Brea, J., & Gerstner, W., 2022. Advances in Neural Information Processing Systems, 35, 35326-35338.\n  * On Sparse Modern Hopfield Model. Hu, J.Y.C., Yang, D., Wu, D., Xu, C., Chen, B.Y., and Liu, H., 2023. Advances in Neural Information Processing Systems, 36.\n  * Stanhop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction. Wu, D., Hu, J.Y.C., Li, W., Chen, B.Y., and Liu, H., 2023. arXiv preprint arXiv:2312.17346.\n  * Bridging Associative Memory and Probabilistic Modeling. Schaeffer, R., Zahedi, N., Khona, M., Pai, D., Truong, S., Du, Y., Ostrow, M., Chandra, S., Carranza, A., Fiete, I.R., and Gromov, A., 2024. arXiv preprint arXiv:2402.10202.\n  * Sparse and Structured Hopfield Networks. Santos, S., Niculae, V., McNamee, D., and Martins, A.F., 2024. arXiv preprint arXiv:2402.13725.\n  * On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis. Hu, J.Y.C., Lin, T., Song, Z., and Liu, H., 2024. arXiv preprint arXiv:2402.04520.\n  * Outlier-Efficient Hopfield Layers for Large Transformer-Based Models. Hu, J.Y.C., Chang, P.H., Luo, R., Chen, H.Y., Li, W., Wang, W.P., and Liu, H., 2024. arXiv preprint arXiv:2404.03828.\n\n\n* ```line46:``` Similarly, here can be benefited by including more recent references\n  > DenseAM family [...] \n\n* ```line51```: A similar motivating question is asked and explored in [17] from a kernel regression perspective. I agree this paper contributes something different and even beyond [17]. However, given the similarity of aims between the two, the current draft should:\n  1. State the overlaps between the two works (both use a kernel approach to DenseAMs, and...).\n  2. State the clear distinctions between the two works (both use a kernel approach to DenseAMs, but...).\n\n  Such discussion will help readers parse the contributions of this work, which is currently lacking.\n\n  [17] Jerry Yao-Chieh Hu, Bo-Yu Chen, Dennis Wu, Feng Ruan, and Han Liu. Nonparametric modern hopfield models. arXiv preprint arXiv:2404.03900, 2024. URL https://arxiv.org/pdf/2404.03900.pdf.\n \n\n* ```line92```: \n  > those results have been recently applied to associative memory [15].\n\n  I might have missed something, but it would be better to make the discussion of prior work more complete if it's presented in its current form (e.g., a paragraph highlighting the novelty and contributions benchmarked against existing works). The application of the kernel trick to associative memory is not new. While [16,17] are discussed right after [15], I feel it's better to cite them together as [15,16,17]. Additionally, there are two papers related to the kernel trick in associative memory, which I have attached below. Again, I am not sure of their exact relevance, so I will leave it to the authors to decide whether they should be included and discussed.\n\n  - Iatropoulos, G., Brea, J., & Gerstner, W. (2022). Kernel memory networks: A unifying framework for memory modeling. Advances in neural information processing systems, 35, 35326-35338.\n  \n  -  Schaeffer, R., Zahedi, N., Khona, M., Pai, D., Truong, S., Du, Y., Ostrow, M., Chandra, S., Carranza, A., Fiete, I.R. and Gromov, A., 2024. Bridging Associative Memory and Probabilistic Modeling. arXiv preprint arXiv:2402.10202.\n\n* ```line97```: \n  > Another paper [17] uses kernels for studying the sparse modern Hopfield network. \n\n  This is incorrect. \n\n  As stated in their abstract, [17] presents a general framework (referred to as nonparametric) to analyze and design modern Hopfield networks. The sparse network is just a special case of their framework. Additionally, [17] has already introduced the construction of random feature HMNs. While this construction lacks detailed analysis, which is a limitation they acknowledged, it would be more accurate to say that the submitted draft serves as a strong companion to [17] with rigorous theoretical and empirical results. I understand that the authors are not obliged to treat arXiv preprints (unpublished papers) too seriously, but it is better to give accurate credit to existing works when possible. To clarify, I am very happy to see this draft improves the results/proposal of [17] with detailed analysis, including multiple retrieval updates.\n\n* I did not check the proofs line by line. However, I had a quick skim through them and found some typos and areas for improvement. Please do another round of proofreading if possible.\n\n\n---\n\n### Update 2024/08/07: raise score from 4 -> 5 -> 6 -> 7\n\nQUESTIONS:\n* ```line109```: why you call the number of updates as *layers*? Are you recurrently passing the output as input to the same network or the network really has many layers? For example, recurrent layer is RNN sense is the later. \n\n\n* ```Proposition 1,2,3```: what is peak memory? why we should care about it?\n\n* ```Time Complexity```: I appreciate the analysis of the computational complexity of the proposed method. From my view, this proposal is a special case of [Hu2024] with $L=1$ (where $L$ is the sequence length in their analysis). [Hu2024] provides a characterization of all sub-quadratic (efficient) variants of the modern Hopfield model. Essentially, your proposal is an efficient approximation to the softmax DenseAMs. Hence, the relevance holds. Can you discuss how your proposal fits into the results of [Hu2024]?\n  \n  * [Hu2024] Hu, J.Y.C., Lin, T., Song, Z. and Liu, H., 2024. On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis. arXiv preprint arXiv:2402.04520.\n\n\n* ```line255, Exp Observation 2```: This observation is a bit counterintuitive. Normally, large $\\beta$ means low-temperature region and leads to more stable/accurate retrieval, e.g., infinity $\\beta$ leads to argmax-retrieval. Why random DrDAM shows otherwise?  \n\n\n\n* ```Fig3```: Intuitively, why larger $\\beta$ needs larger $Y$?\n\n* ```line303```: What's the point of approximating energies and update dynamics? Shouldn't the significance lie in the approximation error of retrieval?\n\n* I am wondering if you have explored the learning aspect of the proposed method. It is known that many DenseAMs are connected to transformer attention and can be used as a learning model or layer. Can you explore this part a bit as in [17]? If not, please explain why.\n\n* Can you kindly remind me which part of this work supports the claim \"making it possible to introduce new memories without increasing the number of weights\"?\n\nLIMITATIONS:\nThere is a limitation section but no broader impact or impact statement expect in the checklist.\n\n\nAppendix A of the draft acknowledges that the experimental explorations are limited. However, it would benefit from discussing comparisons with baselines and including some vital ablations. This would help clarify the contribution and soundness of the proposed method. See above for other limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper offers a kernel-based approximation of Dense Associative Memory that allows for Hebbian encoding in a space of randomized basis functions. The main advantage when compared with the exact approach is that information concerning all patterns is shared in a single weight tensor, without requiring additional weights for each new pattern.\n\nThe authors provide some complexity results and bounds on the Euclidean deviation from the exact retrieved pattern in the case of exponential Dense Associative Networks (i.e. modern Hopfield networks). In a series of numerical experiments, the author show that the approximation breaks down for low values of the inverse temperature and for large number of patterns when compared with the number of basis functions.\n\nSTRENGTHS:\n1) The paper addresses perhaps the greatest limitation of Dense Associative Memories with non-quadratic energy, namely the fact that they cannot compress the patterns into a single shared weight matrix. This has potentially great practical relevance for the implementation of these models in practical applications. Furthermore, it connects the theory of non-quadratic Dense Associative Memories with a very large body of theoretical research on Hebbian learning and classical Hopfield models.\n\n2) The proposed approximate algorithm is theoretically sound as it connects with important known results on kernel machines and feature expansions.\n\n3) The exposition is clear and it is relatively easy to follow even for non-specialists. \n\n4) The numerical analysis of the error profile of the approximation error is thorough and insightful, especially when interpreted in combination with the theoretical results.\n\nWEAKNESSES:\n1) The embedding of this paper in the existing literature is limited. Feature expansion methods have been studied for classical Hopfield and generalized models [1,2,3], and there are uncited existing papers connecting Dense Associative Memories and kernel methods [4]. It would also be useful to discuss the recent results connecting exponential Dense Associative Memories (i.e. modern Hopfield networks) with generative diffusion models [5], since as pointed out in the paper the connection provides an alternative way to learn the same energy function in a fixed synaptic structure. It could also be useful to discuss more theoretical results involving random features, for example [6] and [7].\nIt would be good to have this treatment organized in a proper Related Work section. \n\n2) While the approximation is based on a sound idea, its derivation is very informal. It would be very useful to derive the formula using variational techniques or other known systematic methods for deriving approximations. It could also be interesting to connect it with similar approximations in Gaussian Processes, since kernel feature expansions are commonly used for GPs.\n\n3) I am not convinced about the usefulness of the bound in Eq.12 in non-trivial regimes due to the exponential dependence on the energy. \n\n4) Given the simplicity of the proposed method and its similarities with known approaches such as [TODO], I am not convicted that the contribution is relevant enough for this conference.\n\nReferences:\n[1] Liwanag, Arnold, and Suzanna Becker. \"Improving associative memory capacity: one-shot learning in multilayer Hopfield networks.\" Proceedings of the 19th Annual Conference of the Cognitive Science Society. Vol. 442. 1997.\n[2] Barra, Adriano, Matteo Beccaria, and Alberto Fachechi. \"A new mechanical approach to handle generalized Hopfield neural networks.\" Neural Networks 106 (2018): 205-222.\n[3] Yilmaz, Ozgur. \"Machine Learning Using Cellular Automata Based Feature Expansion and Reservoir Computing.\" Journal of Cellular Automata 10 (2015).\n[4] Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models\n[5] Ambrogioni, Luca. \"In search of dispersed memories: Generative diffusion models are associative memory networks.\" arXiv preprint arXiv:2309.17290 (2023).\n[6] Negri, Matteo, et al. \"Random Feature Hopfield Networks generalize retrieval to previously unseen examples.\" Associative Memory {\\&} Hopfield Networks in 2023. 2023.\n[7] Negri, Matteo, et al. \"Storage and learning phase transitions in the random-features hopfield model.\" Physical Review Letters 131.25 (2023): 257301.\n\nQUESTIONS:\n1) I find intuitively strange that the bound in Eq.13 depends on the absolute value of the energy instead of on a relative energy. Can you provide some intuition for this result?\n\nLIMITATIONS:\nThe limitations of the proposed approximate method are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

comments: SUMMARY:
This paper studies a method to modify associative memory network weights when introducing new memories. The proposed uses random features and is shown to approximate the energy function of the conventional ones.

STRENGTHS:
1. The proposed method and results are novel to me, understanding associative memories (AMs) through random feature seems to be a great connection between AMs and random feature transformers.
2. The paper is well written and easy to follow.

WEAKNESSES:
1. Why using hamming distance when calculating retrieval error when the theoretical results use L2?
2. To my understanding, the proposed method is an approximation to DAM. Thus the paper should at least compare the proposed method to DAM in thr experimental section.
3. The field of associative memories, Hopfield networks are getting increased attention recently, I think a related work section would benefit this paper a lot.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
This paper addresses its limitations in the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to interpret the energy function of Dense Associative Memory as relying on a kernel that can be approximated by kernel-specific random feature maps. \nThe approximation allows to condensate the stored patterns into a tensor whose size is independent of the number of patterns stored, similarly to what can be done in classical and polynomial Hopfield networks. \nNew patterns can then be incorporated without changing the dimensionality of the parameter space. \nThey provide bounds for the approximation error of the iterates in the gradient descent procedure that uses their energy approximation.\n\nSTRENGTHS:\n- I appreciated the idea of trying to use the kernel approximation in order to overcome the computational limits of DAMs. \n- The paper is well written and the idea is clearly explained.\n\nWEAKNESSES:\n- The main limitation of the work in my opinion is that one has to go to very  large values of the feature size Y in order to have a good approximation.  In fact, looking at the error bound in eq. (12), it seems that Y has to be taken of the order of O(K^2*D) in order to have a good approximation, which is much larger than the size of the memory matrix (K*D). This not taking into account the exp(beta*E) term in the bound, which could penalize high-energy configurations a lot.\n\n- Numerical Experiments are limited and not fully convincing. For instance, in Fig. 1 the 4x12288=49152 memory matrix has to be replaced by a several times larger Y-sized vector in order to obtain a good approximation.\n\nQUESTIONS:\n- Can the author show a plot similar to Fig. 2 but now showing the theoretichal prediction from eq. 12 (say for L=1 and alpha=1) compared to the actual error?\nProbably this information could just be plotted in Fig. 2B. \n\n- It would be nice to make a plot similar to Fig. 2 but now as a function of K.\n\n- In order to help the reader and improve consistency, p in Algorithm 1 and at the beginning of Section 3 could denote the same thing.\n\n- Assuming that the random feature version is considered as DAM per se instead of an approximation to another DAM, could the author comment on its capacity in presence of random memories? It is linear in Y?\n\n- line 215. Usually, the EDP model also contains an x^2 term outside the logarithm.\n\nLIMITATIONS:\nThe authors could discuss more the limitations of their work. In particular, the fact that the approximation requires a very large feature size Y in order to be accurate. This could limit the practical use of the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}