Relating Hopfield Networks to Episodic Control
Authors: Hugo Chateau-Laurent
Frederic Alexandre
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Episodic Control
Episodic Control
Hopfield Network
Associative Memory
Reinforcement Learning
Episodic Memory
Primary area: deep_learning_architectures
TL;DR: The Differentiable Neural Dictionary of Neural Episodic Control is a Hopfield Network

Abstract: Neural Episodic Control is a powerful reinforcement learning framework that employs a differentiable dictionary to store non-parametric memories. It was inspired by episodic memory on the functional level, but lacks a direct theoretical connection to the associative memory models generally used to implement such a memory. We first show that the dictionary is an instance of the recently proposed Universal Hopfield Network framework. We then introduce a continuous approximation of the dictionary readout operation in order to derive two energy functions that are Lyapunov functions of the dynamics. Finally, we empirically show that the dictionary outperforms the Max separation function, which had previously been argued to be optimal, and that performance can further be improved by replacing the Euclidean distance kernel by a Manhattan distance kernel. These results are enabled by the generalization capabilities of the dictionary, so a novel criterion is introduced to disentangle memorization from generalization when evaluating associative memory models.

Relating Hopfield Networks to Episodic Control
Hugo Chateau-Laurent O
Inria centre of the University Bordeaux, France IMN, CNRS UMR 5293, France
LaBRI, CNRS UMR 5800, France
hugo. chateaulaarentt@maii...
Frédéric Alexandre
Inria centre of the University of Bordeaux, France
IMN, CNRS UMR 5293, France
LaBRI, CNRS UMR 5800, France
frederic. alexandre@inria.ff
Abstract
Neural Episodic Control is powerful reinforcement learning framework that em- ploys a differentiable dictionary to store non-parane non-parametric memories. It was inspired by episodic memory the functional level, but lacks direct theoretical connec- tion to the associative memory models generally used to implement such a memory. We first show that the dictionary is an instance of the recently proposed Universal Hopfield Network framework. We then introduce a continuous approximation of the dictionary readout operation in order to derive energy functions that are Lyapunov functions of the dynamics. Finally, we empirically show that the dictionary outperforms the Max separation function, which had previously been argued to be optimal, and that performance can further be improved by replacing the Euclidean distance kernel by Manhattan distance kernel. These results are enabled by the generalization capabilities of the dictionary, so a novel criterion is introduced disentangle memorization from generalization when evaluating associative memory models.
Introduction
Episodic memory is the ability to remember information about specific situation. An influential model of episodic memory is the Hopfield Network (I) a recurrent associative memory that can learn pattern in one shot and recall it, given some partial or noisy cue. Some important limitations have been addressed with the development of differentiable continuous Hopfield Networks (2) and their connection to deep learning (3 4), thus providing renewed interest the field of associative memory. Episodic memory has also been studied as an efficient way to control reinforcement learning in so-called episodic control, particularly in the initial steps of learning ( but explicit link has been made between Hopfield Networks and control algorithms. Such a link could lead to the development of more efficient controllers and memory models. It could also shed light how the hippocampus, the seat of episodic memory the brain, contributes to behavior.
In this paper, novel connection is established between the fields of associative memory and shown in Section 2 that the differentiable neural dictionary (DND) introduced in the context of episodic control (6) rapid way to store and retrieve experiences, is mathematically close to the Hopfield Network. Retrieval from a DND can indeed be decomposed into
reinforcement learning.
'Currently postdoc CerCo, CNRS UMR 5549, Université de Toulouse, France.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Projection
2
V sep(sim K,q)
K
V
Separation
K *(sim, (K, 9))
sep, (K 9)
Similarity
Σ, 1o (K 9))
sim, ( 8+ K; q11
Figure 1: Differentiable Neural Dictionary lookup as retrieval from a Universal Hopfield Network.
similarity, separation and projection operations, just as any instance of the general Universal Hopfield Network framework (UHN) (7) recently proposed to encompass both classical and recent models of associative memory. DND, similarity scores are computed using the Euclidean distance and separated with k-nearest neighbor algorithm. The projection operation of DND is the same for all existing UHN instances, including the traditional Hopfield Network. DND can thus be thought of single-shot associative memory model, just like Hopfield Networks and their modern continuous variants. These models can often be defined by an energy function which decreases as memories are retrieved. We show that energy functions also be derived for the recall operation of the DND).
In Section 3 experiments are conducted to compare the DND to state of the art associative memory models in memorization tasks. It is shown that the DND outperforms concurrent models in gen- eralization tasks. Its performance is further improved by replacing the Euclidean distance by the Manhattan distance, predicted by the original UHN study (7). A new criterion is introduced to sess performance, and it found that the k-nearest neighbor separation of DND favors generalization over memorization, compared the simpler Max separation function.
2 Differentiable Neural Dictionary Hopfield Network
The UHN framework encompasses a family of associative memory models in which retrieval is performed by computing the similarity between a query and keys K (sim function), separating the similarity scores with function sep, then projecting the results to the output space with some value matrix V:
V sep(sim(K K, q))-
(1)
In the original binary Hopfield Network (I) the similarity function is the dot product and sep is the identity function. Modern continuous variants have been proposed (8) that improve storage capacity by using more elaborate separation functions such as Softmax (4) to push apart memory attractors.
On the other hand, Neural Episodic Control is reinforcement learning architecture that introduces the DND way to store associations between sensory observations and Q-value estimates. The reading operation of the DND is:
k
S wi,
(2)
where contains the normalized inverse distances between the query and the k nearest observation keys, and & contains the values of nearest observations (i.c. their Q-value estimates). The inverse distances are computed using the following kernel function:
sim, K, q) 5+ K q
(3)

---

with K the observation keys, q the query, and 5 = 10-3. The k-nearest neighbor function can be written as:
sep(æ:) = k(x)/ Σ k2(
(4)
if a; among the top k values of æ,
(5)
0
otherwise.
We can thus rewrite Equation 2 as:
sep(sim( K, g)),
(6)
which closely resembles the reading operation of UHN (Equation D
Note that the Euclidean similarity function of UHN 7 is the same as the kernel function of Neural Episodic Control (6)
sim, (
(7)
8 E, Kij - 9;)²
1
(8)
8 + V E) (Ki 9;) 9,)
(9)
8 + | -
In sum, the kernel function of DND acts as a similarity function and is equivalent to the Euclidean similarity function of UHN. Furthermore, the k-nearest neighbor algorithm sparsifies the result of the similarity function by cancelling the contribution of the most distant experiences. The output of the algorithm is then normalized before being projected to the value space. This constitutes a novel separation function for the UHN framework, we call it M-Max. It is worth noting that this separation function is similar to applying a threshold on the similarity function, like what is done the sparse distributed memory model (9), which has also been cast as UHN (7). The only difference is that the threshold for sparse distributed memory is fixed, while it must be dynamic for selecting a constant number i of neighbors. Furthermore, with k 1, the separation function is equivalent the Max separation function of UHN. The remaining difference between DND and other UHN instances is that the output of the DND scalar value, while UHN models can store vector values. In a modification of DND (10), multidimensional values have been stored. In fact, the DND can simply be extended with matrix of value vectors V. Equation 6 thus becomes:
V sep(sim( K, q)).
(10)
An abstract energy function has been derived for UHN models (7). It defined as:
E(K, Σ -f sep Σ sim K s. v;)
(11)
with v the activity of value neurons which are initialized with the query q. and updated to produce the output pattern z (in the case of autoassociation where V = K
The energy function requires the gradient of the separation function to be nonzero, which is not the case for & (as defined in Equation 5). A few adjustments thus need to be made to k in order to derive the energy function for the DND. Let (2x) denote the Sigmoid function, defined
(x)
(12)
Tte
An adjusted Sigmoid function with threshold è and steepness parameter 3 > 0, is used to define continuous approximation of k:
a
Kig ( = (3,03(1;; - 0))
(13)
3; Σ sim K s.j. v.).
(14)
3

---

A second Sigmoid function is used to count the number of selected dimensions (R/ entries higher than 1/2) and adjust è to ensure there are only k of them:
- = + k
(B
1 E5 2)
(15)
where < a < governs the threshold dynamics and Bg controls the steepness of the second 16
Sigmoid. In Appendix A we show that θ is convergent with a < 3. Equation 13 has a nonzero gradient, and as B and By grow to infinity, it approaches Equation | Hence, it can be used for the separation function of Equation I1 providing DND with an energy function. A second energy function is the update of O:
E = ((0) - k Σ Bk (O()(())))) 0)) - -1)]
(16)
3 Associative memory performance of the Differentiable Neural Dictionary
In the previous section, the differentiable neural dictionary of Neural Episodic Control has been shown to be a Universal Hopfield Network. In principle, DND can thus be used as associative memory. In this section, the MNIST, CIFARIO 10 and Tiny ImageNet datasets are used to test the robustness and capacity of DND as associative memory model, using the same methods as for the other UHN instances (7) unless otherwise mentioned
Example reconstructions of images by DND are shown in Figure 2 as well Figures 8 and 7 Memorics are separated by keeping the k-nearest neighbors only (k-Max separation function). For simplicity, the kd-tree of the original Neural Episodic Control implementation is not used, nor is the continuous k-Max version that makes use of Equation 13 Instead, all similarity scores are computed and those not selected are zeroed out (Equation 5) In Figure 2 using h = 50 like in the original Neural Episodic Control publication (6) gives an output from which the original image can be recognized, but the model unable to properly separate memories and the resulting output is blurry. The Max separation function is equivalent to selecting the nearest neighbor (A: = 1) and provides much clearer output. In fact, Max is the best separation function benchmarked for UHN (7). Even when the output is blurry, the k > 1 models seem to properly capture the statistics of the dataset, such as the fact that central pixels are globally more active than those on the borders. Recall accuracy is typically assessed in absolute terms, by checking that the difference between output and response below threshold (7). It is worth exploring whether the statistical modeling capacities of k > can consistently improve performance with this criterion, despite the Max function having theoretically unbounded capacity with respect to the dimensionality of the query 77). On the other hand, a new criterion will be introduced to evaluate recall in relation to other stored patterns, given that the main function of episodic memory is to reconstruct information corresponding to a particular situation (memorization) rather than to generalize.
Throughout the paper, we distinguish between two key aspects of associative memory models, adhering to the terms of (7): capacity and retrieval. Capacity refers to the number of unique images (or memories) that can be stored while maintaining accurate recall. Retrieval, on the other hand, focuses on the model's performance in recalling these stored images when they are presented with incomplete or noisy cues. measures the resilience of memory recall in the presence of distortion.
3.1 Capacity with different functions
In DND, the similarity between memories and the query is computed using Euclidean function (Equation 3). While rarely used in associative memory models, this function was found to outperform the more common dot product 77.. An even better performing similarity function was the inverse of
the Manhattan distance:
1
abs(K sim; (K.q) () 8 abs((k 9;)
(17)
"The code is available http:://hww..oooooooooo//////oo/ and is based on /ettp::/githin http:///w...ooeeeeee/eeeee/ee//r MMmory (MIT license).

---

Noise Variance
0.1
0.3
0.5
0.8
1.5
6
Cue
6
6
6
6
Max
5-Max
50-Max
Figure 2: Example reconstructions of noisy MNIST digits by DND. The top row shows the input cue with increasing amount of noise. The following rows show the reconstruction of the stored memory using k = 1, 5, 50.
The capacity of the model under different similarity (Euclidean and Manhattan) and separation functions is assessed by quantifying correctly retrieved data when increasing number of MNIST, CIFARIO and Tiny ImageNet images stored (Figures and [9: Table D Half-masked images given as input, and trial is correct if the sum of squared pixel differences between the output and the actual image is less than threshold of 50. The Manhattan similarity function outperforms the Euclidean function, especially when k is low. Furthermore, the best & value is highly dependent on the dataset. In MNIST, the best & is for both Euclidean and Manhattan functions. In CIFAR10, the best is 2 with Euclidean similarity and with the Manhattan function. In Tiny ImageNet, Max (k: 1) outperforms other functions.
frection Currectly Ratrivved
0.8
0.8
0.6
Correctly
0.6
Max
Max
0.4
2-Max
2-Max
5-Max
5-Max
0.2
10-Max
0.2
10-Max
Idantity
Identity
200
400
600
800
1000
200
400
600
800
1000
Images Stored
Images
(a) Euclidean similarity
(b) Manhattan similarity
Figure 3: Capacity of associative memory with different similarity and separation functions assessed with MNIST. Plots represent the means and standard deviations of 10 simulations. trial is correct when the difference between the output and the actual memory is under a threshold.
3.2 Retrieval with different functions
In order to test robustness of the memory, the ability to recall memories from noisy cues is analyzed. Independent pero-mean Gaussian noise with variance thus added to the query images pixelwise. Performance is evaluated using sets of 100 images.
5

---

Table Capacity of associative memory with different similarity and separation functions assessed with MNIST. CIFARIO and Tiny ImageNet datasets. Reported are means and standard deviations of the simulations of Figure For each dataset and similarity function, the best performance highlighted in bold.
Separator
MNIST
CIFARIO
Tiny
Euclidean Similarity
Max
0.739 = 0.14
0.220 + 0.18
0.223 0.21
2-Max
0.826 0.11
0.236 + 0.18
0.015 0.02
5-Max
0.851 0.08
0.117 - 0.09
0.010
10-Max
0.838 0.08
0.095 0.08
0.010
50-Max
0.801 0.09
0.087 0.09
0.010
Identity
0.809 0.08
0.088 0.09
0.02
Manhattan Similarity
Max
0.835 = 0.10
0.451 + 0.21
0.669
2-Max
0.886 0.08
0.369 + 0.20
0.011
5-Max
0.887 0.07
0.106 0.08
0.010
10-Max
0.826 0.08
0.075
0.00
50-Max
0.775 0.11
0.067 0.06
0.010
Identity
0.804 0.09
0.063 + 0.06
0.010
Like capacity, the best i for retrieval depends the dataset (Figure 10 Table 2). Here again, the performance is better with the Manhattan similarity than with the Euclidean similarity for low k. and worse for high k. In MNIST, the best ls is 50 with both Euclidean and Manhattan similarities. In CIFARIO, 10, is the best k. Like for capacity, Max outperforms other functions with Tiny ImageNet images.
1.0
Max
Frecton Correctly Retrieved
Max
0.0
5-Max
Max
50-Max
0.6
Fraction Correctly
0.6
Identity
Max
2.Max
S-Max
0.2
10-Max
50-Max
Identily
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Noise variance (sigma)
Noise variance (sigma)
(a) Absolute criterion
(b) Memorization criterion
Figure 4: Retrieval capability against increasing levels of noise. Plots represent the means and standard deviations of 10 simulations with different sets of MNIST images.
As hypothesized, Max is not always the best performing function with the absolute accuracy criterion, both for capacity and retrieval tasks. It can indeed outperformed by higher k values, meaning that taking into account more memories than the single most similar one can lead to more precise recall absolute terms (i.e. as assessed by a threshold). The performance of the identity and 50-Max functions in the MNIST dataset are surprisingly good, even under very high levels of noise (Figure 4ah In fact, as pixel values are restricted to lie in the range [0, 1], very unlikely that enough information remains in the image to correctly identify when > 1. Hence, a plausible explanation that high & functions model the dataset such that they output mixture of many images that is sometimes classified as correct retrieval, although is not necessarily closer the query image than

---

Table 2: Retrieval capability against noise. Reported are means and standard deviations of the 10 simulations of Figure 10 For each dataset and similarity function, the best performance is highlighted
bold.
Separator
MNIST
CIFARIO
Tiny
Euclidean Similarity
Max
0,667 = 0.44
0,574 + 0.45
0.580 0.44
2-Max
0.692 0.41
0.588 + 0.44
0.310 0.43
5-Max
0.735 0.37
0,455 + 0.41
0.190
10-Max
0.789 0.31
0.357 0.39
0.128 + 0.33
50-Max
0.904 + 0.09
0.205 + 0.30
0.002 0.01
Identity
0.830 0.10
0.083 0.09
0.000
Manhattan
Similarity
Max
0,672 = 0.43
0.627 = 0.44
0.44
2-Max
0.699 0.40
0.628 + 0.43
0.223
0.39
5-Max
0.741 = 0.36
0.424 + 0.38
0.009 0.02
10-Max
0.794 0.30
0.282 0.28
0.002+0.:0
50-Max
0.891 + 0.07
0.085 + 0.06
0.000
Identity
0.781 0.05
0.049 + 0.02
0.000
any other of the dataset. This especially true for MNIST which contains simple pictures that are more similar to each other than CIFARIO and Tiny ImageNet.
3.3 Performance with memorization criterion
In order prevent associative memory models from modeling the statistics of the dataset rather than focusing on the query image to output the actual memory, a novel criterion is introduced. Instead of the absolute threshold, retrieval must be good relatively to other images. More precisely, the novel criterion is such that a trial is correct if and only if the sum of squared pixel differences between the truth and the output lower or equal to the sum of squared pixel differences between the output and any other memory, that is, if:
E(s - Kzy)² min [ (;))
(18)
where K, is the correct pattern to retrieve.
Capacity is now assessed using this new criterion (Figures 5 and 11 Table 3). The Manhattan function still outperforms the Euclidean similarity. Most crucially, the best performance always obtained with the Max function.
Retrieval is then tested with the new criterion (Figures 4b and 12 Table 4. Once again, the best performance is obtained with the Manhattan similarity. Furthermore, k = 1 almost always
outperforms other values. Note that performance with is very similar.
3.4 Relationship between k-Max and Softmax
Like k-Max, the Softmax function virtually cancels out the contribution of distant memories, espe- cially when 3, the scaling parameter of its input, is high. It does it by normalizing exponentiated similarity scores:
Bx
Softmax (x)
(19)
e
While is discrete parameter, ,3 is continuous, which makes the Softmax function harder to optimize but perhaps more flexible. Here, the two separation functions are compared. For each dataset, 100 images are encoded. The noise is set to for MNIST and 0.75 for CIFAR10 dataset and Tiny ImageNet. The results are shown in Figures 6a and 13 with the absolute criterion and in Figures 6b

---

frection Currectly Retrieved
2-Max
Fraction Correctly Retrieved
2-Max
0.8
5-Max
0.8
S-Max
10-Max
10-Max
0.6
50-Max
0.6
50-Max
Identity
Identity
0.4
0.2
0.2
0.0
0
200
400
600
800
1000
200
400
600
800
1000
Images Storad
Images Stored
(a) Euclidean similarity
(b) Manhattan similarity
Figure 5: Capacity of associative memory with different similarity and separation functions assessed with MNIST. Plots represent the means and standard deviations of 10 simulations. Here, trial is correct when the difference between the output and the actual memory is lower than the difference between the output and any other stored memory.
Table 3: Capacity of associative memory with different similarity and separation functions assessed with MNIST, CIFARIO and Tiny ImageNet datasets. Reported are means and standard deviations of the 10 simulations of Figure For each dataset and similarity function, the best performance is highlighted in bold.
Separator
MNIST
CIFARIO
Tiny
Euclidean Similarity
Max
0.624 0.22
0.223 + 0.19
0.223 + 0.21
2-Max
0.624 0.22
0.222
0.19
0.222 + 0.21
5-Max
0.256 0.24
0.132 0.13
0.112
10-Max
0.199 0.24
0.104 0.14
0.098 0.13
50-Max
0.171 = 0.25
0.086 0.14
0.095
Identity
0.169 0.25
0.085 + 0.14
0.114
Manhattan
Similarity
Max
0.793 + 0.14
0.502 + 0.24
0.669 0.24
2-Max
0.790 0.14
0.486 + 0.25
0.505 0.36
5-Max
0.255 0.22
0.176
0.155
10-Max
0.187 0.22
0.103 0.15
0.119
50-Max
0.163 = 0.23
0.090
0.113
Identity
0.162 0.23
0.088 + 0.15
0.113
and 14 with the memorization criterion. Except for the condition with CIFARIO, Euclidean similarity and absolute criterion (Figure (3b), the Softmax can always outperform k-Max.
Discussion
In this paper, DND, which has initially been introduced in the context of reinforcement learning (6), has been shown to be mathematically related Hopfield Networks (I). The Universal Hopfield Network framework has recently been introduced to encompass the traditional Hopfield Network, modern variants and related models (7). These models recall memories with common sequence of operations: similarity, separation and projection. It has been shown that retrieval from DND is also done with these operations. Hence, DND is an instance of the Universal Hopfield Network framework. For the sake of mathematical analysis, a continuous approximation of DND recall has

---

Table 4: Retrieval capability against noise. Reported are means and standard deviations of the 10 simulations of Figure 12
For each dataset and similarity function, the best performance is highlighted
bold.
Separator
MNIST
CIFARIO
Tiny
Euclidean Similarity
Max
0.661 + 0.44
0.574 + 0.45
0.580 + 0.44
2-Max
0.661 0.44
0.575 + 0.44
0.579 + 0.44
5-Max
0.576 0.41
0.366 + 0.42
0.416
10-Max
0.522 0.39
0.288 0.40
0.314 +.41
50-Max
0.246 0.34
0.134 + 0.29
0.160 0.32
Identity
0.150 0.32
0.029 0.05
0.101 + 0.24
Manhattan
Similarity
Max
0.665 0.44
0.621 + 0.44
0.622
2-Max
0.664 0.43
0.621 + 0.44
0.44
5-Max
0.523 = 0.36
0.316 + 0.37
0.352
10-Max
0.422 0.31
0.177 0.25
0.228
+ 0.33
50-Max
0.063 0.04
0.026 0.02
0.030 0.03
Identity
0.022 0.02
0.012 + 0.00
0.012 0.01
8
50000
40000
30000
20000
10000
50000
40000
30000
20000
10000
Fraction Correctly Retrieved
Softmax
0.8
0.12
0.10
0.6
Softmax
0.04
0.2
0.02
(a) Absolute criterion
(b) Memorization criterion
Figure 6: Retrieval capability as function of and 3 parameters the k-Max and Softmax separation functions respectively. In each simulation, 100 MNIST images encoded, then queried with noise of Plots represent the means and standard deviations of 10 simulations with different sets of images.
been proposed to comply with the requirements of the energy function of UHN and derive second Lyapunov function of the dynamics.
This novel link connects the fields of associative memory and reinforcement learning. The similarity function of DND is Euclidean and has already been shown to yield high capacity 7 On the other hand, the k-nearest neighbor is not commonly used separation function for associative memory. One thing to note is that the time complexity of k-nearest neighbor search is (logn) when implemented with k-d trees (TT. In contrast, the Softmax function has a time complexity of O(n). Thus, one of the present objectives was to assess the performance of the more efficient k-Max separation function.
Interestingly, controls degree of separation, and setting function studied by (7). While having theoretically unbounded capacity, the Max function can transition sharply from one memory attractor another when noise of increasing amplitude is added to the query. With Figure 2 it was hypothesized that higher values could be better at modelling
is equivalent to using the Max

---

datasets and improve the performance assessed in absolute terms. Simulations indeed revealed higher capacity and better retrieval from noisy queries with & I, especially with simple datasets like MNIST. However, these results depend on the way performance is evaluated. Traditionally, the evaluation of retrieval is based on some distance evaluation of the memory output and the actual image, which must not exceed some threshold fixed by the experimenter. This is widespread method for evaluating associative memory models, but one must choose the threshold wisely, as setting too high can result in false positíves with the model grossly reproducing statisties of the dataset (generalization). This for example seems to be the case when retrieval is assessed using the MNIST dataset. The performance of 50-Max (and even the identity function) remains high despite very strong noise. Therefore, another way of evaluating retrieval was introduced, which does not consider the output in absolute terms, but rather compares it to the whole memory set. Retrieval is deemed correct if and only if the output resembles the actual image more than any other stored memory. Memory models thus cannot benefit from modeling statistics of the dataset, and must rather focus on recalling the distinguishing characteristics of the query (memorization). Using this method, the Max function outperforms the others. Ideally, performance should be evaluated in both absolute and relative terms to ensure that recall is accurate and stands out from other memories.
This raises the question of what is the function of associative memory. Modeling statistics of dataset related to generalization, which is typically the main goal of machine learning. The objective of associative memory is somewhat different. Instead of generalizing, to recall the exact information corresponding to the individual memory. This is reminiscent of the division of labor between episodic and semantic memory (2)). When it comes to episodic control however, that is the use of episodic memories for action control, some generalization is desirable. This is especially the case in Neural Episodic Control in which the selection of actions only relies on episodes, the DND thus constituting a bottleneck. Initially, episodic control (not to be confused with its implementation in Neural Episodic Control) has been introduced as way of speeding up the learning of reinforcement agent and, after the initial episodic control phase, it is desirable that more robust controllers can take over 3 A biologically inspired alternative Neural Episodic Control would be to supplement episodic memory with other controllers whose function is to generalize. The episodic memory would then no longer be bottleneck. and could instead be devoted to memorizing the specifics of situations. That being said, there is also an ongoing debate about the fact that episodic memory could also integrate a part of generalization and not only store the specificities of episodes
associative memory aims
13 14 15
5
Limitations and Future Work
In this paper, we mainly focused on evaluating the capacity and retrieval performance of associative memory models. Conversely the application of DND associative memory, the novel theoretical link also implies that any instance of UHN can be used for episodic control. It is possible that the Manhattan function could consistently improve sample efficiency, outperforming the Euclidean kernel of DND, as it does on the associative memory tasks. The Softmax function, which has been proven powerful in transformers (16), and more performant than k-Max in the present study, could also improve episodic control agents. RL experiments are being conducted in this regard.
Finally, the fact that DND is theoretically related to Hopfield Networks provides biological basis to Neural Episodic Control, as the most influential model of the hippocampus relies heavily on similar associative memory mechanisms (17). Hence, this study opens up new avenues of research at the frontier of the fields of associative memory, reinforcement learning and neuroscience.
Acknowledgements
Experiments presented in this paper were carried out using the PlaFRIM experimental testbed, supported by Inria, CNRS (LABRI and IMB), Université de Bordeaux, Bordeaux INP and Conseil Régional d' A'uuitaieee (see https:: //www/ www.wppafrrimmm fr).
Both authors were funded by Inria. No additional sources of funding were received in support of this work.
The authors declare that they have no competing financial interests or relationships with entities that could be perceived to influence the work presented in this paper.
10

---

The authors would like to thank Thierry Viéville for his help deriving the energy functions, as well as Dolton Fernandes, who discovered the Universal Hopfield Network paper.
References
[1] J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities.," Proceedings of the national academy of sciences, vol. 79, no. 8, pp. 2554-2550 1982.
[2] Hopfield, "Neurons with graded response have collective computational properties like those of two-state neurons.." Proceedings of the national academy of sciences, vol. 81, no. 10, pp. 3088-3092, 1984.
[3] D. Krotov and J. Hopfield, "Dense associative memory for pattern recognition," " Advances in neural information processing systems, vol. 29, 2016.
[4] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleit- ner, M. Pavlović, G. K. Sandve, et al., "Hopfield networks is all you need," arXiv preprint arXiv:2008.00 02211, 2020.
[5] M. Lengyel and P. Dayan, "Hippocampal contributions control: the third way," Advances in neural information processing systems, vol. 20, 2007.
[6] A. Pritzel, B. Uria, S. Srinivasan, A. P. Badia, O. Vinyals, D. Hassabis, D. Wierstra, and
C. Blundell, "Neural episodic control," in International conference on machine learning.
pp. 2827-2836, PMLR, 2017.
[7] B. Millidge, T. Salvatori, Y. Song, T. Lukasiewicz, and R. Bogacz, "Universal hopfield networks: general framework for single-shot associative memory models," " in International Conference on Machine Learning, pp. 15561-15583, PMLR, 2022.
[8] D. Krotov, "A new frontier for hopfield networks." Nature Reviews Physics, pp. 1-2, 2023.
[9] Kanerva, Sparse distributed memory. MIT press, 1988.
[10] M. Fortunato, Tan, R. Faulkner, S. Hansen, Puigdomènech Badia, G. Buttimore, C. Deck, J. Z. Leibo, and C. Blundell, "Generalization of reinforcement learners with working and episodic memory," Advances in neural information processing systems, vol. 32, 2019.
[11] J. L. Bentley, "Multidimensional binary search trees used for associative searching," " Communi- cations of the ACM, vol. 18, no. 9, pp. 509-517. 1975.
[12] L. McClelland, B. L. McNaughton, and C. O'Reilly, "Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.." " Psychological review, vol. 102, no. 3, p. 419, 1995.
[13] K. L. Stachenfeld, M. M. Botvinick, and S. J. Gershman, "The hippocampus as predictive map." Nature neuroscience, vol. 20, no. 11, pp. 1643 1653, 2017.
[14] J. C. Whittington, T. H. Muller, S. Mark, G. Chen, C. Barry, N. Burgess, and T. E. Behrens, "The tolman-eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation," Cell, vol. 183, no. 5, pp. 1249-1263, 2020.
[15) E. Spens and N. Burgess, "A generative model of memory construction and consolidation," biokxiv, pp. 2023-01 01, 2023.
[16] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[17] D. Marr, "Simple memory: a theory for archicortex," Philosophical Transactions of the Royal Society of London, B, Biological Sciences, vol. 262, no. 841, pp. 23-81, 1971.

---

A
Convergence of the threshold θ
From Equations
1 and
3 we obtain:
t F F(0),
(20)
BR [3(3(2 01) 2
1
F ( θ
k
(21)
A(0)
thus
Be ((((((=)))) - 2 - 3r Bko' (3(xx=-))) 3 o'
(O) = a
(22)
1=1
((0)=-A1(9)
Since from Equation 12 of the Sigmoid: 0 < d'(x) (c1)2+0-2/2) S 1/4, while by design < a,
0 < A, < BR. we obtain:
(00)= -10=1-1 with < V 16
(23)
so that if a < B then 0 < (0) < thus 0 < F (0) < so that the recurrent series defining O0 monotonic convergent This convergence is verified for all æ. that if their values vary during the convergence, the final value of & may vary, but always in converging mode.
Furthermore, A (e) decreases along the iteration and be used as energy (i.e. Lyapounov function) for the recurrence, in complement the abstract energy given in Equation [T] To avoid any interference between both converging processes, the implementation level, the continuous value of θ is calculated as fast local iteration loop, so that it then almost constant when adjusting the dynamic related to Equation 11
'For a <
KP z we still have |F (0) < 1, since -1 < F' (0) < 1, thus convergence, but the convergence
32
may be oscillatory. Since sim |F' (e) the fixed point is at the edge of stability, thus monotonic convergence preferable the numerical level.
12

---

B Example reconstructions
Noise Variance
0.3
0.5
0.8
Cue
Max
5-Max
50-Max
Figure 7: Example reconstructions of noisy CIFARIO images by DND. The top row shows the input cue with increasing amount of noise. The following rows show the reconstruction of the stored memory using k 1,5,50.
Noise Variance
0.5
0.8
Cue
Max
50-Max 5-Max
Figure 8: Example reconstructions of noisy Tiny ImageNet images by DND. The top row shows the input cue with increasing amount of noise. The following rows show the reconstruction of the stored memory using k 1,5,50 50.
13

---

C
Capacity with absolute criterion
Max
Mar
Facke Cennkr TADKNN
30-Mac
0.4
Sae
8:4-
0:2
2002
402
2.00
Irages
Staned
mage
(a) MNIST with Euclidean similar- (b) CIFARIO with Euclidean simi- (c) Tiny ImageNet with Euclidean larity
ity
similarity
Poddte Ceneely Mereed
2
2-Max
tousee Cannecta
26-
50-18
Brordiyy
Max
2-80x
0.-
02
0.2
200
203
66c
903
3002
2:30
602
6030
800
1000
2:33
410
600
3000
1000
Images Stree
Imgia Staned
mmgg: Store
(d) MNIST with Manhattan similar- ity
(e) CIFARIO with Manhattan simi- larity
Tiny ImageNet with Manhattan similarity
Figure 9: Capacity associative memory with different similarity and separation functions assessed with MNIST, CIFAR 1 and Tiny ImageNet datasets. Plots represent the means and standard deviations of 10 simulations. trial is correct when the difference between the output and the actual memory under threshold.
14

---

D Retrieval with absolute criterion
08
2 Sp-Mas
2dee Mac Identty
66-
Prader
Mix
01
30-4ac
02
0.2
04
24
08
1.0
12
14
00
04
12
34
Aolave Scom))
Make
(a) MNIST with Euclidean similar- (b) CIFARIO with Euclidean simi- ity
(c) Tiny ImageNet with Euclidean similarity
larity
1.0
20
3.3
Poeter CaReery Rectord
08-
66-
248xr
S0-Mac
Re50 AATAVCE 5.0m
St0mao
Made rigmel
(d) MNIST with Manhattan similar- ity
(c) CIFARIO with Manhattan simi- (f) Tiny ImageNet with Manhattan larity
similarity
Figure 10: Retrieval capability against increasing levels of noise. Plots represent the means and standard deviations of 10 simulations with different sets of images. A trial is correct when the difference between the output and the actual memory is under a threshold.
15

---

E
Capacity with memorization criterion
Mix
Max
Mar
P Denokr MAtWeN
8.
12-6
30-Mac
8:4
0:3
2002
Staned
mage
(a) MNIST with Euclidean similar- (b) CIFARIO with Euclidean simi- (c) Tiny ImageNet with Euclidean similarity
ity
larity
Mich
2
2-Max
06-
50 Max
26.
50-18
0.2
02
0.2
200
3002
2:30
602
6030
820
1000
2:33
410
600
3000
1000
Images Stoee
Imgia Staned
magg: Store
(d) MNIST with Manhattan similar- ity
(e) CIFARIO with Manhattan simi- larity
Tiny ImageNet with Manhattan similarity
Figure 11: Capacity of associative memory with different similarity and separation functions assessed with MNIST, CIFAR1O and Tiny ImageNet datasets. Plots represent the means and standard deviations of 10 simulations. Here, a trial is correct when the difference between the output and the actual memory is lower than the difference between the output and any other stored memory.
16

---

Retrieval with memorization criterion
MMn
M0
Sp-Mas
01
02
0.2
50
04
24
0.
100
12
14
00
04
12
34
Neoo Awlove Scom))
Malee voibac 10yro)
(a) MNIST with Euclidean similar- ity
(b) CIFARIO with Euclidean simi- larity
(c) Tiny ImageNet with Euclidean similarity
1.0
20
2.3
Pabtee Canter Rectord
2M
2M4
06-
66-
10
vatance Siom
St0ma
Made righel
(d) MNIST with Manhattan similar- (c) CIFARIO with Manhattan simi- (f) Tiny ImageNet with Manhattan ity
larity
similarity
Figure 12: Retrieval capability against increasing levels of noise. Plots represent the means and standard deviations of 10 simulations with different sets of images. Here, trial is correct when the difference between the output and the actual memory is lower than the difference between the output and any other stored memory.
17

---

G vs. k with absolute criterion
50060
42003
30300
20003
50000
43000
23030
30000
50303
43000
30203
6.08
6.00
0.04
02-
642
0.93
(a) MNIST with Euclidean similar- (b) CIFARIO with Euclidean simi- (c) Tiny ImageNet with Euclidean
ity
larity
similarity
Rrade Ceneely Ratreee
6:25
0.00
0.0
Canoeity
635
6:55
0.00
02
6.0
100
100
(d) MNIST with Manhattan similar- (e) CIFARIO with Manhattan simi- Tiny ImageNet with Manhattan ity
larity
similarity
Figure 13: Retrieval capability as function of & and 33 parameters of the k-Max and Softmax separation functions respectively. Plots represent the means and standard deviations of 10 simulations with different sets of images. trial is correct when the difference between the output and the actual memory is under threshold.
18

---

B vs. k with memorization criterion
40000
20000
22000
10303
50000
40000
20000
30000
50303
43030
30263
0:30
0.22
0.00
6.09
0.00
6.2
(a) MNIST with Euclidean similar- (b) CIFARIO with Euclidean simi- (c) Tiny ImageNet with Euclidean
ity
larity
similarity
Sel/roo
6.25
3.10
1
635
6.00
030
0.00
6.00
100
(d) MNIST with Manhattan similar- (e) CIFARIO with Manhattan simi- Tiny ImageNet with Manhattan ity
larity
similarity
Figure 14: Retrieval capability as function of and parameters of the k-Max and softmax separation functions respectively. Plots represent the means and standard deviations of 10 simulations with different sets of images. Here, trial is correct when the difference between the output and the actual memory is lower than the difference between the output and any other stored memory.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction were written order to describe the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses possible new directions that could not be explored in the time scope, such as the evaluation of similarity and separation functions in reinforcement learning tasks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: A complete proof was provided for the convergence of O.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Experiments are described in sufficient details to reproduce the results and the code is made available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code made available and can already used to reproduce the results. Additional instructions will be provided before publication to improve reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/ Staalees//ooeSSuumssiiiiiie..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//mdeeeSbmmmaia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper does specify all the experimental details necessary to understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars which are defined in the text.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiments presented in the paper are light but details about the computer resources (identity of the cluster) used omitted in the submission preserve anonymity and will be included in the acknowledgements after the review process.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:://neut http:///wwweppepppppieee/eeeeeee cc/ppubblct/thhhcccuueeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper presents theoretical work. There are many potential societal consequences of our work, none which we feel must be specifically highlighted.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper presents theoretical work and therefore poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators of the original code used in the paper properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code is made available and can already be used to reproduce the results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "81yyAYB0wj", "reviewer": "Reviewer_4Gfy", "summary": "This paper establishes a novel connection between differentiable neural dictionaries (DNDs) used in episodic control for reinforcement learning and Hopfield networks used as associative memory models. The authors show that DNDs can be formulated within the Universal Hopfield Network (UHN) framework, derive energy functions for DND recall, and conduct experiments comparing DND performance to other associative memory models on image reconstruction tasks.\n\nI believe that this paper presents a valuable theoretical contribution by connecting DNDs to the UHN framework, supported by extensive empirical evaluation. The work has the potential to impact both reinforcement learning and associative memory research. However, the paper would benefit from a bit clearer organization, a more explicit discussion of practical implications, and ideally some exploration of the impact on RL tasks. Finally, the addition of a conceptual figure or schematic would further strengthen the paper by making its key ideas more accessible and memorable. With these improvements, I believe this paper would be a strong contribution to NeurIPS.", "questions": "### I have the following questions/suggestions:\n- How do you expect the choice of similarity and separation functions to impact sample efficiency and performance in RL tasks using episodic control?\n- Have you considered how the memorization vs. generalization trade-off might be dynamically adjusted in a learning system? Could this provide benefits in certain RL scenarios?\n- How does the computational cost of DND retrieval with k-Max separation compare to other UHN instances, particularly for large memory sizes?\n- Have you considered the relation between this model and other episodic memory-like approaches, including for instance generative models for video, LLMs, and model-based RL?\n\n### Suggestion for improvement: \nI think the paper would greatly benefit from the addition of a high-level schematic or conceptual figure that visually illustrates the connection between DNDs and the UHN framework. Such a figure could perhaps\n1. Show the parallel structures and operations in DNDs and UHNs side by side.\n2. Illustrate how the similarity, separation, and projection operations map between the two frameworks.\n3. Visualize the proposed continuous approximation of the k-Max function.\n4. Demonstrate how the energy functions relate to the model's dynamics.\nA clear, well-designed figure with any (or all) of these, would significantly enhance the paper's accessibility, especially for readers less familiar with either DNDs or UHNs. It would also help to crystallize the paper's main theoretical contribution and make the work more memorable and impactful.\n\n### Minor points:\n- The discussion of biological plausibility in relation to hippocampal models adds an interesting interdisciplinary perspective, but it’s very brief. I would extend it if possible.\n- The paragraph in line 180 is quite long and difficult to follow. I know you were trying to keep the 9 page limit but in the camera ready version I would recommend you to break it into two paragraphs and rephrase a few sentences.", "limitations": "### Overview:\nThe authors have made an effort to address some limitations of their work, which is commendable. However, there is room for improvement in this area:\n1. Scope of experiments: The authors acknowledge that they did not explore the evaluation of similarity and separation functions in reinforcement learning tasks due to time constraints. This is a good start, but they could expand on why this limitation is important and how it might impact the broader applicability of their findings.\n2. Theoretical vs. practical implications: While the paper provides a strong theoretical foundation, it could benefit from a more explicit discussion of the limitations in translating these theoretical insights into practical applications, especially in reinforcement learning contexts.\n3. Scalability: The paper doesn't adequately address potential limitations in scaling the proposed methods to very large datasets or complex, high-dimensional state spaces that might be encountered in real-world reinforcement learning tasks.\n4. Computational resources: There's no discussion of the computational requirements for the various methods compared, which could be a significant limitation in certain applications.\n5. Negative societal impact: The authors do not explicitly discuss potential negative societal impacts of their work. While this research is primarily theoretical, it would be beneficial to consider and address potential misuse or unintended consequences of more efficient episodic control in AI systems, in the camera ready version of this paper.\n### Suggestions:\n1. Include a dedicated \"Limitations and Future Work\" section.\n2. Discuss how to address these limitations in future research.\n3. Consider potential societal impacts (e.g., privacy implications, risks of enhanced AI memory systems).\n4. Briefly address computational resource requirements and scalability.\nAddressing these points would demonstrate a more comprehensive understanding of the work's broader implications and potential impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Despite some organisation issues I mention below, I still find this paper relatively easy to follow and with good quality of English, plots, and articulation of ideas.\n- The theoretical contribution linking DNDs to Hopfield networks and the UHN framework is significant and well-argued. This connection opens up interesting avenues for cross-pollination between reinforcement learning and associative memory research.\n- The derivation of energy functions for DND recall, including a novel continuous approximation, is mathematically sound and adds to our theoretical understanding of these models.\n- The experimental comparisons are cover three datasets (MNIST, CIFAR10, Tiny ImageNet) and evaluating different similarity and separation functions.\n- The introduction of a new \"memorization criterion\" for evaluating associative memory performance is thoughtful and helps distinguish between memorization and generalization capabilities.", "weaknesses": "- While the experiments are comprehensive, they focus solely on image reconstruction tasks. Given the paper's motivation from reinforcement learning, it would have been valuable to include experiments or discussion on how these findings might impact episodic control in RL settings.\n- The paper is quite dense and could benefit from clearer organization. Some important findings, like the superior performance of the Manhattan distance similarity function, are buried in the results sections and could be highlighted more prominently.\n- The discussion, while interesting, feels somewhat speculative and disconnected from the main technical contributions. This section could be tightened or more clearly linked to the paper's core findings.\n- The paper lacks a clear discussion of computational complexity trade-offs between different similarity and separation functions. This would be particularly relevant for practical applications in RL or large-scale associative memory tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer 4Gfy for the detailed and constructive feedback on our submission. We appreciate the recognition of our significant theoretical contribution and extensive empirical evaluation. Below, we address each of your comments and suggestions in detail.\n\n1. Comment: \"While the experiments are comprehensive, they focus solely on image reconstruction tasks. Given the paper's motivation from reinforcement learning, it would have been valuable to include experiments or discussion on how these findings might impact episodic control in RL settings.\"\n    - Answer: We understand your concern. We used three different image datasets, including CIFAR10 and Tiny ImageNet, which are naturalistic and high-dimensional, to evaluate the performance of memory models comprehensively. Image reconstruction tasks provide a robust testbed for assessing memory models due to their complexity and interpretability. We are now working on evaluating the models on RL tasks, which will hopefully give rise to a new manuscript. We have added a paragraph in the Discussion section to outline these future directions.\n\n2. Comment: \"The paper is quite dense and could benefit from clearer organization. Some important findings, like the superior performance of the Manhattan distance similarity function, are buried in the results sections and could be highlighted more prominently.\"\n    - Answer: We have reorganized the results section to highlight key findings, including the superior performance of the Manhattan distance similarity function, and improved the overall clarity of the manuscript.\n\n3. Comment: \"The discussion, while interesting, feels somewhat speculative and disconnected from the main technical contributions. This section could be tightened or more clearly linked to the paper's core findings.\"\n    - Answer: We have revised the discussion section to more clearly link it to our core technical contributions, replacing speculative content related to neuroscience by referenced claims and focusing on practical implications and future research directions.\n\n4. Comment: \"The paper lacks a clear discussion of computational complexity trade-offs between different similarity and separation functions. This would be particularly relevant for practical applications in RL or large-scale associative memory tasks.\"\n    - Answer: We thank the reviewer for raising this very interesting point. We have now provided a comparative analysis of the computational costs for DND retrieval with k-Max separation versus other UHN instances. Neural episodic control implements k-Max using a k-d tree, whose average search complexity is O(log n). In comparison, the complexity of the softmax function as used in modern Hopfield networks is O(n). This difference in complexity can have significant implications for large memory sizes, where the k-Max approach can disregard significant portions of the memory.\n\n5. Question: \"How do you expect the choice of similarity and separation functions to impact sample efficiency and performance in RL tasks using episodic control?\"\n    - Answer: We are very much looking forward to finding out how the choice of similarity and separation functions impacts sample efficiency and performance in RL tasks. We believe that functions promoting better memory, such as the Manhattan distance, can enhance sample efficiency by enabling more effective recall of relevant past experiences. We have elaborated on this in the Discussion section and are currently working on experiments to validate these hypotheses in RL settings.\n\n6. Question: \"Have you considered how the memorization vs. generalization trade-off might be dynamically adjusted in a learning system? Could this provide benefits in certain RL scenarios?\"\n    - Answer: Yes, gradually shifting from a strategy that memorizes specific examples to one that generalizes aligns with the complementary roles of episodic and semantic memory in cognitive science. We discuss how parameters such as k and beta can be dynamically adjusted during training to balance memorization and generalization. This approach can provide significant benefits by adapting the model to different phases of learning and varying task requirements in reinforcement learning scenarios.\n\n7. Question: \"How does the computational cost of DND retrieval with k-Max separation compare to other UHN instances, particularly for large memory sizes?\"\n    - Answer: See 4.\n\n8. Question: \"Have you considered the relation between this model and other episodic memory-like approaches, including generative models for video, LLMs, and model-based RL?\"\n    - Answer: We have expanded the related work section to discuss how our model relates to other episodic memory-like approaches, such as generative models for video, large language models (LLMs), and model-based RL, highlighting unique advantages and potential synergies.\n\n9. Suggestion: \"The paper would greatly benefit from the addition of a high-level schematic or conceptual figure.\"\n    - Answer: We agree and have added a high-level figure that visually illustrates the connection between DNDs and the UHN framework, including parallel structures, operations, and mapping of similarity, separation, and projection functions.\n\n10. Comment: \"The discussion of biological plausibility in relation to hippocampal models is very brief.\"\n    - Answer: See 3.\n\n11. Comment: \"The paragraph in line 180 is long and difficult to follow.\"\n    - Answer: We have revised the paragraph, breaking it into two shorter paragraphs and rephrasing sentences for better readability.\n\n12. Suggestions: \"Include a dedicated \"Limitations and Future Work\" section.\" and \"Consider potential societal impacts\"\n    - Answer: We included a \"Limitations and Future Work\" section and briefly consider societal impacts.\n\nWe hope these revisions address your concerns and enhance the quality and impact of our paper. Thank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zfDC0agp4U", "reviewer": "Reviewer_UfDJ", "summary": "The paper introduces the differentiable neural dictionary, which uses template-based memory storage, relating it mathematically to Hopfield Networks within the UHN framework. This novel model is shown to be capable of storing memories through operations such as similarity, separation, and projection, thus demonstrating high capacity and adaptability. The model employs different separation functions like k-nearest neighbor and Max function, with the latter transitioning sharply between memory states upon noise increments. It also discusses how the dictionary outperforms traditional models in associative memory tasks by using different distance kernels (Euclidean, Manhattan), which aids in better generalization over memorization. The discussion extends to the potential applications of DND in episodic control within reinforcement learning, suggesting that DND can speed up learning by reducing the bottleneck in decision processes and integrating generalization into episodic memory. The text ends by suggesting further research into the biological basis of DND and its relationship with neural mechanisms of memory.", "questions": "Please see the \"Weaknesses\" section.", "limitations": "The article extensively discusses its limitations towards the end.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article is clearly articulated and provides detailed experimental procedures.\n2. This paper presents a novel approach to integrating associative memory with reinforcement learning. Specifically, it re-derives mathematical formulations, transforming the form of the Differentiable Neural Dictionary (DND) into the Universal Hopfield Network framework (UHN). This transformation leads to the derivation of corresponding energy functions from the UHN.\n3. The paper demonstrates the capacity of the DND model on MNIST, CIFAR-10, and Tiny ImageNet datasets using various functions. It evaluates the model’s retrieval capability against noise, performance based on the memorization criterion, and the relationship between k-Max and Softmax functions.", "weaknesses": "The article starts from the Hopfield Network framework, and my concern lies in the fairness of introducing the k-nearest neighbor for comparisons in associative memory. This introduction seems to bring in additional memory information, potentially skewing the comparisons. Furthermore, the paper suggests that evaluating retrieval should involve comparisons with the entire dataset. This approach deviates from the essential nature of memory retrieval tasks. If there are enough elements in the dataset for comparison, such as with the 50-Max criterion, selecting the most likely candidate from a set, even in the presence of significant noise, seems justifiable. However, this method may not accurately reflect the true performance of memory retrieval under more typical conditions where fewer comparison points are available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer UfDJ for the constructive feedback and positive remarks on our work. We appreciate the recognition of our novel approach and detailed experiments. Below, we address the specific concerns raised.\n\n1. Comment: \"The article starts from the Hopfield Network framework, and my concern lies in the fairness of introducing the k-nearest neighbor for comparisons in associative memory. This introduction seems to bring in additional memory information, potentially skewing the comparisons.\"\n    - Answer: It is not clear to us what additional memory information is added by introducing the k-nearest neighbor function. If the concern is about memory complexity, it is not impacted by this new separation function. We added information about the computational complexity of the different UHN instances and highlight that the search complexity of k-max is better than other separation functions when implemented with a k-d tree. If the concern is about additional hyperparameters, k is equivalent to softmax's beta, so k-max uses the same number of hyperparameters as softmax.\n\n2. Comment: \"The paper suggests that evaluating retrieval should involve comparisons with the entire dataset. This approach deviates from the essential nature of memory retrieval tasks.\"\n    - Answer: We do not mean that retrieval should always involve comparisons with the entire dataset. Our intent is to highlight that the standard criterion, which evaluates performance in absolute terms, differs from a criterion that assesses how close the retrieval is to the correct image relative to other images in the training set. This distinction is important as it influences whether the evaluation favors memorization or generalization strategies. We rephrased key sentences in the Discussion section.\n\nWe hope these clarifications address your concerns and enhance the understanding and impact of our work. Thank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTHdoA1ed3", "reviewer": "Reviewer_9C8q", "summary": "This paper establishes a direct connection between two important frameworks: neural episodic control and Universal Hopfield Network. It further derives Lyapunov functions for the dynamics and explores the ability of Neural Episodic Control to function as a memory system.", "questions": "- Can the authors further clarify how does the proposed connection between Hopfield Networks and reinforcement learning algorithms advance the current understanding of episodic memory in neural networks?\n- In deriving the energy functions, specific modifications were made to the separation function κ. What are the theoretical justifications for these modifications, and how do they impact the overall model performance?\n- The paper suggests that the Manhattan distance kernel improves performance over the Euclidean distance kernel. Can you provide a detailed analysis of why this might be the case?\n- The introduction of a new criterion to disentangle memorization from generalization is an interesting contribution. How does this criterion compare to existing methods of evaluating associative memory models?", "limitations": "None", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The idea of the paper is original and the manuscript is clear. The new theoretical connection is significant and important in relating previously unrelated frameworks.", "weaknesses": "While the contribution is relatively important, it is solely based on a mathematical equivalence between the two frameworks. It is a bit unclear to me, whether such an equivalence runs deep and might affect the way memory and RL systems co-function together. In a way, it feels like this paper could dig deeper into an underlying unifying framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 9C8q for the constructive feedback and positive remarks on our work. We appreciate the recognition of the originality and clarity of our manuscript, as well as the significance of the theoretical connections made. Below, we address the specific questions and points raised.\n\n1. Comment: \"While the contribution is relatively important, it is solely based on a mathematical equivalence between the two frameworks. It is a bit unclear to me, whether such an equivalence runs deep and might affect the way memory and RL systems co-function together. In a way, it feels like this paper could dig deeper into an underlying unifying framework.\"\n    - Answer: Thank you for highlighting the importance of our contribution. To better highlight the connection between DND and UHN, we have added a new figure illustrating this equivalence. Our empirical results also provide clear insights into how RL methods based on episodic control could be improved by leveraging this connection. We are currently testing these hypotheses in RL settings, which we believe will further validate and deepen our understanding of this unifying framework.\n\n2. Question: \"Can the authors further clarify how does the proposed connection between Hopfield Networks and reinforcement learning algorithms advance the current understanding of episodic memory in neural networks?\"\n    - Answer: The proposed connection allows us to leverage the well-studied dynamics of Hopfield Networks to improve the efficiency and effectiveness of neural episodic control in RL. By understanding how memories are stored and retrieved within this framework, we can design RL systems that better integrate episodic memory, leading to improved decision-making and faster learning. This connection enhances our understanding of episodic memory by providing a unified theoretical framework that explains memory dynamics in both associative memory models and RL systems.\n\n3. Question: \"In deriving the energy functions, specific modifications were made to the separation function [k-max]. What are the theoretical justifications for these modifications, and how do they impact the overall model performance?\"\n    - Answer: The modifications to k-max were made to derive the energy functions for neural episodic control. We ensured the new function has nonzero gradient while preserving the functional properties of k-max. As beta and beta_k grow to infinity, the new function approaches k-max. Hence, given sufficently large beta and beta_k, the overall model performance is not affected. We have included this justification in the revised manuscript.\n\n4. Question: \"The paper suggests that the Manhattan distance kernel improves performance over the Euclidean distance kernel. Can you provide a detailed analysis of why this might be the case?\"\n    - Answer: Similar to the findings of Millidge et al. (2022) in their Universal Hopfield Network paper, we observe that the Manhattan distance kernel outperforms the Euclidean distance kernel. The selection of similarity functions is crucial because it significantly impacts the ranking of different memories. While the exact reason why the Manhattan distance yields better results is not entirely clear, one possible explanation is that the Euclidean distance involves squaring the differences, which can introduce distortion, whereas the Manhattan distance preserves linearity.\n\n5. Question: \"The introduction of a new criterion to disentangle memorization from generalization is an interesting contribution. How does this criterion compare to existing methods of evaluating associative memory models?\"\n    - Answer: We thank the reviewer for highlighting out contribution. Our new criterion provides a more nuanced evaluation by explicitly distinguishing between an associative memory model's ability to memorize specific instances and its ability to generalize across similar instances. We show that the k-max separation function (k>1) can yield state of the art performance with the standard generalization performance. This contradicts the prediction of Millidge et al. (2022) that 1-max is the optimal separation function. We also point out that 1-max is optimal when performance is assessed with the new memorization criterion. Hence, our work challenges the canonical criterion and allows for a richer evaluation of associative memory models.\n\nWe hope these clarifications address your concerns and enhance the understanding and impact of our work. Thank you again for your valuable feedback, which has helped improve our manuscript significantly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xgctKwfG6W", "reviewer": "Reviewer_ebPC", "summary": "This paper introduces a formulation of an energy function which involves retrieving top-k memories while making a connection to both Neural Episodic Control and Associative Memory. The novel energy function utilized in this work demonstrates superior performance in the retrieval setting which involves image in-painting where the top half of an image is masked out. Additionally, the work explores the efficacy of the model under the retrieval setting in which queries are perturbed with a range of noise values.", "questions": "Why is the range of $\\beta$, 0 to 50000, chosen for figure 5? Such values seem to be too large, while the range of k makes sense.\n\nLooking at figures 6 and 8, it seems that when k = 1, the model performs best on CIFAR10 and Tiny ImageNet which is an opposite trend to MNIST. What is the explanation behind this fact?\n\nFor figure 7, as k increases, the performance of the model increases given the setting of denoising. What is the number of images that the model is being evaluated on? Could a visual example of denoising Tiny ImageNet images be provided?\n\nWhat is the threshold utilize to determine whether a recovered pattern is memorized or not for each of the dataset?\n\n\"The novel criterion is such that a trial is correct if and only if the squared pixel difference between the truth and the output is lower than it is between the output and any other memory\" --- Why is there no equation which describes this important criterion? \n\nGeneral comment: Since the maximum chosen k value is set to 50 in the tables, I think it would be good to include k = 100 in such tables as it is demonstrated in figure 5.", "limitations": "The connection to Neural Episodic Control and RL is too brief. Additionally, the function does not beat the softmax based energy function in terms of retrieval while as the number of neighbors k increases, the performance is worsen on CIFAR10 and TinyImageNet datasets.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "The proposed energy function is novel and well thought. The formulation follows the Neural Episodic Control and Universal Hopfield Network (UHN) paradigms. Moreover, the experiments are interesting, and the model is shown to have great performance, specifically the illustrations of improved memorization capacity across a variety of k-Max functions demonstrated on MNIST. When it comes to denoising, the proposed model is able to recover patterns very well as k increases.", "weaknesses": "Although the experiments are good, many of the experiments are ablation studies of the introduced energy function, while there is one experiment section contrasting the proposed energy function and softmax based energy function, which illustrates a worse performance of the new function. Moreover, when dealing with RGB images, the function performs badly as k-neighbors increases. Finally, the connection to RL is fascinating but it seems too brief in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[concise because hitting char lim]\n\nThank you for your constructive feedback and insights on our work.\n\n1. Comment: \"Although the experiments are good, many of the experiments are ablation studies of the introduced energy function, while there is one experiment section contrasting the proposed energy function and softmax based energy function, which illustrates a worse performance of the new function.\"\n    - Answer: Yes, and we think these are important findings. The Manhattan similarity function consistently outperforms the Euclidean one, even with softmax, as noted by Millidge et al. (2022). This suggests that NEC can be improved through our connection between the DND and UHN. We also argue that softmax could improve the flexibility and performance of NEC. Conversely, k-max constitutes a strong alternative due to its superior search complexity, especially when implemented with a k-d tree, as we now argue in the new version of the manuscript.\n\n2. Comment: \"Moreover, when dealing with RGB, the function performs badly as k increases.\"\n    - Answer: Several factors could indeed influence how robust performance is to changes in k, such as the use of RGB images, the higher dimensionality and more naturalistic nature of complex images compared to MNIST.\n\n3. Comment: \"The connection to RL is fascinating but it seems too brief in the paper.\"\n    - Answer: Thank you for finding the connection fascinating. Our paper establishes a comprehensive theoretical link between NEC and associative memory models. Additionally, we present a set of new empirical results demonstrating the applicability of NEC to associative memory tasks. We are now working on exploring how insights from associative memory can in turn enhance NEC.\n\n4. Question: \"Why is the range of beta, 0 to 50000, chosen for figure 5? Such values seem to be too large, while the range of k makes sense.\"\n    - Answer: The experiment for Figure 5 is computationally expensive, so we aimed to identify peak performance across a wide range for all datasets and similarity functions. We found that peak performance does not always occur with low beta values (fig. 11, 10). Finally, the range already demonstrates that softmax most often outperforms k-max. Please let us know if you have any more concerns.\n\n5. Question: \"Looking at figures 6 and 8, it seems that when k = 1, the model performs best on CIFAR10 and Tiny ImageNet which is an opposite trend to MNIST. What is the explanation behind this fact?\"\n    - Answer: We believe your comment may be a reference to Figures 6 and 7 instead. We indeed observe that for lower-dimensional datasets like MNIST (and CIFAR-10 to a lesser extent), values of k>1 can lead to improved performance. This finding contradicts the statement by Millidge et al. (2022) that k=1 is always optimal and motivates our introduction of a new performance criterion where k=1 becomes optimal. These points are discussed in our abstract (lines 8 to 14) and throughout the manuscript.\n\n6. Question: \"For figure 7, as k increases, the performance of the model increases given the setting of denoising. What is the number of images that the model is being evaluated on? Could a visual example of denoising Tiny ImageNet images be provided?\"\n    - Answer: Thank you very much for highlighting this crucial detail. We apologize for the oversight. Similar to the approach used by Millidge et al. (2022), we evaluated the model using sets of 100 images. We have now included this information in the revised manuscript. We also appreciate your suggestion to provide a visual example of denoising, and have added such examples for Tiny ImageNet in the supplementary material.\n\n7. Question: \"What is the threshold utilize to determine whether a recovered pattern is memorized or not for each of the dataset?\"\n    - Answer: The error threshold is used for the absolute criterion and is set to 50, consistent with the approach used by Millidge et al. (2022), who utilized the same datasets. We mentioned this criterion in line 125 of the manuscript, and we have now made it clearer in the revised version to ensure this important detail is easily accessible.\n\n8. Question: \"Why is there no equation which describes [the generalization] criterion?\"\n    - Answer: Thank you for pointing this out. Upon review, we realized that our previous description was incorrect. The correct criterion allows the sum of squared pixel differences to be equal, not just less, for a trial to be correct. We appreciate your suggestion to include an equation, as it prompted us to clarify this aspect. \n    \n        The retrieval output is denoted as $z$, and the matrix of stored memories is $K$. A trial is considered correct if the sum of squared pixel differences between $z$ and the correct memory $K_{\\text{correct}}$ is equal to the minimum difference between $z$ and any memory $K_i$, i.e. $z - K_{\\text{correct}} \\|^2 = \\min_{i} \\| z - K_i \\|^2$.\n        This adjustment to the phrasing does not affect our results, as our implementation already adhered to this correct criterion. We have included this equation and corrected the phrasing in the revised manuscript to ensure clarity and accuracy.\n\n9. Comment: \"Since the maximum chosen k value is set to 50 in the tables, I think it would be good to include k = 100 in such tables as it is demonstrated in figure 5.\"\n    - Answer: Thank you for your suggestion regarding the inclusion of k=100 in the tables, but we do not have such data. For tables, the max is set to k=50 like in the original NEC paper. Other experiments have consistently shown that performance peaks before k=50, as evidenced by the results in Figures 5, 10, and 11. Let us know if you have any more concerns.\n\nWe appreciate the time and effort that Reviewer ebPC has put into evaluating our paper. Your constructive feedback has helped us clarify key aspects of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NQAgjpzs98", "reviewer": "Reviewer_w6Cb", "summary": "The paper shows that the Differentiable Neural Dictionary (DND) used in the context of reinforcement learning is mathematically equivalent to an Hopfield network with heteroassociative memories. Based on this observation, the paper generalises DND using the formulation of Universal Hopfield Networks (UHN) and studies the effect on 'retrieval' and 'capacity' of different separation and similarity functions. Additionally, the authors introduce a new criterion to measure the retrieval capability of an associative memory, which, they argue is a better proxy for memorisation.", "questions": "Some minor points and questions: \n- In Section 2 what are the dimensions of the vectors and matrices, e.g. V, K, q, $\\phi$, etc? Please rectify directly in the paper.\n- The definition of the similarity function used by DND has an unusual $\\delta$ in it. That is probably there for numerical stability reasons and should be ignored in the mathematical treatment. \n- Equations (7-9) are obvious and not needed. \n- The dynamic and differentiable way of defining the threshold (Equations 15-16) is clever but not necessary since you can just zero-out the non top-K ones? (which you do in the experiments if I understood correctly) \n- It is not really clear to me the difference between capacity and retrieval, which is central to section 3 so you might want to state it somewhere at the beginning of it. The line between them seems very thin to me. \n- Line 134-135 maybe you wanted to say the opposite?\n-In section 3 you are testing an hypothesis which has not been clearly spelled out before. If I understand correctly, your hypothesis is that the k-max function is useful for generalisation or memorisation depending on the value of k chosen. I believe you have tried to mention this in the introduction of Section 3 (lines 109-112) but I think it's not clear and confusing at the moment.", "limitations": "I can see no major limitations, but the paper could have been developed more.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is generally well written and has a good structure. Particular strengths reside in: \n\n- Solid contribution: the characterisation of differentiable neural dictionaries as associative memories can help future research in the field of neural episodic control by applying the advancements of the research on associative memories to the DND implementation. \n- Soundness of the approach: the authors establish the connection theoretically and then experimentally study the effect of changing the form of the DND associative memory on the memory capacity. They use enough baselines and different datasets, so the results are convincing.", "weaknesses": "I think the paper is relatively light, meaning that there is little content. In particular, the authors focus most of the experiments on studying the effect of different similarity and separation functions on capacity for storing images, whilst they could have investigated the implications in the reinforcement learning domain more. It's unclear why the study of the 'max' separating function in comparison to softer versions thereof is so important for this paper: it indeed seems that section 3 belongs to another paper, which talks about the characteristics of different separating functions.\nAnother example is the Discussion section, which occupies a page and a half mostly with speculative sentences and arbitrary connections to neuroscience research, which are far from justified in the context of this paper. It would have been much better if the paper actually empirically or theoretically attempted at establishing this connections. For an example see lines 217-219.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer w6Cb for the constructive feedback and for acknowledging the strengths of our paper, including the solid contribution and soundness of our approach. We appreciate the recognition of our work in characterizing Differentiable Neural Dictionaries (DNDs) as associative memories and its potential to advance research in neural episodic control. We also thank you for pointing raising valid comments. We directly included explicitly changes you proposed (e.g. clarification of matrix dimensionality), and we address additional concerns and questions that call for a response below.\n\n1. Comment: \"I think the paper is relatively light [...]\"\n    - Answer: We agree that further exploring reinforcement learning (RL) implications is valuable, and we are actively pursuing this direction in ongoing work. The primary focus of this manuscript was to describe the theoretical connection between DNDs and Hopfield networks and to evaluate the memory capabilities of DNDs. Due to the page limit and dense supplementary information, we concentrated on establishing this foundational work. We look forward to sharing new results on RL applications in future publications.\n\n2. Comment: \"It's unclear why the study of the 'max' separating function in comparison to softer versions thereof is so important for this paper: it indeed seems that section 3 belongs to another paper, which talks about the characteristics of different separating functions.\"\n    - Answer: Thank you for your feedback. All experiments, including the one you mentioned, evaluate the k-max (not only max) function to demonstrate its unique properties and effectiveness in different contexts. We believe this comparison provides valuable insights into optimizing memory retrieval strategies. More precisely, it suggests that NEC could be improved by replacing k-max by softmax. Does this clarification address your concern?\n\n3. Comment: \"Another example is the Discussion section, which occupies a page and a half mostly with speculative sentences and arbitrary connections to neuroscience research\"\n    - Answer: Thank you for pointing that out. We do think the connection to neuroscience can be made and \"adds an interesting interdisciplinary perspective\" (Reviewer 4Gfy). However, we replaced speculative content by referenced claims.\n\n4. Question: \"The dynamic and differentiable way of defining the threshold (Equations 15-16) is clever but not necessary since you can just zero-out the non top-K ones? (which you do in the experiments if I understood correctly)\"\n    - Answer: You are right. This new definition was introduced to derive the energy functions for neural episodic control, but it was not used in the experiments.\n\n5. Question: \"It is not really clear to me the difference between capacity and retrieval, which is central to section 3 so you might want to state it somewhere at the beginning of it. The line between them seems very thin to me.\"\n    - Answer: Thank you for bringing this to our attention. We adhered to the distinction of Millidge et al. (2022). Capacity refers to the maximum number of images the DND can store while maintaining accurate memory representation. Retrieval, on the other hand, measures the model’s ability to accurately recall stored images when subjected to increasing levels of noise. We have added a definition and explanation of these terms in the manuscript to clarify their distinction.\n\nWe appreciate the constructive feedback provided by Reviewer w6Cb, which has been instrumental in enhancing the clarity and depth of our manuscript. We have addressed the concerns and questions raised, and we believe these revisions have strengthened our work. We are committed to further exploring the implications of our findings in reinforcement learning and look forward to sharing these insights in future publications. Thank you again for your thoughtful and valuable comments, which have helped us improve our contribution to the field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their detailed and constructive feedback on our submission. We are pleased that the reviewers recognize our significant theoretical contribution of linking Differentiable Neural Dictionaries (DNDs) to the Universal Hopfield Network (UHN) framework (R.4Gfy) and the extensive empirical evaluation we conducted (R.4Gfy). The clarity and articulation of our manuscript were also noted positively (R.UfDJ, R.4Gfy, R.9C8q, R.w6Cb, R.ebPC).

Additionally, our novel approach to integrating associative memory with reinforcement learning was appreciated (R.UfDJ), along with the introduction of a new "memorization criterion" (R.4Gfy) and the originality and significance of relating previously unrelated frameworks (R.9C8q). The value of deriving new energy functions for DNDs was also recognized, both theoretically (R.4Gfy) and empirically (R.ebPC).

In response to the reviewers' comments, we have enhanced the clarity, organization, and practical relevance of our manuscript. We addressed concerns about the focus on image reconstruction tasks, provided a clearer discussion of practical implications and computational complexity, and added visual aids to improve accessibility.

In individual rebuttals, we detail our responses to each of the reviewers' comments and outline the revisions made to address their concerns.

Meta-review: The paper connects differentiable dictionaries, used in neural episodic control, with a new model from the Dense Associative Memory family. The paper received favorable reviews and for this reason is accepted. I request that in addition to addressing the issues raised by the reviewers the authors better position their contribution relative to the line of work on Dense Associative Memories (Modern Hopfield Networks). 

For instance, the sentence “Some important limitations have been addressed with the development of differentiable continuous Hopfield Networks (2)”  in lines 18, 19 is incorrect. Continuous differentiable Hopfield Networks have been developed by J.Hopfield in 1984 in this paper: https://www.pnas.org/doi/abs/10.1073/pnas.81.10.3088, long before reference (2). The contribution of Ref (2) was to connect the continuous formulation of Modern Hopfield Networks from this paper: https://papers.nips.cc/paper_files/paper/2016/hash/eaae339c4d89fc102edd9dbdb6a28915-Abstract.html with the attention mechanism; and introduce a specific model (from the continuous Modern Hopfield Network family) with a softmax activation. When I read the introduction I am left with an impression that the authors have a limited understanding of the origins of the ideas and methods that they are using in their work. This issue should be addressed in the camera ready version.
Decision: Accept (poster)