SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hamza Tahir Chaudhry $^{1,2}$ , Jacob A. Zavatone-Veth $^{2,3}$ , Dmitry Krotov $^{5}$ , Cengiz Pehlevan $^{1,2,4}$
$^{1}$ John A. Paulson School of Engineering and Applied Sciences,
$^{2}$ Center for Brain Science,  $^{3}$ Department of Physics,
MIT-IBM Watson AI Lab, IBM Research,
Cambridge, MA 02142
hchaudhry@g.harvard.edu, jzavatoneveth@g.harvard.edu,
krotov@ibm.com, cpehlevan@seas.harvard.edu
Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules. However, these networks suffer from limited sequence capacity (maximal length of the stored sequence) due to interference between the memories. Inspired by recent work on Dense Associative Memories, we expand the sequence capacity of these models by introducing a nonlinear interaction term, enhancing separation between the patterns. We derive novel scaling laws for sequence capacity with respect to network size, significantly outperforming existing scaling laws for models based on traditional Hopfield networks, and verify these theoretical results with numerical simulation. Moreover, we introduce a generalized pseudoinverse rule to recall sequences of highly correlated patterns. Finally, we extend this model to store sequences with variable timing between states' transitions and describe a biologically-plausible implementation, with connections to motor neuroscience.
Memory is an essential ability of intelligent agents that allows them to encode, store, and retrieve information and behaviors they have learned throughout their lives. In particular, the ability to recall sequences of memories is necessary for a large number of cognitive tasks with temporal or causal structure, including navigation, reasoning, and motor control [1-9].
Computational models with varying degrees of biological plausibility have been proposed for how neural networks can encode sequence memory [1-3, 10-22]. Many of these are based on the concept of associative memory, also known as content-addressable memory, which refers to the ability of a system to recall a set of objects or ideas when prompted by a distortion or subset of them. Modeling associative memory has been an extremely active area of research in computational neuroscience and deep learning for many years, with the Hopfield network becoming the canonical model [23-25].
Unfortunately, a major limitation of the traditional Hopfield Network and related associative memory models is its capacity: the number of memories it can store and reliably retrieve scales linearly with the number of neurons in the network. This limitation is due to interference between different memories during recall, also known as crosstalk, which decreases the signal-to-noise ratio. Large amounts of crosstalk results in the recall of undesired attractor states of the network [26-29].
Figure 1: SeqNet and Polynomial DenseNet ( $d = 2$ ) are simulated with  $N = 300$  neurons and  $P = 100$  patterns. One hundred curves are plotted as a function of time, each representing the overlap of the network state at time  $t$  with one of the patterns,  $m^{\mu} = (1/N)\sum_{i=1}^{N}\xi_i^\mu S_i$ . The curves are ordered using the color code described on the right (patterns in the beginning and end of the sequence are shaded in yellow and red respectively). A. SeqNet quickly loses the correct sequence, indicated by the lack of alignment of the network state with the correct pattern in the sequence ( $m^{\mu} \ll 1$ ). B. The Polynomial DenseNet faithfully recalls the entire sequence and maintains alignment with one of the patterns at any moment in time,  $m^{\mu} \approx 1$ .
Recent modifications of the Hopfield Network, known as Dense Associative Memories or Modern Hopfield Networks (MHNs), overcome this limitation by introducing a strong nonlinearity when computing the overlap between the state of the network and memory patterns stored in the network [30, 31]. This leads to greater separation between partially overlapping memories, thereby reducing crosstalk, increasing the signal-to-noise ratio, and increasing the probability of successful recall [32].
Most models based on the Hopfield Network are autoassociative, meaning they are designed for the robust storage and recall of individual memories. Thus, they are incapable of storing sequences of memories. In order to adapt these models to store sequences, one must utilize asymmetric weights in order to drive the network from one activity pattern to the next. Many such models use temporally asymmetric Hebbian learning rules to strengthen synaptic connections between neural activity at one time state and the next time state, thereby learning temporal association between patterns in a sequence [1, 3, 10, 11, 16, 17, 22].
In this paper, we extend Dense Associative Memories to the setting of asymmetric weights in order to store and recall long sequences of memories. We work directly with the update rule for the state of the network, allowing us to provide an analytical derivation for the sequence capacity of our proposed network. We find a close match between theoretical calculation and numerical simulation, and further establish the ability of this model to store and recall sequences of correlated patterns. Additionally, we examine the dynamics of a model containing both symmetric and asymmetric terms. Finally, we describe applications of our network as a model of biological motor control.
2 DenseNets for Sequence Storage
Traditional Hopfield Networks and MHNs, as described in Appendix B, are capable of storing individual memories. What about storing sequences? Assume that we want to store a sequence of  $P$  patterns,  $\pmb{\xi}^{1} \rightarrow \pmb{\xi}^{2} \rightarrow \dots \rightarrow \pmb{\xi}^{P}$ , where  $\xi_{j}^{\mu} \in \{\pm 1\}$  is the  $j^{th}$  neuron of the  $\mu^{th}$  pattern and the network will transition from pattern  $\pmb{\xi}^{\mu}$  to  $\pmb{\xi}^{\mu + 1}$ . Let  $N$  be the number of neurons in the network and  $\mathbf{S}(t) \in \{-1, +1\}^{N}$  be the state of the network at time  $t$ . We want to design a network with dynamics such that when the network is initialized in pattern  $\pmb{\xi}^{1}$ , it will traverse the entire sequence. We define a network, SeqNet, which follows a discrete-time synchronous update rule:
$$
T _ {S N} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {j \neq i} J _ {i j} S _ {j} \right] = \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu + 1} m _ {i} ^ {\mu} \right], \quad m _ {i} ^ {\mu} := \frac {1}{(N - 1)} \sum_ {j \neq i} \xi_ {j} ^ {\mu} S _ {j}, \tag {1}
$$
<sup>1</sup>We impose periodic boundary conditions and define  $\pmb{\xi}^{P + 1}\equiv \pmb{\xi}^{1}$ . Boundary terms have a sub-leading contribution to the crosstalk, so a model with open boundary conditions will have the same scaling of capacity.
2One can also consider an asynchronous update rule in which one neuron is updated at a time [23, 26].
where  $\mathbf{S}(t + 1) = T_{SN}(\mathbf{S})$  and  $J_{ij} = \frac{1}{N}\sum_{\mu = 1}^{P}\xi_i^{\mu +1}\xi_j^\mu$  is an asymmetric matrix connecting pattern  $\xi^{\mu}$  to  $\xi^{\mu +1}$ . Note that we are excluding self-interaction terms  $i = j$ . We also rewrote the dynamics in terms of  $m_i^\mu$ , the overlap of the network state  $\mathbf{S}$  with pattern  $\xi^{\mu}$ . When the network is aligned most closely with pattern  $\xi^{\mu}$ , the overlap  $m_i^\mu$  is the largest contribution in the sum and pushes the network to pattern  $\xi^{\mu +1}$ . When multiple patterns have similar overlaps, meaning they are correlated, then there will be low signal-to-noise ratio. This correlation between patterns limits the capacity of the network, limiting the SeqNet's capacity to scale linearly relative to network size.
To overcome the capacity limitations of the SeqNet, we use inspiration from Dense Associative Memories [30] to define the DenseNet update rule:
$$
T _ {D N} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu + 1} f \left(m _ {i} ^ {\mu}\right) \right] \tag {2}
$$
where  $f$  is a nonlinear monotonically increasing interaction function. Similar to MHNs,  $f$  reduces the crosstalk between patterns and, as we will analyze in detail, leads to improved capacity. Figure 1 demonstrates this improvement for  $f(x) = x^2$ .
2.1 Sequence capacity
To derive analytical results for the capacity, we must choose a distribution to generate the patterns. As is standard in studies of the classic HN and MHNs [26-31, 33-36], we choose this to be the Rademacher distribution, where  $\xi_j^\mu \in \{-1, +1\}$  with equal probability for all neurons  $j$  in all patterns  $\mu$ , and calculate the capacity for different update rules. If one is allowed to specially engineer the patterns, even the SeqNet can store a sequence of length  $2^N$  [37], but this construction is not relevant to associative recall of realistic sequences. Rademacher patterns are a more appropriate model for generic patterns while remaining theoretically tractable.
We consider both the robustness of a single transition, and the robustness of propagation through the full sequence. For a fixed network size  $N \in \{2, 3, \ldots\}$  and an error tolerance  $c \in [0, 1)$ , we define the single-transition and sequence capacities by
$$
P _ {T} (N, c) = \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: \mathbb {P} \left[ \mathbf {T} _ {D N} (\boldsymbol {\xi} ^ {1}) = \boldsymbol {\xi} ^ {2} \right] \geq 1 - c \right\} \tag {3}
$$
$$
P _ {S} (N, c) = \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: \mathbb {P} \left[ \cap_ {\mu = 1} ^ {P} \left\{\mathbf {T} _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) = \boldsymbol {\xi} ^ {\mu + 1} \right\} \right] \geq 1 - c \right\}, \tag {4}
$$
respectively, where the probability is taken over the random patterns. Note that for the single-transition capacity we could focus on any pair of subsequent patterns due to translation invariance arising from periodic boundary conditions. Also note that the full sequence capacity is defined by demanding that all transitions are correct. For perfect recall, we want to take the threshold  $c \downarrow 0$ . In the thermodynamic limit in which  $N, P \to \infty$ , we expect for there to exist a sharp transition in the recall probabilities as a function of  $P$ , with almost-surely perfect recall below the threshold value and vanishing probability of recall above [26-29, 31, 33-36]. Thus, we expect the capacity to become insensitive to the value of  $c$  in the thermodynamic limit; this is known rigorously for the classic Hopfield network from the work of Bovier [34].
As we detail in Appendix C, all of our theoretical results are obtained under two approximations. We will validate the accuracy of the resulting capacity predictions through comparison with numerical experiments. First, following Petritis [33]'s analysis of the classic Hopfield network, we use union bounds to control the single-transition and full-sequence capacities in terms of the single-bitflip error probability  $\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \xi_1^2]$ . Using the fact that the patterns are i.i.d., this gives  $\mathbb{P}[\mathbf{T}_{DN}(\pmb{\xi}^\mu) = \pmb{\xi}^{\mu + 1}] \geq 1 - N\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \xi_2^1]$  and  $\mathbb{P}[\cap_{\mu = 1}^P\{\mathbf{T}_{DN}(\pmb{\xi}^\mu) = \pmb{\xi}^{\mu + 1}\}] \geq 1 - N P\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \xi_2^1]$ , respectively, resulting in the lower bounds
$$
P _ {T} (N, c) \geq \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: N \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \leq c \right\}, \tag {5}
$$
$$
P _ {S} (N, c) \geq \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: N P \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \leq c \right\}. \tag {6}
$$
From studies of the classic Hopfield network, we expect for these bounds to be tight in the thermodynamic limit  $(N\to \infty)$ , but we will not attempt to prove that this is so [33, 34]. Second, our theoretical
results are obtained under the approximation of  $\mathbb{P}[T_{HN}(\pmb{\xi}^1)_1 \neq \xi_1^2]$  in the regime  $N, P \gg 1$  by a Gaussian tail probability. Concretely, we write the single-bitflip probability as
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] = \mathbb {P} [ C <   - f (1) ] \tag {7}
$$
in terms of the crosstalk
$$
C = \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu + 1} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right), \tag {8}
$$
which represents interference between patterns that can lead to a bitflip. Then, as the crosstalk is the sum of  $P - 1$  i.i.d. random variables, we approximate its distribution as Gaussian. We then extract the capacity by determining how  $P$  should scale with  $N$  such that the error probability tends to zero as  $N \to \infty$ , corresponding to taking  $c \downarrow 0$  with increasing  $N$ . Within the Gaussian approximation, we can also estimate the capacity at fixed  $c$  by using the asymptotics of the inverse Gaussian tail distribution function to determine how  $P$  should scale with  $N$  such that the error probability is asymptotically bounded by  $c$  as  $N \to \infty$ . This predicts that the effect of non-negligible  $c$  should vanish as  $N \to \infty$ .
For  $P$  large but finite, this Gaussian approximation amounts to retaining only the leading term in the Edgeworth expansion of the tail distribution function [38-41]. We will not endeavour to rigorously control the error of this approximation in the regime of interest in which  $N$  is also large. To convert our heuristic results into fully rigorous asymptotics, one would want to construct an Edgeworth-type series expansion for the tail probability  $\mathbb{P}[C < -f(1)]$  that is valid in the joint limit with rigorously-controlled asymptotic error, accounting for the fact that the crosstalk is a sum of discrete random variables [38-41]. As a simple probe of Gaussianity, we will consider the excess kurtosis of the crosstalk distribution, which determines the leading correction to the Gaussian approximation in the Edgeworth expansion, and describes whether its tails are heavier or narrower than Gaussian [38-41].
2.2 Polynomial DenseNet
Consider the DenseNet with polynomial interaction function,  $f(x) = x^{d}$ , which we will call the Polynomial DenseNet. In Appendix C.1, we argue that the leading asymptotics of the transition and sequence capacities for perfect recall are given by
$$
P _ {T} \sim \frac {N ^ {d}}{2 (2 d - 1) ! ! \log (N)}, \quad P _ {S} \sim \frac {N ^ {d}}{2 (d + 1) (2 d - 1) ! ! \log (N)}. \tag {9}
$$
Note that this polynomial scaling of the single-transition capacity with network size coincides with the capacity scaling of the symmetric MHN [30]. Indeed, as we have excluded self-interaction terms in the update rule, the single-bitflip probabilities for these two models coincide exactly for unbiased Radamacher patterns (Appendix C.1). This allows us to adapt arguments from Demircigil et al. [31] to show that (9) is in fact a rigorous asymptotic lower bound on the capacity (Appendix D). We compare our results for the single-transition and sequence capacities to numerical simulation in Figure 2. The simulation matches theoretical prediction for large network size  $N$ . For smaller  $N$ , there are finite-size effects that result in deviation from theoretical prediction. The crosstalk has non-negligible kurtosis in finite size networks which leads to deviation from the Gaussian approximation.
Furthermore, we point out that for fixed  $N$ , the network capacity does not monotonically increase in the degree  $d$ . Since the factorial function grows faster than the exponential function, every finite network of size  $N$  has a polynomial degree  $d_{max}$  after which the capacity will actually decrease. This is also true for the standard MHN. We demonstrate this numerically in Figure 2B, again noting mild deviations between theory and simulation due to finite-size effects.
2.3 Exponential DenseNet
We have shown the DenseNet's capacity can scale polynomially with network size. Can it scale exponentially? Consider the DenseNet with exponential interaction function,  $f(x) = e^{(N - 1)(x - 1)}$ , which we call the Exponential DenseNet. This function reduces crosstalk dramatically:  $f(m^{\mu}(\mathbf{S})) = 1$  when  $m^{\mu}(\mathbf{S}) = 1$  and is otherwise sent to zero exponentially fast. In Appendix C.2, we show that under the abovementioned approximations one has the leading asymptotics
$$
P _ {T} \sim \frac {\beta^ {N - 1}}{2 \log N} \quad \text {a n d} \quad P _ {S} \sim \frac {\beta^ {N - 1}}{2 \log (\beta) N}, \quad \text {w h e r e} \quad \beta = \frac {\exp (2)}{\cosh (2)} \simeq 1. 9 6 4 \dots \tag {10}
$$
Figure 2: Testing the transition and sequence capacities of DenseNets with polynomial and exponential nonlinearities. A. Scaling of transition capacity  $(\log_{10}(P_T), left)$  and sequence capacity  $(\log_{10}(P_S), right)$  with network size. As network size increases, the variance of the crosstalk decreases and the theoretical approximations become more accurate, resulting in a tight match between theory (solid lines) and simulation (points with error bars). The theory curves are given by Equations 9 and 10. Error bars are computed across realizations of the random patterns (see Appendix G). There is significant deviation between theory and simulation for the sequence capacity of the Exponential DenseNet. We show that this is due to finite-size effects in Section 2.3. B. Transition capacity of Polynomial DenseNets as a function of degree. For any finite network size  $N$ , there is a degree  $d$  that maximizes the transition capacity. The same would be true for the sequence capacity. C. Crosstalk variance (left) and excess kurtosis (right) for the Exponential DenseNet as a function of  $P$  and  $N$ . Variance is proportional to  $P$  and inversely proportional to  $N$ , while the opposite is true for excess kurtosis. See Appendix G for details of our numerical methods.
In Figure 2, numerical simulations confirm this model scales significantly better than the Polynomial DenseNet and enables one to store exponentially long sequences relative to network size. While the ratio between transition and sequence capacities remains bounded for the Polynomial DenseNet, where  $P_T / P_S \sim d + 1$ , the gap for the Exponential DenseNet diverges with network size.
However, we can see in Figure 2A that the empirically measured capacity—particularly the sequence capacity—of the Exponential DenseNet deviates substantially from the predictions of our approximate Gaussian theory. Due to computational constraints, our numerical simulations are limited to small network sizes (Appendix G). Computing the excess kurtosis of the crosstalk distribution with a number of patterns comparable to the capacity predicted by the Gaussian theory reveals that, for the range of system sizes we can simulate, the distribution should deviate strongly from a Gaussian. In particular, if take  $P \sim \beta^{N-1} / (\alpha N)$  for some constant factor  $\alpha$ , then the excess kurtosis increases with network size up to around  $N \approx 56$  (Appendix C.2). Increasing the size of an Exponential DenseNet therefore has competing effects: for a fixed sequence length  $P$ , increasing network size  $N$  decreases the crosstalk variance, which should reduce the bitflip probability, but also increases the excess kurtosis, which reflects a fattening of the crosstalk distribution tails that should increase the bitflip probability. This is illustrated in Figure 2C.
The competition between increasing  $P$  and  $N$  for the Exponential DenseNet is easy to understand intuitively. For a fixed  $N$ , increasing  $P$  means that the crosstalk is equal in distribution to the sum of an increasingly large number of i.i.d. random variables, and thus by the central limit theorem should become increasingly Gaussian. Conversely, for a fixed  $P$ , increasing  $N$  means that each of the
Figure 3: A. Recall of a sequence of 200000 correlated images from the MovingMNIST dataset using DenseNets of size  $N = 784$ . We showcase a 10 image subsequence. The top row depicts the true sequence, the second row depicts SeqNet's performance, the next rows depict the Polynomial DenseNets' performance which increases with degree  $d$ , and the final row depicts the Exponential DenseNet's performance which yields perfect recall. B. Transition capacity of Polynomial DenseNets of size  $N = 100$  relative to pattern bias  $\epsilon$ . Increasing  $\epsilon$  monotonically decreases capacity. Networks with stronger nonlinearities maintain high capacity for large correlation strength. Implementing the generalized pseudoinverse rule decorrelates these patterns and maintains high sequence capacity for much larger correlation. See Appendix G for details of numerical methods.
$P - 1$  contributions to the crosstalk is equal in distribution to the product of an increasing number of i.i.d. random variables—as  $f\left(\frac{1}{N-1}\sum_{j=2}^{N}\xi_j^\mu\xi_j^1\right) = \prod_{j=2}^{N}\exp(\xi_j^\mu\xi_j^1)$ —and thus by the multiplicative central limit theorem each term should tend to a lognormal distribution. In this regime, then, the crosstalk is roughly a mixture of lognormals, which is decidedly non-Gaussian. In contrast, for a Polynomial DenseNet, memorization is easy in the limit where  $N$  tends to infinity for fixed  $P$ , as the crosstalk should tend almost surely to zero as each term  $f\left(\frac{1}{N-1}\sum_{j=2}^{N}\xi_j^\mu\xi_j^1\right) \to 0$  almost surely.
2.4 Recalling Sequences of Correlated Patterns
The full-sequence capacity scaling laws for these models were derived under the assumption of i.i.d Rademacher random patterns. While theoretically convenient, this is unrealistic for real-world data. We therefore test these networks in more realistic settings by storing correlated sequences of patterns, which will lead to greater crosstalk in each transition and thus smaller single-transition and full-sequence capacities relative to network size [26, 36]. However, the nonlinear interaction functions should still assist in separating correlated patterns to enable successful sequence recall.
For demonstration, we store a sequence of 200000 highly-correlated images from the MovingMNIST dataset and attempt to recall this sequence using DenseNets with different nonlinearities [42]. The entire sequence is composed of 10000 unique subsequences concatenated together, where each subsequence is composed of 20 images of two hand-written digits slowly moving through one another. This means there is significant correlation between patterns which will result in large amounts of crosstalk. The results of the DenseNets are shown in Figure 3A, where increasing the nonlinearity of the Polynomial DenseNets slowly improves recall but not entirely, while the exponential network achieves perfect recall. The SeqNet and DenseNets, up until approximately  $d = 50$ , are entirely unable to recall any part of any image, despite the DenseNets being well within the capacity limits predicted by theoretical calculations on uncorrelated patterns.
2.5 Generalized pseudoinverse rule
Can we overcome the DenseNet's limited ability to store correlated patterns? Drawing inspiration from the pseudoinverse learning rule introduced by Kanter and Sompolinsky [43] for the classic Hopfield network, we propose a generalized pseudoinverse (GPI) transition rule
$$
T _ {G P I} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu + 1} f \left(\sum_ {\nu = 1} ^ {P} (O ^ {+}) ^ {\mu \nu} m ^ {\nu} (\mathbf {S})\right) \right], \quad O ^ {\mu \nu} = \frac {1}{N} \sum_ {j = 1} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {\nu}, \tag {11}
$$