SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the overlap matrix  $O^{\mu \nu}$  is positive-semidefinite, so we can define its pseudoinverse  $\mathbf{O}^{+}$  by inverting the non-zero eigenvalues. With  $f(x) = x$ , this reduces to the pseudoinverse rule of [43].
If the patterns are linearly independent, such that  $\mathbf{O}$  is full-rank, we can see that this rule can perfectly recall the full sequence (Appendix E). This matches the classic pseudoinverse rule's ability to perfectly store any set of linearly independent patterns; this is why we choose to sum over  $\nu$  inside the separation function in (11). For i.i.d. Rademacher patterns, linear independence holds almost surely in the thermodynamic limit provided that  $P < N$ .
In Figure 3B, we demonstrate the effect of correlation on the Polynomial DenseNet through studying the recall of biased patterns  $\xi_i^\mu$  with  $\mathbb{P}(\xi_i^\mu = \pm 1) = \frac{1}{2} (1\pm \epsilon)$  for  $\epsilon \in [0,1)$ . We see that the Polynomial DenseNet has better recall at all levels of bias  $\epsilon$  as degree  $d$  increases, although we still expect there to be a maximum degree as described before. However, at large correlation values, they all have low recall, suggesting the need for alternative methods to decorrelate these patterns. This failure is easy to understand theoretically, following van Hemmen and Kuhn [44]'s analysis of the classic Hopfield model: for patterns with bias  $\epsilon$ , the Polynomial DenseNet update rule expands as
$$
T _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} = \operatorname {s g n} \left[ \xi_ {i} ^ {\mu + 1} + (P - 1) \epsilon^ {2 d + 1} + \mathcal {O} (\sqrt {P / N}) \right]. \tag {12}
$$
Therefore, even if  $N$  is large, for  $\epsilon \neq 0$  there must be some value of  $P$  for which the constant bias overwhelms the signal. If  $N \to \infty$  for any fixed  $P$ , then we must have  $P < \epsilon^{-(2d + 1)} + 1$  for the signal to dominate. In Figure 3B, we show the generalized pseudoinverse update rule is more robust to large correlations than the Polynomial DenseNet. While this rule can also be applied to the Exponential DenseNet, simulations fail due to numerical instability coming from small values in the pseudoinverse.
3 MixedNets for variable timing
Thus far, we have considered sequence recall in purely asymmetric networks. These networks transition to the next pattern in the sequence at every timestep, preventing the network from storing sequences with longer timing between elements. In this section, we aim to construct a model where the network stays in a pattern for  $\tau$  steps. Our starting model will be an associative memory model for storing sequences known as the Temporal Association Network (TAN) [1, 10], defined as:
$$
T _ {T A N} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \left[ \xi_ {i} ^ {\mu} m _ {i} ^ {\mu} + \lambda \xi_ {i} ^ {\mu + 1} \bar {m} _ {i} ^ {\mu} \right] \right], \quad \bar {m} _ {i} ^ {\mu} := \frac {1}{N - 1} \sum_ {j \neq i} \xi_ {j} ^ {\mu} \bar {S} _ {j} \tag {13}
$$
where  $\bar{m}_i^\mu$  represents the normalized overlap of each pattern  $\xi^{\mu}$  with a weighted time-average of the network over the past  $\tau$  timesteps,  $\bar{S}_i(t) = \sum_{\rho=0}^{\tau} w(\rho) S_i(t - \rho)$ . The weight function,  $w(t)$ , is generally taken to be a low-pass convolutional filter (e.g. Heaviside step function, exponential decay).
This network combines a symmetric and asymmetric term for robust recall of multiple sequences. The symmetric term containing  $m_i^\mu(t)$ , also referred to as a "fast" synapse, stabilizes the network in pattern  $\xi^\mu$  for a desired amount of time. The asymmetric term containing  $\bar{m}_i^\mu(t)$ , also referred to as a "slow" synapse, drives the network transition to pattern  $\xi^{\mu+1}$ . The  $\lambda$  parameter controls the strength of the transition signal. If  $\lambda$  is too small, no transitions will occur since the symmetric term will overpower it. If  $\lambda$  is too large, transitions will occur too quickly for the network to stabilize in a desired pattern and the sequence will quickly destabilize.
For TAN, Sompolinsky and Kanter [10] used numerical simulations to estimate the capacity as approximately  $P_{TAN} \sim 0.1N$ , defining capacity as the ability to recall the sequence in correct order with high overlap (meaning that a small proportion of incorrect bits are allowed in each transition). Note that this model can fail in two ways: (i) it can fail to recall the correct sequence of patterns, or (ii) it can fail to stay in each state for the desired amount of time.
To address these issues, we consider the following dynamics:
$$
T _ {M N} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \left[ \xi_ {i} ^ {\mu} f _ {S} \left(m _ {i} ^ {\mu}\right) + \lambda \xi_ {i} ^ {\mu + 1} f _ {A} \left(\bar {m} _ {i} ^ {\mu}\right) \right] \right] \tag {14}
$$
At  $\epsilon = 1$ , the patterns will be deterministic with  $\xi_{i}^{\mu} = +1$ .
Figure 4: Capacity of the Polynomial MixedNet. A. We simulate MixedNets with  $N = 100$ ,  $\tau = 5$  and attempt to store  $P = 40$  patterns. The Temporal Association Network (left), corresponding to a linear MixedNet with  $d_{S} = 1 = d_{A}$ , fails to recover the sequence. Increasing the nonlinearities to  $d_{S} = 2 = d_{A}$  (center) recovers the correct sequence order, but not the timing. Increasing the nonlinearities to  $d_{S} = 10 = d_{A}$  (right) recovers the correct sequence order and timing. B. Transition capacity  $\log_{10}(P_T)$  of the Polynomial MixedNet as a function of network size. Each panel has a fixed symmetric nonlinearity  $f_{S}(x) = x^{d_{S}}$  indicated by the panel's title. As network size increases, crosstalk variance decreases and theoretical approximations in Equation 3 become more accurate to tightly match the simulations. Note that as expected, the capacity scales according to the minimum of  $d_{S}$  and  $d_{A}$ . C. As in B, but for the sequence capacity  $\log_{10}(P_S)$ .
We call this model the MixedNet, and seek to analyze the relationship between the symmetric and asymmetric terms in driving network dynamics and their impact on sequence capacity. As before, the asymmetric term will try to push the network to the next state at every timestep, while the symmetric term tries to maintain it in its current state for  $\tau$  timesteps. We will allow different nonlinearities for  $f_{S}$  and  $f_{A}$ , and analyze their effect on transition and sequence capacity.
We demonstrate the effectiveness of the Polynomial MixedNet, where for simplicity we set  $f_{S}(x) = f_{A}(x) = x^{d}$ , in Figure 4A. While TAN fails completely, a polynomial nonlinearity of  $d = 2$  enables recall of pattern order but the network does not stay in each pattern for  $\tau = 5$  timesteps. Further increasing the nonlinearity to  $d = 10$  recovers the desired sequence with correct order and timing.
Theoretical analysis of the capacity of the MixedNet (14) for general memory length  $\tau$  is challenging due to the extended temporal interactions. We therefore consider single-step memory ( $\tau = 1$ ), and show that even in this relatively tractable special case new complications arise relative to our analysis of the DenseNet. Alternatively, we can interpret the MixedNet with  $\tau = 1$  as an imperfectly-learned DenseNet. If one imagines the network learns its weights through a temporally asymmetric Hebbian rule with an extended plasticity kernel, and its state is not perfectly clamped to the desired transition, the coupling from  $\xi^{\mu}$  to  $\xi^{\mu + 1}$  could be corrupted by coupling  $\xi^{\mu}$  to itself [22].
We first consider the setting where both interaction functions are polynomial,  $f_{S}(x) = x^{d_{S}}$  and  $f_{A}(x) = x^{d_{A}}$ , and refer to this network as the Polynomial MixedNet. This model is analyzed in detail in Appendix F.1. Interestingly, this model's crosstalk variance forms a bimodal distribution, as shown in Figure F.1. This complicates the analysis, but once bimodality is accounted for one can approximate the capacity using a similar argument to that of the DenseNet. We find that
$$
P _ {T} \sim \frac {(\lambda - 1) ^ {2}}{2 \gamma_ {d _ {S} , d _ {A}}} \frac {N ^ {\min  \left\{d _ {S} , d _ {A} \right\}}}{\log N}, \quad P _ {S} \sim \frac {(\lambda - 1) ^ {2}}{2 (\min  \left\{d _ {S} , d _ {A} \right\} + 1) \gamma_ {d _ {S} , d _ {A}}} \frac {N ^ {\min  \left\{d _ {S} , d _ {A} \right\}}}{\log N}, \tag {15}
$$
where  $\gamma_{d_S,d_A}$  is a multiplicative factor defined as
$$
\gamma_ {d _ {S}, d _ {A}} = \left\{ \begin{array}{l l} (2 d _ {S} - 1)!! & , \text {i f} d _ {S} <   d _ {A} \\ (\lambda^ {2} + 1) (2 d _ {S} - 1)!! + 2 \lambda [ (d _ {S} - 1)!! ] ^ {2} \mathbf {1} \left\{d _ {S} \text {e v e n} \right\} & , \text {i f} d _ {S} = d _ {A} \\ \lambda^ {2} (2 d _ {A} - 1)!! & , \text {i f} d _ {S} > d _ {A}. \end{array} \right. \tag {16}
$$
In Figure 4B-C, we show that simulations match the theory curves well as  $N$  increases. We demonstrate theoretical and simulations results for the Exponential MixedNet in Appendix F.2.
4 Biologically-Plausible Implementation
Since biological neural networks must store sequence memories [2, 5-8], one naturally asks if these results can be generalized to biologically-plausible neural networks. A straightforward biological interpretation of the DenseNet is problematic, as a network with polynomial interaction function of degree  $d$  is equivalent to having a neural network with many-body synapses between  $d + 1$  neurons. This can be seen by expanding the Polynomial DenseNet in terms of a weight tensor of  $d + 1$  neurons:
$$
S _ {i} (t + 1) = \operatorname {s g n} \left[ \sum_ {j _ {1}, \dots , j _ {d}} J _ {i j _ {1} \dots j _ {d}} S _ {j _ {1}} (t) \dots S _ {j _ {d}} (t) \right], \quad J _ {i, j _ {1}, \dots , j _ {d}} = \frac {1}{N ^ {d}} \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu + 1} \xi_ {j _ {1}} ^ {\mu} \dots \xi_ {j _ {d}} ^ {\mu} \tag {17}
$$
This is biologically unrealistic as synaptic connections usually occur between two neurons [45]. In the case of the Exponential DenseNet, one can interpret its interaction function via a Taylor series expansion, implying synaptic connections between infinitely many neurons which is even more problematic. Similar difficulties arise in models with sum of terms with different powers [46].
To address this issue, we again take inspiration from earlier work in MHNs. Krotov and Hopfield [47] addressed this concern for symmetric MHNs by reformulating the network using two-body synapses, where the network was partitioned into a bipartite graph with visible and hidden neurons (see [48] for an extension of this idea to deeper networks). The visible neurons correspond to the neurons in our network dynamics,  $\mathbf{S}_j$ , while the hidden neurons correspond to the individual memories stored within the network. They are connected through a weight matrix. Since we are working with an asymmetric network, we modify their approach and define two sets of synaptic weights:  $W_{j\mu}$  connects visible neuron  $v_{j}$  to hidden neuron  $h_\mu$
$M_{\mu j}$  connects hidden neuron  $h_\mu$  to visible neuron  $v_j$ . This yields the same dynamics exhibited in Equation (2), absorbing the nonlinearity into the hidden neurons' dynamics.
Figure 5: Biologically-plausible implementation of DenseNet with two-body synapses.
For the DenseNet, we define the weights as  $W_{j\mu} \coloneqq \frac{1}{N}\xi_j^\mu$  and  $M_{\mu j} \coloneqq \xi_j^{\mu + 1}$ . For the MixedNet, we redefine the weight matrix  $M_{\mu j} = \xi_j^\mu + \lambda \xi_j^{\mu + 1}$ . The update rules for the neurons are as follows:
$$
h _ {\mu} (t) := f \left[ \sum_ {j} W _ {j \mu} v _ {j} (t) \right], \quad v _ {j} (t + 1) := \operatorname {s g n} \left[ \sum_ {\mu} M _ {\mu j} h _ {\mu} (t) \right] \tag {18}
$$
Note that these networks' transition and sequence capacities,  $P_T$  and  $P_S$ , now scale linearly with respect to the total number of neurons in this model,  $N$  visible neurons and  $P$  hidden neurons. However, the network capacity still scales nonlinearly with respect to the number of visible neurons.
Finally, we remark that this network is reminiscent of recent computational models for motor action selection and control via the cortico-basal ganglia-thalamo-cortical loop, in which the basal ganglia inhibits thalamic neurons that are bidirectionally connected to a recurrent cortical network [5, 49, 50]. This relates to our model as follows: the motor cortex (visible neurons) executes an action, each
thalamic unit (hidden neurons) encodes a motor motif, and the basal ganglia silences thalamic neurons (external network modulating context). In particular, the role of the basal ganglia in this network suggests a novel mechanism of context-dependent gating within Hopfield Networks [51]. Rather than modulating synapses or feature neurons in a network, one can directly inhibit (activate) memory neurons in order to decrease (increase) the likelihood of transitioning to the associated state. Similarly, thalamocortical loops have been found to be important to song generation in zebra finches [52]. Thus, the biological implementation of the DenseNet can provide insight into how biological agents reliably store and generate complex sequences.
5 Discussion and Future Directions
We introduced the DenseNet for the reliable storage and recall of long sequences of patterns, derived the scaling of its single-transition and full-sequence capacity, and verified these results in numerical simulation. We found that depending on the choice of nonlinear interaction function, the DenseNet could scale polynomially or exponentially. We tested the ability of these models to recall sequences of correlated patterns, by comparing the recall of a sequence of MovingMNIST images with different nonlinearities. As expected, the network's reconstruction capabilities increased with the nonlinearity power  $d$ , with perfect recall achieved by the exponential nonlinearity. To further increase the capacity, we introduced the generalized pseudoinverse rule and demonstrated in simulation its ability to maintain high capacity for highly correlated patterns. We also introduced and analyzed the MixedNet to maintain patterns within sequences for longer periods of time. Finally, we described a biologically plausible implementation of the models with connections to motor control.
There has recently been a renewed interest in storing sequences of memories. Steinberg and Sompolinsky [53] store sequences in Hopfield networks by using a vector-symbolic architecture to bind each pattern to its temporal order in the sequence, thus storing the entire sequence as a single attractor. However, this model suffers from the same capacity limitations as the Hopfield Network. Whittington et al. [54] suggest a mechanism to control sequence retrieval via an external controller, analogous to the role we ascribe to the basal ganglia for context-dependent gating. Herron et al. [55] investigate a mechanism for robust sequence recall within complex systems more broadly, reducing crosstalk by directly modulating interactions between neurons rather than the inputs into neurons. Tang et al. [56] propose a model for sequential recall akin to SeqNet with an implicit statistical whitening process. Karuvally et al. [57] introduce a model closely related to the biologically-plausible implementation of our MixedNet and analyze it in the setting of continuous-time dynamics, allowing for intralayer synapses within the hidden layer and different timescales between the hidden and feature layers.
While we have focused on a generalization of the fixed-point capacity for sequence memory, this is not the only notion of capacity one could consider. In other studies of MHNs, instead of considering stability as the probability of staying at a fixed point, researchers quantify the probability that the network will reach a fixed point within a single transition [31, 58, 59]. This approach allows one to quantify noise-robustness and the size of each memory's basin of attraction [35]. More broadly, one could consider other definitions of associative memory capacity not addressed here, including those that depend only on network architecture and not on the assumption of a particular learning rule [60, 61]. However, as compared to the relatively simple analysis that is possible for the fixed-point capacity of a Hopfield network using a Hebbian learning rule, analyzing these alternative notions of capacity in nonlinear networks can pose significant technical challenges [61-63].
In this work, we limited ourselves to theoretical analysis of discrete-time networks storing binary patterns. An important direction for future research would be to go beyond the Gaussian theory in order to develop accurate predictions of the Exponential DenseNet capacity. There are also many potential avenues for extending these models and methods to continuous-time networks, continuous-valued patterns, computing capacity for correlated patterns, testing different weight functions, and examining different network topologies. Finally, we hope to take inspiration from the recent resurgence of RNNs in long sequence modeling to use this model for real-world tasks [64, 65].
We thank Matthew Farrell, Shanshan Qin, and Sabarish Sainathan for useful discussions and comments on earlier versions of our manuscript. HC was supported by the GFSD Fellowship, Harvard GSAS Prize Fellowship, and Harvard James Mills Peirce Fellowship. JAZ-V and CP were supported by NSF Award DMS-2134157 and NSF CAREER Award IIS-2239780. CP received additional support from a Sloan Research Fellowship. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. The computations in this paper were run on the FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at Harvard University.
[1] D Kleinfeld and H Sompolinsky. Associative neural network model for the generation of temporal patterns. theory and application to central pattern generators. Biophysical Journal, 54 (6):1039-1051, 1988.
[2] Michael A. Long, Dezhe Z. Jin, and Michale S. Fee. Support for a synaptic chain model of neuronal sequence generation. Nature, 468(7322):394-399, Nov 2010. ISSN 1476-4687. doi:10.1038/nature09514. URL https://doi.org/10.1038/nature09514.
[3] Maxwell Gillett, Ulises Pereira, and Nicolas Brunel. Characteristics of sequential activity in networks with temporally asymmetric Hebbian learning. Proceedings of the National Academy of Sciences, 117(47):29948-29958, November 2020. doi:10.1073/pnas.1918674117.
[4] Stefano Recanatesi, Ulises Pereira-Obilinovic, Masayoshi Murakami, Zachary Mainen, and Luca Mazzucato. Metastable attractors explain the variable timing of stable behavioral action sequences. Neuron, 110(1):139–153, 2022.
[5] Luca Mazzucato. Neural mechanisms underlying the temporal organization of naturalistic animal behavior. eLife, 11:e76577, 2022.
[6] Edmund T Rolls and Patrick Mills. The generation of time in the hippocampal memory system. Cell Reports, 28(7):1649-1658, 2019.
[7] Alexander B. Wiltschko, Matthew J. Johnson, Giuliano Iurilli, Ralph E. Peterson, Jesse M. Katon, Stan L. Pashkovski, Victoria E. Abraira, Ryan P. Adams, and Sandeep Robert Datta. Mapping sub-second structure in mouse behavior. Neuron, 88(6):1121-1135, 2015. ISSN 0896-6273. doi:https://doi.org/10.1016/j.neuron.2015.11.031. URL https://www.sciencedirect.com/science/article/pii/S0896627315010375.
[8] Jeffrey E. Markowitz, Winthrop F. Gillis, Maya Jay, Jeffrey Wood, Ryley W. Harris, Robert Cieszkowski, Rebecca Scott, David Brann, Dorothy Koveal, Tomasz Kula, Caleb Weinreb, Mohammed Abdal Monium Osman, Sandra Romero Pinto, Naoshige Uchida, Scott W. Linderman, Bernardo L. Sabatini, and Sandeep Robert Datta. Spontaneous behaviour is structured by reinforcement without explicit reward. Nature, 614(7946):108-117, Feb 2023. ISSN 1476-4687. doi:10.1038/s41586-022-05611-2. URL https://doi.org/10.1038/s41586-022-05611-2.
[9] Cengiz Pehlevan, Farhan Ali, and Bence P Ölveczky. Flexibility in motor timing constrains the topology and dynamics of pattern generator circuits. Nature communications, 9(1):977, 2018. doi:https://doi.org/10.1038/s41467-018-03261-5.
[10] H. Sompolinsky and I. Kanter. Temporal Association in Asymmetric Neural Networks. Physical Review Letters, 57(22):2861-2864, December 1986. doi:10.1103/PhysRevLett.57.2861.
[11] Zijian Jiang, Ziming Chen, Tianqi Hou, and Haiping Huang. Spectrum of non-Hermitian deep-Hebbian neural networks. Physical Review Research, 5:013090, Feb 2023. doi:10.1103/PhysRevResearch.5.013090. URL https://link.aps.org/doi/10.1103/PhysRevResearch.5.013090.
[12] Ulises Pereira and Nicolas Brunel. Unsupervised learning of persistent and sequential activity. Frontiers in Computational Neuroscience, 13:97, 2020.
[64] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
[65] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349, 2023.
[66] Dmitry Krotov and John Hopfield. Dense associative memory is robust to adversarial inputs. Neural computation, 30(12):3151-3167, 2018.
[67] HH Chen, YC Lee, GZ Sun, HY Lee, Tom Maxwell, and C Lee Giles. High order correlation model for associative memory. In AIP Conference Proceedings, volume 151, pages 86-99. American Institute of Physics, 1986.
[68] Demetri Psaltis and Cheol Hoon Park. Nonlinear discriminant functions and associative memories. In AIP conference Proceedings, volume 151, pages 370-375. American Institute of Physics, 1986.
[69] Pierre Baldi and Santosh S Venkatesh. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters, 58(9):913, 1987.
[70] E Gardner. Multiconnected neural network models. Journal of Physics A: Mathematical and General, 20(11):3453, 1987.
[71] Laurence F Abbott and Yair Arian. Storage capacity of generalized networks. Physical review A, 36(10):5091, 1987.
[72] D Horn and M Usher. Capacities of multiconnected memory models. Journal de Physique, 49 (3):389-395, 1988.
[73] Elena Agliari, Alberto Fachechi, and Chiara Marullo. Nonlinear PDEs approach to statistical mechanics of dense associative memories. Journal of Mathematical Physics, 63(10):103304, 2022. doi:10.1063/5.0095411. URL https://doi.org/10.1063/5.009541.
[74] Elena Agliari, Linda Albanese, Francesco Alemanno, Andrea Alessandrelli, Adriano Barra, Fosca Giannotti, Daniele Lotito, and Dino Pedreschi. Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning. arXiv, 2022. doi:10.48550/ARXIV.2211.14067. URL https://arxiv.org/abs/2211.14067.
[75] Linda Albanese, Francesco Alemanno, Andrea Alessandrelli, and Adriano Barra. Replica symmetry breaking in dense Hebbian neural networks. Journal of Statistical Physics, 189(2): 24, Sep 2022. ISSN 1572-9613. doi:10.1007/s10955-022-02966-8. URL https://doi.org/10.1007/s10955-022-02966-8.
[76] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 02 2013. ISBN 9780199535255. doi:10.1093/acprof:oso/9780199535255.001.0001. URL https://doi.org/10.1093/acprof:oso/9780199535255.001.0001.
[77] DLMF. NIST Digital Library of Mathematical Functions. http://dlmf.nist.gov/, Release 1.1.1 of 2021-03-15, 2021. URL http://dlmf.nist.gov/. F. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds.
[78] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online], 2, 2010. URL http://yann.lecun.com/exdb/mnist.
A Review of Modern Hopfield Networks
Here we review the Hopfield network and its modern generalization as an auto-associative memory model. These ideas will be helpful for storing sequences in network dynamics.
A.1 The Hopfield Network