SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now consider the sequence capacity, which requires that  $NP\mathbb{P}[T_{MN}(\pmb{\xi}^1)_1 \neq \xi_1^2] \to 0$ . Then, making the same Ansatz for  $P$  as above, we have
$$
N P \mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{2 \sqrt {2 \pi}} \frac {(\lambda - 1) ^ {2}}{\gamma_ {d _ {S} , d _ {A}}} \frac {1}{(\alpha \log N) ^ {3 / 2}} N ^ {\min  \left\{d _ {S}, d _ {A} \right\} + 1 - \alpha / 2}, \tag {F.61}
$$
which tends to zero provided that  $\alpha \geq 2(\min\{d_S, d_A\} + 1)$ , yielding a predicted capacity of
$$
P _ {S} \sim \frac {(\lambda - 1) ^ {2}}{2 \left(\min  \left\{d _ {S} , d _ {A} \right\} + 1\right) \gamma_ {d _ {S}, d _ {A}}} \frac {N ^ {\min  \left\{d _ {S} , d _ {A} \right\}}}{\log N}. \tag {F.62}
$$
F.2 Exponential MixedNet
We now consider the Exponential MixedNet, with  $f_{S}(x) = f_{A}(x) = e^{(N - 1)(x - 1)}$ . With this, we have the first moments
$$
\begin{array}{l} \mathbb {E} [ f _ {S} (\Xi) ] = \mathbb {E} [ f _ {A} (\Xi) ] = \exp [ - (N - 1) ] \mathbb {E} \left[ \exp \left(\sum_ {j = 2} ^ {N} \xi_ {j}\right) \right] (F.63) \\ = \exp [ - (N - 1) ] \prod_ {j = 2} ^ {N} \mathbb {E} [ \exp (\xi_ {j}) ] (F.64) \\ = \left(\frac {\cosh (1)}{\exp (1)}\right) ^ {N - 1} (F.65) \\ \end{array}
$$
$$
\mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] = \mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right] = \left(\frac {\cosh (2)}{\exp (2)}\right) ^ {N - 1} = \frac {1}{\beta^ {N - 1}}, \tag {F.66}
$$
where as in Appendix C.2 we let
$$
\beta = \frac {\exp (2)}{\cosh (2)} \simeq 1. 9 6 4 0 3. \tag {F.67}
$$
$$
\frac {\exp (1)}{\cosh (1)} \simeq 1. 7 6 1 5 9, \tag {F.68}
$$
the conditional mean of the crosstalk is then
$$
\begin{array}{l} \mathbb {E} [ C \mid \xi_ {1} ^ {1} ] = \xi_ {1} ^ {1} \{1 + \lambda \mathbb {E} [ f _ {A} (\Xi) ] \} + \mathbb {E} [ f _ {S} (\Xi) ] (F.69) \\ = \xi_ {1} ^ {1} \left\{1 + \lambda \left(\frac {\cosh (1)}{\exp (1)}\right) ^ {N - 1} \right\} + \left(\frac {\cosh (1)}{\exp (1)}\right) ^ {N - 1} (F.70) \\ \sim \xi_ {1} ^ {1}, (F.71) \\ \end{array}
$$
where the corrections are exponentially small in an absolute sense. The leading part of the conditional variance of the crosstalk is
$$
\begin{array}{l} \operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] \sim P \left\{\mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] + 2 \lambda \mathbb {E} \left[ f _ {S} (\Xi) \right] \mathbb {E} \left[ f _ {A} (\Xi) \right] + \lambda^ {2} \mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right] \right\} (F.72) \\ \sim \frac {P}{\beta^ {N - 1}} \left\{1 + 2 \lambda \left(\frac {\cosh (1) ^ {2}}{\cosh (2)}\right) ^ {N - 1} + \lambda^ {2} \right\} (F.73) \\ \sim \frac {P}{\beta^ {N - 1}} \left(1 + \lambda^ {2}\right), (F.74) \\ \end{array}
$$
where we note that
$$
\frac {\cosh (1) ^ {2}}{\cosh (2)} \simeq 0. 6 3 2 9 0 1 \tag {F.75}
$$
hence the other contribution is exponentially suppressed in a relative sense.
$$
\mathbb {E} \left[ C \mid \xi_ {1} ^ {1} \right] \sim \xi_ {1} ^ {1} \tag {F.76}
$$
$$
\operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] \sim \frac {P}{\beta^ {N - 1}} \left(1 + \lambda^ {2}\right), \tag {F.77}
$$
hence from the general argument above we have
$$
\mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{2 \sqrt {2 \pi}} \frac {\sqrt {\left(1 + \lambda^ {2}\right)}}{\lambda - 1} \sqrt {\frac {P}{\beta^ {N - 1}}} \exp \left(- \frac {1}{2} \frac {(\lambda - 1) ^ {2}}{1 + \lambda^ {2}} \frac {\beta^ {N - 1}}{P}\right) \tag {F.78}
$$
for  $\lambda > 1$ . We now want to determine the capacity, starting with the single-transition capacity, for which we must have  $N\mathbb{P}[T_{MN}(\pmb{\xi}^1)_1 \neq \xi_1^2] \to 0$ . Recalling that we want to have  $\mathrm{var}[C | \xi_1^1] \to 0$ , we make the Ansatz
$$
P \sim \frac {1}{\alpha} \frac {(\lambda - 1) ^ {2}}{\lambda^ {2} + 1} \frac {\beta^ {N - 1}}{\log N} \tag {F.79}
$$
$$
N \mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{2 \sqrt {2 \pi \alpha \log N}} N ^ {1 - \alpha / 2}. \tag {F.80}
$$
This tends to zero if  $\alpha \geq 2$ , hence we conclude that the Gaussian theory predicts
$$
P _ {T} \sim \frac {1}{2} \frac {(\lambda - 1) ^ {2}}{\lambda^ {2} + 1} \frac {\beta^ {N - 1}}{\log N}. \tag {F.81}
$$
We now want to determine the sequence capacity, which requires that  $N P \mathbb{P}[T_{M N}(\pmb{\xi}^{1})_{1} \neq \xi_{1}^{2}] \to 0$ . Following our analysis of the Exponential DenseNet in Appendix C.2, we make the Ansatz that
$$
P \sim \frac {1}{\alpha} \frac {(\lambda - 1) ^ {2}}{\lambda^ {2} + 1} \frac {\beta^ {N - 1}}{N}, \tag {F.82}
$$
$$
N P \mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{2 \sqrt {2 \pi \alpha N}} \frac {1}{\alpha \beta} \frac {(\lambda - 1) ^ {2}}{\lambda^ {2} + 1} \exp \left[ \left(\log \beta - \frac {\alpha}{2}\right) N \right]. \tag {F.83}
$$
This tends to zero if  $\alpha \geq 2\log \beta$ , giving a predicted sequence capacity of
$$
P _ {S} = \frac {1}{2 \log \beta} \frac {(\lambda - 1) ^ {2}}{\lambda^ {2} + 1} \frac {\beta^ {N - 1}}{N}. \tag {F.84}
$$
Thus, for both definitions of capacity, the Gaussian theory's prediction of the capacity of the Exponential MixedNet is
$$
\frac {(\lambda - 1) ^ {2}}{\lambda^ {2} + 1} \tag {F.85}
$$
times the capacity of the Exponential DenseNet analyzed in Appendix C.2. This factor tends to zero from above as  $\lambda \downarrow 1$ , and gradually increases to 1 as  $\lambda \rightarrow \infty$ . Note that even without explicitly computing the excess kurtosis, we expect the intuition from the Exponential DenseNet to carry over to this setting. Indeed, the numerical simulations in Figure F.2 show that the transition capacity is well captured by the Gaussian theory while the sequence capacity shows significant deviation for small MixedNets.
Figure F.2: The capacities of Exponential MixedNets with  $\lambda = 2.5$  are plotted as a function of network size. (A) Transition capacity for the Exponential MixedNet, which closely matches theoretical prediction. The predicted capacity is shown by the solid line with dots, while square error bars show the results of numerical experiment. (B) Sequence capacity for the Exponential MixedNet, which diverges from theoretical prediction.
G Numerical implementation
Source code is available on GitHub at https://github.com/Pehlevan-Group/ LongSequenceHopfieldMemory. Experiments were run on the Harvard University FAS RC Cannon HPC cluster (https://www.rc.fas.harvard.edu/), using Nvidia A100 80GB GPUs. This limited the maximum number of patterns we could store in memory simultaneously to approximately  $10^{6}$  patterns, restricting our experimental evaluation of the Exponential DenseNet to approximately  $N = 25$  neurons.
G.1 Transition capacity
Numerical simulations for transition capacity were conducted as follows: For a given model of size  $N$ , start by initializing 100 sequences of Rademacher distributed patterns of length  $P_0$ , where  $P_0 = 2P^*$  is well above the model's predicted capacity  $P^*$ . This initialization for  $P_0$  was found through trial and error, where the method detects if you start below capacity. The model's update rule is applied in parallel across all patterns and across all sequences. If errors are made for any pattern in any sequence, 100 new random sequences are generated with smaller length  $P_1 = 0.99P_0$ . This is repeated, with the new sequence length being  $P_{t + 1} = 0.99P_t$ , until 100 sequences are generated for which no error is made in any transition. This entire process is repeated 20 times starting from  $P_0$  in order to obtain error bars.
G.2 Sequence capacity
Numerical simulations for sequence capacity were conducted in a similar fashion. For a given model of size  $N$ , start by initializing 100 sequences of Rademacher distributed patterns of length  $P_0$ , where  $P_0$  is well above the model's capacity. Starting from the first pattern of each sequence, the model's update rule is applied serially for each sequence. As soon as an error is obtained within any sequence, 100 new random sequences are generated with smaller length  $P_1 = 0.99P_0$ . This is repeated, with the new sequence length being  $P_{t + 1} = 0.99P_t$ , until 100 sequences are generated for which no error is made. This entire process is repeated 20 times starting from  $P_0$  in order to obtain error bars.
G.3 MovingMNIST
For the MovingMNIST experiments in Section 2.4, the images were pre-processed to have binarized pixel values. There were 10000 subsequences, each containing 2 handwritten digits from the MNIST dataset moving through each other across 20 images, that were concatenated to construct the entire sequence of 200000 images [78]. Then, different models were run from initialization and their output for different time steps was displayed in Figure 3.
S25
G.4 Generalized pseudoinverse rule
For numerical simulations of the generalized pseudoinverse rule in 2.5, the transition capacity of the Polynomial DenseNet was simulated in a similar method as described above. However, the Exponential DenseNet suffered from numerical instability when calculating the pseudoinverse of the overlap matrix, resulting in floating point error. Therefore, we showed results only for the Polynomial DenseNet.
S26