SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we analyze Mnemosyne's compact associative memory (CAM) from the theoretical point of view. We show that, as its regular non-compact counterpart, it is capable of storing patterns, but (as opposed to the former) in the implicit manner. We start by providing a concise introduction to associative memory models as "analytical" (rather than combinatorial) energy-based nearest-neighbor search systems. We then present our main result (Theorem 4.3) stating that "on average" CAMs can restore exponentially many (in space dimensionality) number of patterns, provided that they are spread "well enough". To the best of our knowledge, this is the first result of this type.
4.1 Regular exponential associative memory
As in several other papers providing a theoretical analysis of the associative memory, we consider feature vectors taken from the set  $\{-1, + 1\} ^N$ . We denote by  $\{\xi^{\mu}\}_{\mu = 1}^{M}$  the set of all the memory-vectors to be stored. For the given input  $\xi$ , in the regular exponential associative memory model, the energy of the system is defined as:
$$
E _ {\text {r e g}} \left(\xi ; \xi^ {1}, \dots \xi^ {M}\right) = - \sum_ {\mu = 1} ^ {M} \exp \left(\xi^ {\top} \xi^ {\mu}\right). \tag {6}
$$
The dynamical system defining the interactions with the associative memory and whose goal is to retrieve the relevant memory for the given input vector  $\sigma \in \{-1, + 1\} ^N$  (its nearest neighbor in  $\{\xi \}_{\mu = 1}^{M}$ ) has the following form:  $\sigma \rightarrow T_{i_1}(\sigma)\rightarrow T_{i_2}(T_{i_1}(\sigma))\rightarrow \ldots$  , for the initial point  $\sigma$  , where  $i_1,i_2,\ldots$  are chosen independently at random and  $T_{j}:\{-1, + 1\}^{N}\to \{-1, + 1\}^{N}$  only updates the jth entry of its input as follows (for  $\xi [j;x]$  denoting vector  $\xi$  , but with its jth dimension equal to  $x$  ..
$$
T _ {j} (\xi) [ j ] = \operatorname {s g n} \left[ E _ {\text {r e g}} \left(\xi [ j; - 1 ]; \xi^ {1}, \dots \xi^ {M}\right) - E _ {\text {r e g}} \left(\xi [ j; 1 ]; \xi^ {1}, \dots \xi^ {M}\right) \right] \tag {7}
$$
Thus at every step, a random dimension of the input vector is chosen and its value is flipped if that operation decreases the energy of the system.
Definition 4.1 (the capacity of the associative models). We say that the model described above stores memories  $\{\xi^{\mu}\}_{\mu = 1}^{M}$  if there exists  $\rho \in (0,\frac{1}{2})$  such that  $T_{j}(\widehat{\xi}^{\mu})[j] = \xi_{j}^{\mu}$  for any  $j$  and  $\widehat{\xi}^{\mu}$  taken from the Hamming ball  $\mathcal{B}(\xi^{\mu},\rho N)$  centered in  $\xi^{\mu}$  and of Hamming radius  $\rho N$ .
It was proven in [21] that if the memories are chosen uniformly and independently at random from  $\{-1, + 1\} ^N$ , then with probability approaching 1 as  $N\to \infty$  the model stores all the memories as long as the memory-set is not too large. Most importantly, the upper bound on the memories-set size is exponential in  $N$ .
Remark 4.2. Despite the exponential capacity of the model, all its memories need to be explicitly stored (to compute the energy-function) and compute time is proportional to their number. Thus for large number of memories, the space and time complexity makes the retrieval infeasible in practice.
4.2 Mnemosyne's Compact Associative Memory (CAM)
We denote by  $\omega_{1},\omega_{2},\ldots \omega_{r}$  samples chosen independently from the multivariate Gaussian distribution  $\mathcal{N}(0,\mathbf{I}_N)$  and define the energy of the system as follows (see: Sec. 3.3.1, A.1):
$$
E _ {\text {r a n d}} \left(\xi ; \xi^ {1}, \dots \xi^ {M}\right) = \phi_ {F + +} (\xi) ^ {\top} \mathbf {M} \left(\xi^ {1}, \dots , \xi^ {M}\right), \tag {8}
$$
where  $\mathbf{M}$  is given as:  $\mathbf{M}(\xi^1, \dots, \xi^M) = -\sum_{\mu=1}^{M} \phi_{F++}(\xi^\mu)$ . We refer to the number of random projection vectors  $\omega$  as the number of random features (RFs).
Equipped with this new energy function, we define the corresponding dynamical system in the same way as for the regular associative memory model. Calculating energy  $E_{\mathrm{rand}}$  can be now
done efficiently in time  $O(r)$ , once vector  $\mathbf{M}$  is computed (thus independently from the number of implicitly stored memories). In the online/streaming setting, when the memories come one by one, updating vector  $\mathbf{M}$  can be done in time  $O(Nr)$  per query.
4.3 The capacity of Mnemosyne's memory
We are ready to present our main theoretical result. We assume the setting from Sec. 4.2. The extended version of this result, providing in addition concentration results, is given in Sec. A.2.
Theorem 4.3 (storage of compact associative memories). Denote by  $\xi^1, \dots, \xi^M \in \{-1, +1\}^N$  the memory-vectors. Assume that the Hamming distance between any two memory-vectors is at least  $\tau N$  for some  $\tau > 0$ . Take some  $0 < \rho < \frac{\tau}{2}$ . Then the following is true for any memory-vector  $\xi^l$  for  $l = 1, \dots, \mu$  and any input  $\widehat{\xi}^l \in \mathcal{B}(\xi^l, \rho N)$  as long as  $M \leq \exp(2N(\tau - 2\rho)) \frac{1 - e^{-2}}{2e^2}$ : the expected change of the energy of the compact associative memory system  $\Delta(E_{\mathrm{rand}})$  associated with flipping the value of the dimension of  $\widehat{\xi}^l$  is positive if that operation increases the distance from its close neighbor  $\xi^l$  and is negative otherwise.
Remark 4.4. Theorem 4.3 says that the expected value of the change of the energy of the system has the correct sign even if the number of stored patterns is exponential in their dimensionality, provided that the patterns are well separated. By the analogous argument as the one from the proof of Theorem 4.3 in Sec. A.2, a similar statement for the method applying other RF-mechanism for softmax-kernel estimation can be derived. However  $\phi_{F++}$  provides the smallest variance among all competitors as the most accurate currently known mechanism for the unbiased softmax-kernel approximation.
This section is organized as follows. Sec. 5.1 is a warm-up, where we provide initial comparison of Mnemosyne with several hand-designed optimizers and an LSTM-based learned optimizer baseline, on the tasks of training smaller NN architectures. Our results show that Mnemosyne consistently outperforms other variants and that popular Adam optimizer [34] is the second best. Note also that Adam is an optimizer of choice for training large Transformer models. Therefore in the following sections, we focus on the detailed comparison of Mnemosyne with Adam for larger architectures.
In Sec. 5.2, we test the coordinate-wise Mnemosyne for ViT fine-tuning and soft prompt-tuning  $11\mathrm{B}+$  T5XXL models. Optimizer's memory scales linearly with the NN size for the coordinate-wise variant and Adam. However, depending on the size and number of temporal attention layers in the coordinate-wise Mnemosyne, the linear multiplicative constant can be prohibitively large. This restriction is alleviated by the tensor-wise variant. In Sec. 5.3, we present the results for the tensorwise Mnemosyne on BERT MLM pre-training and ViT fine-tuning. The tensor-wise Mnemosyne's memory state scales with the number of tensors instead of the number of model parameters. However, this variant requires meta-training with larger tensors for best generalization results since now each tensor serves as a single training example instead of each parameter which was the case for the coordinate-wise variant. To take advantage of the efficiency of the tensor-wise and the efficacy of the coordinate-wise variant, we present Super-Mnemosyne, in Sec. 5.4, that merges them.
All Mnemosyne variants are meta-trained on small scale MLP and VIT training tasks for a short horizon of 100 steps. More detailed description of the meta-training set-up is given in the Appendix (Sec: B) along with the details of the experiments in this section. Additional results including: (a) comparison of the CAM mechanism with regular attention (Sec. B.2), (b) studies over different RF-mechanisms (Sec. B.3), ablations over: (c) different discount factors and different number of random features for the CAM mechanism (Sec. B.5) and (d) different depths of the temporal modules of Mnemosyne (Sec. B.6) are also given in the Appendix.
We have finalized the design choices for Mnemosyne optimizer used throughout this section based on the extensive theoretical analysis in previous sections and various ablation studies detailed in the Appendix. In particular:
1. The choice of the random feature (RF) mechanism is motivated by strong theoretical guarantees for the capacity of the corresponding associative memory. Other RF mechanisms that do not admit our theoretical analysis clearly underperform in the ablation results.
2. Our default hyperbolic cosine RF mechanism is motivated by the provably reduced estimation variance as explained previously.
Figure 3: First three plots: Test loss curves for training ViTs with Mnemosyne on: MNIST, CIFAR10 and CIFAR100. Minimal feature engineering is applied. Last plot: Test loss curves for training an MLP on MNIST. Both Mnemosyne and LSTM are used within VeLO architecture using sophisticated feature engineering.
3. We choose hyperbolic cosine RFs as they are shown more robust than regular positive RFs in ablation.
4. Our use of exponential discount strategy (EDS) to smoothly discount the past is theoretically motivated previously. We show empirical verification of EDS in the appendix and choose the discount factor based on ablation results.
5.1 Warm-up: Mnemosyne vs other optimizers for smaller NN architectures
We start by comparing coordinate-wise Mnemosyne with standard optimizers: Adam [34], RMSProp [78], SGD [33] as well as popular learnable optimizers using LSTMs [2]. All optimizers were tested on the task of training Vision Transformer (ViT) [23] architectures not seen during meta-training. Considered ViT has 3 attention and MLP layers of size 16 and 2 heads. Loss curves for image classification with ViTs on different datasets (MNIST, CIFAR10, CIFAR100) are shown as first three plots in Fig. 3. Minimal feature engineering techniques were applied. In the last plot of Fig. 3, we inserted Mnemosyne into VeLO optimizer [48] which originally used LSTMs. This learnable optimizer applies sophisticated feature engineering to mitigate the problem of catastrophic forgetting of the LSTM-cells. Note that we did not replicate their large-scale meta-training set-up, only the optimizer architecture and features. Mnemosyne outperforms its counterparts in all these scenarios. In particular, it successfully trains attention-based architectures on unseen tasks.
Now we proceed with the comparison of Mnemosyne and Adam with different learning rates on larger Transformers. We emphasize that we do not conduct any hyperparameter tuning for Mnemosyne.
5.2 Coordinate-wise Mnemosyne results
ViT last-layer fine-tuning: We tested a coordinate-wise Mnemosyne with 2 temporal encoders on ViT fine-tuning [23]. Due to memory constraints, we only trained the last layer of ViT by freezing other parameters of the model. The results on three different datasets:Imagenet2012, places365 and caltech-birds-2011 for ViT-H(32) (36 layers, 16 heads,  $32 \times 32$  patch shape) are presented in Fig. 4. Additional results, including ablations with varying architecture sizes and other datasets, are shown in Appendix Sec: B.8. We conclude that Mnemosyne (without any hyperparameter tuning) matches the top-performing Adam variant. Note that Adam is very sensitive to the choice of the learning rate lr.
In Table 1 we compare coordinate-wise Mnemosyne with two other classes of learnable optimizers: a) Those using S4 memory units [27] based on state space machines as an alternative to LSTMs and Transformers b) The Lion family of learnable optimizers [14] using a different approach of symbolic programming. In the case of ViT-L architecture, we skip S4 due to poor performance on the previous tasks. Mnemosyne is clearly the best among all three variants. In contrast to Lion, Mnemosyne does not require any hyperparameter tuning.
Figure 4: Fine-tuning the last layer of ViT-H(32) on different datasets with coordinate-wise Mnemosyne.
Method	ViT-H (ps=14)	ViT-H (ps=8)	ViT-L (ps=32)
Mnemosyne	72.3	81.4	74.0
S4	7.5	35.0	-
Lion lr=1e-1	67.2	76.4	66.8
Lion lr=1e-2	68.3	80.0	70.0
Lion lr=2e-3	70.0	80.0	71.4
Lion lr=1e-3	72.3	81.2	71.4
Lion lr=1e-4	70.0	81.0	71.4
Table 1: Comparison of coordinate-wise Mnemosyne with Lion and S4-based learned optimizer. Accuracy after fine-tuning the last layer of different ViT architectures and different patch sizes (ps) onImagenet2012 dataset for 500 iterations.
ViT multi-layer fine-tuning: The light Mnemosyne variant with a single temporal encoder can scale up to fine-tune 2 Transformer layers along with the last layer. We show the result for fine-tuning ViT-B(16) on Cifar100 in Fig. 5. The non-Mnemosyne layers are trained using an untuned Adam and the Adam variants for comparison fine-tune the full model. We observe that full-model finetuning is a hard task where all but one Adam variant fail to reach the optimal accuracy. Here, even by training a part of the network with Mnemosyne, we achieve the best accuracy. Also note that the best Adam learning rate in this experiment  $(1e^{-4})$  was performing poorly subsequent results as well.
Soft prompt-tuning T5XXL: We tested coordinate-wise Mnemosyne for soft prompt-tuning 11B+ T5XXL Transformers [55] on the SuperGLUE task [38]. This method was introduced as a scalable alternative to fine-tuning pre-trained models for several downstream tasks, with learnable prompts injected into Transformer layers and modulating its behaviour. The trainable soft-prompt contains 12288 parameters. Mnemosyne outperforms all Adam variants (see: Fig. 6).
Figure 5: Fine-tuning ViT-B(16).
for the previous one. We see this in
Figure 6: Soft prompt-tuning T5XXL.
5.3 Tensor-wise Mnemosyne results
In this section, we evaluate the memory-efficient tensor-wise Mnemosyne.
BERT NLP Transformer pre-training: We showcase the ability of tensor-wise Mnemosyne to pre-train a BERT-base text Transformer [22] on the standard masked language modeling (MLM) task.
Coordinate-wise Mnemosyne was not applicable here due to the memory footprint and thus it became a practical test for the memory efficient tensor-wise variant. With that variant, we were able to train 86M parameters of Bert-base model thereby showcasing the scalability of tensor-wise Mnemosyne. We compare Mnemosyne with different variants of Adam in Fig 7. We see that Mnemosyne matches the performance of the best Adam variant even though it was never exposed to the MLM task during meta-training. Several Adam variants get stuck in a local optima, worse than that found using Mnemosyne.
Figure 7: MLM task with BERT.
ViT multi-layer fine-tuning: Now we show the performance
of tensor-wise Mnemosyne on fine-tuning ViTs. Although tensor-wise Mnemosyne can scale up to the full ViT model, here we freeze large tensors in the Transformer layers and fine-tune the rest. Training large tensors with tensor-wise Mnemosyne requires data and compute intensive large scale meta-training (to ensure that spatial encoders learn how to encode long sequences well). This will be the focus of the future work. For this experiment, we consider the ViT-H(32) model and three datasets:
Cifar100, places365, imagenet in Fig. 8. Mnemosyne shows equivalent or better performance as compared to the optimal Adam.
Figure 8: Fine-tuning multi-layer ViT-H(32) on different datasets with tensor-wise Mnemosyne.
Figure 9: Fine-tuning full ViT-B(16) on different datasets with Super-Mnemosyne.
5.4 Super-Mnemosyne: combining coordinate- and tensor-wise strategies
In this last set of ViT-experiments, we combine tensor-wise and coordinate-wise light Mnemosyne. Since applying the former on large tensors requires more intense meta-training which is out of the scope of this paper, we decided to optimize the largest tensors with the coordinate-wise and others with the tensor-wise Mnemosyne (to keep meta-training simple). This combination turned out to minimize total memory usage. As we see in Fig. 9, Mnemosyne outperforms optimal Adam variants.
6 Broader impacts & limitations
We believe that Mnemosyne opens a research on attention-based optimizers for general-purpose optimization. Our system should be used responsibly due to the potential for misuse, significant societal impact, and carbon footprint of Transformers [3, 10, 70]. In the future we want to analyze in more depth the impact on more complex meta-training strategies on the quality of learned optimizers.
We proposed a new class of learnable optimizers applying efficient spatio-temporal attention, called Mnemosyne. We show that they outperform their LSTM-based counterparts and can be successfully used to fine/soft prompt-tune and pre-train large Transformer models, matching optimal hard-coded variants without any hyper-parameter tuning, often producing top performing models.
A.1 Kernels & their linearizations for temporal encoders in Mnemosyne
We tested different transformations  $\phi$  and discovered that those leading to most accurate approximation of the softmax-kernel lead to most effective memory mechanisms for Mnemosyne's temporal encoders (see: Sec. 3.3). Our starting variant is the so-called  $FAVOR+$  mechanism from [18], given as follows for  $\Gamma(\mathbf{z}, r) \stackrel{\mathrm{def}}{=} \frac{1}{\sqrt{r}} \exp \left( -\frac{\|\mathbf{z}\|^2}{2} \right)$  and  $\omega_1, \ldots, \omega_r \sim \mathcal{N}(0, \mathbf{I}_N)$ :
$$
\phi_ {F +} (\mathbf {z}) = \Gamma (\mathbf {z}, r) \left(\exp \left(\omega_ {1} ^ {\top} \mathbf {z}\right), \dots , \exp \left(\omega_ {r} ^ {\top} \mathbf {z}\right)\right) ^ {\top} \tag {9}
$$
Random vectors  $\omega_{1},\dots,\omega_{r}$  form a block-orthogonal ensemble (see: [18]). We applied also its improvement relying on the so-called hyperbolic cosine random features, where  $\prod$  is the concatenation operator:
$$
\phi_ {H F +} (\mathbf {z}) = \Gamma (\mathbf {z}, r) \prod_ {i = 1} ^ {\frac {r}{2}} \left(\exp \left(\omega_ {i} ^ {\top} \mathbf {z}\right), \exp \left(- \omega_ {i} ^ {\top} \mathbf {z}\right)\right) ^ {\top} \tag {10}
$$
Both randomized transformations provide unbiased estimation of the softmax-kernel, yet the latter one (that can be cast as modified  $\phi_{F+}$  via the antithetic Monte Carlo trick) has provably lower approximation variance.
A.1.1 The curious case of linearization with bounded features
The last variant for the efficient estimation of the softmax-kernel we applied, is a very recent mechanism  $FAVOR++$  from [44], given as:
$$
\phi_ {F + +} (\mathbf {z}) = \frac {D}{\sqrt {r}} \prod_ {i = 1} ^ {r} \exp \bigl (- \widehat {A} \| \omega_ {i} \| _ {2} ^ {2} + B \omega_ {i} ^ {\top} \mathbf {z} + C \| \mathbf {z} \| ^ {2} \bigr) ^ {\top},
$$
where we have:  $\widehat{A} = -A$ ,  $B = \sqrt{1 + 4\widehat{A}}$ ,  $C = -\frac{1}{2}$ ,  $D = (1 + 4\widehat{A})^{\frac{N}{4}}$ ,  $A = 1 - \frac{1}{\rho}$  and  $\rho \in (0,1)$  is a free parameter. As opposed to the previous variants, mechanism  $\phi_{F++}(\mathbf{z})$  provides an estimation via bounded random variables (since  $\widehat{A} > 0$ ), leading to stronger concentration results (beyond second moment) and still unbiased approximation.
The optimal choice of  $\rho$  depends on the kernel inputs. The formula for  $\rho$  optimizing the variance of the kernel matrix estimation  $\mathcal{K} = [\mathrm{K}(\mathbf{q}^i,\mathbf{k}^j)]_{i,j = 1,\dots ,M}$  induced by the softmax-kernel K (in the bi-directional case) is not tractable. However choosing  $\rho$  by optimizing certain derivative of the variance-objective was showed to work well in several applications [44]:
$$
\rho^ {*} = \frac {\sqrt {(2 \gamma + N) ^ {2} + 8 N \gamma} - 2 \gamma - N}{4 \gamma} \tag {11}
$$
for  $\gamma = \frac{1}{M^2}\sum_{i=1}^{M}\sum_{j=1}^{M}\|\mathbf{q}^i + \mathbf{k}^j\|^2$ . Since  $\gamma$  can be rewritten as:  $\gamma = \frac{1}{M^2}(\sum_{i=1}^{M}\|\mathbf{q}^i\|_2^2 + \sum_{j=1}^{M}\|\mathbf{k}^j\|_2^2 + 2\mathbf{q}^\top \mathbf{k})$  for  $\mathbf{q} = \sum_{i=1}^{M}\mathbf{q}^i$  and  $\mathbf{k} = \sum_{j=1}^{M}\mathbf{k}^j$ , computing  $\rho^*$  in the bi-directional setting can be clearly done in time linear in  $M$  as a one-time procedure. Then the computation of  $\mathbf{h}_{\mathrm{Mne}}(M)$  follows. Compute-time per memory-vector remains  $O_M(1)$ .
However Mnemosyne's temporal encoder applied uni-directional attention. In the uni-directional case, we have:  $\gamma_{t} = \frac{1}{t}\sum_{j = 1}^{t}\| \mathbf{q}^{t} + \mathbf{k}^{j}\|^{2} = \frac{1}{t} (\| \mathbf{q}^{t}\|_{2}^{2} + \sum_{j = 1}^{t}\| \mathbf{k}^{j}\|_{2}^{2} + 2(\mathbf{q}^{t})^{\top}\mathbf{k}(t))$ , where  $\mathbf{k}(t) = \sum_{j = 1}^{t}\mathbf{k}^{j}$  for  $t = 1,\dots,M$ . Instead of one  $\gamma$ , we now have  $M$  values  $\gamma_{t}$  since not all memories are known at once. We can still achieve  $O_{1}(M)$  compute-time per  $\gamma_{t}$ , repeating the trick from the bi-directional case, but that would need to be followed by the re-computation of  $\phi (\mathbf{k}^{\mu})$  (with new  $\rho$ -parameter) for  $\mu = 1,\ldots,t$  which of course is not possible since vectors  $\{\mathbf{k}\}_{\mu = 1}^{t}$  are not explicitly stored (and for a good reason - computational benefits), see: Eq. 3.
Thickening Mnemosyne's memory: To obtain efficient uni-directional Mnemosyne's memory cell also for the  $\phi_{F + + }$ -mechanism, we propose to "thicken" in that setting the hidden state from Eq. 3, replacing  $\mathbf{h}_{\mathrm{Mne}}(t) = (\mathbf{N}_t,\Psi_t)$  with  $\mathbf{H}_{\mathrm{Mne}}(t) = (\{\mathbf{N}_t^\rho \}_{\rho \in \Omega},\{\Psi_t^\rho \}_{\rho \in \Omega},\Sigma_t,\Lambda_t)$ , where we have:  $\Sigma_{t} = \sum_{j = 1}^{t}\mathbf{k}^{j}$ ,  $\Lambda_{t} = \sum_{j = 1}^{t}\| \mathbf{k}^{j}\|_{2}^{2}$  and furthermore:  $\mathbf{N}_t^\rho ,\Psi_t^\rho$  correspond to versions of  $\mathbf{N}_t$  and  $\Psi_t$
respectively, using parameter  $\rho$  to define mapping  $\phi$ . The set  $\Omega$  is obtained by discretizing interval  $(0,1)$  into a fixed number of chunks  $c$  (and effectively quantizes  $\rho \in (0,1)$ ). The strategy is now clear: when the new pattern comes, we first update the entire thickened state, and then compute  $\rho^{*}$ . We finalize by finding  $\rho \in \Omega$  closest to  $\rho^{*}$  to transform an input and using for that the "slice" of the hidden state corresponding to  $\rho$ . We see that all these operations can be made efficiently with only  $c$ -multiplicative term (independent from the number of patterns  $M$ ) in space and time complexity.
FAVOR++ mechanism, as FAVOR+, can also be adapted to its hyperbolic cosine variant. In practice FAVOR+ mechanism worked similarly to FAVOR++, yet the proper adaptation of the latter one was important, since (see: Sec. 4), this variant provides strongest theoretical guarantees for the capacity of the entire compact associative memory model.
A.2 The proof of the extended version of Theorem 4.3
We start by providing an extended version of Theorem 4.3, enriched with the exact formula of the variance of  $\Delta(E_{\mathrm{rand}})$ . We prove it below. We borrow the notation from Sec. A.1.
Theorem A.1 (storage of compact associative memories). Denote by  $\xi^1, \ldots, \xi^M \in \{-1, +1\}^N$  the memory-vectors. Assume that the Hamming distance between any two memory-vectors is at least  $\tau N$  for some  $\tau > 0$ . Take some  $0 < \rho < \frac{\tau}{2}$ . Then the following is true for any memory-vector  $\xi^l$  for  $l = 1, \ldots, \mu$  and any input  $\widehat{\xi}^l \in \mathcal{B}(\xi^l, \rho N)$  as long as  $M \leq \exp(2N(\tau - 2\rho)) \frac{1 - e^{-2}}{2e^2}$ : the expected change of the energy of the compact associative memory system  $\Delta(E_{\mathrm{rand}})$  associated with flipping the value of the dimension of  $\widehat{\xi}^l$  is positive if that operation increases the distance from its close neighbor  $\xi^l$  and is negative otherwise. Furthermore, the variance of  $\Delta(E_{\mathrm{rand}})$  is of the form: