SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 2.2 sets the threshold for a memory pattern  $\xi_{\mu}$  to be considered stored at a fixed point of  $\mathcal{T}$ ,  $\mathbf{x}_{\mu}^{\star}$ . However, this definition does not imply that the fixed points of  $\mathcal{T}$  are also stationary points of the energy function  $\mathcal{H}$ . In fact, monotonicity of (2.5) does not assure the existence of stationary points of energy  $\mathcal{H}$  [Sriperumbudur and Lanckriet, 2009]. To establish a well-defined Hopfield model, we need two types of convergence guarantees. The first is the convergence between  $\mathbf{x}_{\mu}^{\star}$  and  $\xi_{\mu}$ , which ensures that the retrieved memory is close to the stored memory. The second is the convergence of  $\mathcal{H}$  to its stationary points through the dynamics of  $\mathcal{T}$ , which ensures that the system reaches a state of minimal energy. The following lemma provides the convergence results for both.
Lemma 2.2 (Convergence of Retrieval Dynamics  $\mathcal{T}$ ). Suppose  $\mathcal{H}$  is given by (2.1) and  $\mathcal{T}(\mathbf{x})$  is given by (2.5). For any sequence  $\{\mathbf{x}_t\}_{t=0}^{\infty}$  defined by  $\mathbf{x}_{t'+1} = \mathcal{T}(\mathbf{x}_{t'})$ , all limit points of this sequence are stationary points if they are obtained by iteratively applying  $\mathcal{T}$  to  $\mathcal{H}$ .
Recall that a fixed point of  $\mathcal{T}$  with respect to  $\mathcal{H}$  is a point where  $\mathbf{x} = \mathcal{T}(\mathbf{x})$ , and a generalized fixed point is a point where  $\mathbf{x} \in \mathcal{T}(\mathbf{x})$ . For more details, refer to [Sriperumbudur and Lanckriet, 2009].
Proof Sketch. We verify and utilize Zangwill's global convergence theory [Zangwill, 1969] for iterative algorithms  $\mathcal{T}$ , to first show that all the limit points of  $\{\mathbf{x}_t\}_{t=0}^{\infty}$  are generalized fixed points and  $\lim_{t\to\infty}\mathcal{H}(\mathbf{x}_t)=\mathcal{H}(\mathbf{x}^\star)$ , where  $\mathbf{x}^\star$  are some generalized fixed points of  $\mathcal{T}$ . Subsequently, by [Sriperumbudur and Lanckriet, 2009, Lemma 5], we show that  $\{\mathbf{x}^\star\}$  are also stationary points of  $\operatorname{Min}_{\mathbf{x}}[\mathcal{H}]$ , and hence  $\mathcal{H}$  converges to local optimum. A detailed proof is shown in Appendix E.4. Intuitively, Lemma 2.2 indicates that the energy function converges to local optimum, i.e.  $\lim_{t\to\infty}\mathcal{H}(\mathbf{x}_t)\to\mathcal{H}(\mathbf{x}^\star)$ , where  $\mathbf{x}^\star$  are stationary points of  $\mathcal{H}$ . Consequently, it offers formal justifications for the retrieval dynamics (2.5) to retrieve stored memory patterns  $\{\pmb{\xi}_{\mu}\}_{\mu\in[M]}$ : for any query (initial point)  $\mathbf{x}$ ,  $\mathcal{T}$  monotonically and iteratively approaches stationary points of  $\mathcal{H}$ , where the memory patterns  $\{\pmb{\xi}_{\mu}\}_{\mu\in[M]}$  are stored. As for the retrieval error, we provide the following theorem stating that  $\mathcal{T}$  achieves a lower retrieval error compared to its dense counterpart.
Theorem 2.2 (Retrieval Error). Let  $\mathcal{T}_{\mathrm{Dense}}$  be the retrieval dynamics of the dense modern Hopfield model [Ramsauer et al., 2021]. It holds  $\| \mathcal{T}(\mathbf{x}) - \pmb{\xi}_{\mu}\| \leq \| \mathcal{T}_{\mathrm{Dense}}(\mathbf{x}) - \pmb{\xi}_{\mu}\|$  for all  $\mathbf{x}\in S_{\mu}$ . Moreover,
$$
\left\| \mathcal {T} (\mathbf {x}) - \boldsymbol {\xi} _ {\mu} \right\| \leq m + d ^ {1 / 2} m \beta \left[ \kappa \left(\underset {\nu \in [ M ]} {\operatorname {M a x}} \left\langle \boldsymbol {\xi} _ {\nu}, \mathbf {x} \right\rangle - \left[ \boldsymbol {\Xi} ^ {\mathrm {T}} \mathbf {x} \right] _ {(\kappa)}\right) + \frac {1}{\beta} \right], \tag {2.6}
$$
where  $\left[\Xi^{\mathsf{T}}\mathbf{x}\right]_{(\kappa)}$  is the  $\kappa$ th-largest element of  $\Xi^{\mathsf{T}}\mathbf{x} \in \mathbb{R}^{M}$  following the sparsemax definition (2.3).
Proof. A detailed proof is shown in Appendix E.3.
Interestingly, (2.6) is a sparsity dependent bound<sup>4</sup>. By denoting  $n \coloneqq \| \mathbf{x} \|$ , the second term on the RHS of (2.6) is dominated by the sparsity dimension  $\kappa$  as it can be expressed as  $\kappa \left(1 - \left[\Xi^{\top} \mathbf{x}\right]_{(\kappa)} / (nm)\right) \propto \alpha \kappa$  with a constant  $0 \leq \alpha \leq 2$ . When  $\Xi^{\top} \mathbf{x}$  is sparse (i.e.  $\kappa$  is small), the bound is tighter, vice versa.
Remark 2.3 (Faster Convergence). Computationally, Theorem 2.2 implies that  $\mathcal{T}$  requires fewer iterations to reach fixed points with the same amount of error tolerance compared to  $\mathcal{T}_{\mathrm{dense}}$ . Namely,  $\mathcal{T}$  retrieves stored memory patterns faster and therefore more efficiently, as evidenced in Figure 2.
Remark 2.4 (Noise-Robustness). Moreover, in cases of contaminated patterns with noise  $\eta$ , i.e.  $\widetilde{\mathbf{x}} = \mathbf{x} + \boldsymbol{\eta}$  (noise in query) or  $\widetilde{\pmb{\xi}}_{\mu} = \pmb{\xi}_{\mu} + \pmb{\eta}$  (noise in memory), the impact of noise  $\pmb{\eta}$  on the sparse retrieval error (2.6) is linear, while its effect on the dense retrieval error (2.7) is exponential. This suggests the robustness advantage of the sparse Hopfield model, as evidenced in Figure 1.
2.3 Sparse Hopfield Layers for Deep Learning
The sparse Hopfield model can serve as a versatile component for deep learning frameworks, given its continuity and differentiability with respect to parameters. Corresponding to three types of Hopfield Layers proposed in [Ramsauer et al., 2021], we introduce their sparse analogs: (1) SparseHopfield, (2) SparseHopfieldPooling, (3) SparseHopfieldLayer. Layer SparseHopfield has memory (stored or key) patterns  $\Xi$  and query (state) pattern  $\mathbf{x}$  as inputs, and associates these two sets of patterns via the sparse retrieval dynamics (2.5). This layer regards the transformer attention layer as its one-step approximation, while utilizing the sparse-max [Martins and Astudillo, 2016] on attention matrix. Layer SparseHopfieldPooling and Layer SparseHopfieldLayer are two variants of SparseHopfield, whose input patterns are memory patterns and query patterns from previous layers or external plugin, respectively. SparseHopfieldPooling, whose query patterns are learnable parameters, can be interpreted as performing a pooling operation over input memory patterns. SparseHopfieldLayer, by contrast, has learnable memory patterns that maps query patterns to hidden states with sparsemax activation. Thus it can substitute a fully connected layer within deep learning architectures. See (D.12) and the implementation Algorithm 1 in Appendix D, and [Ramsauer et al., 2021, Section 3] for more details of these associations. In Section 4, we apply these layers and compare them with their dense counterparts in [Ramsauer et al., 2021] and other baseline machine learning methods.
3 Fundamental Limits of Memory Capacity of Sparse Hopfield Models
How many patterns can be stored and reliably retrievable in the proposed model? We address this by decomposing it into two sub-questions and answering them separately:
(A) What is the condition for a pattern  $\xi_{\mu}$  considered well stored in  $\mathcal{H}$ , and correctly retrieved?
4 Notably,  $\| \mathcal{T}(\mathbf{x}) - \pmb{\xi}_{\mu}\|$  is also upper-bounded by a sparsity-independent but  $M,\beta$  -dependent bound
$\| \mathcal{T}(\mathbf{x}) - \pmb{\xi}_{\mu}\| \leq \| \mathcal{T}_{\mathrm{Dense}}(\mathbf{x}) - \pmb{\xi}_{\mu}\| \leq 2m(M - 1)\exp \left\{-\beta \left(\langle \pmb{\xi}_{\mu},\mathbf{x}\rangle -\max_{\nu \in [M],\nu \neq \mu}\langle \pmb{\xi}_{\mu},\pmb{\xi}_{\nu}\rangle\right)\right\} .$  (2.7)
(B) What is the number, in expectation, of the patterns satisfying such condition?
For (A), we first introduce the notion of separation of patterns following [Ramsauer et al., 2021],
Definition 3.1 (Separation of Patterns). The separation of a memory pattern  $\xi_{\mu}$  from all other memory patterns  $\Xi$  is defined as its minimal inner product difference to any other patterns:
$$
\Delta_ {\mu} := \operatorname * {M i n} _ {\nu , \nu \neq \mu} \left[ \langle \boldsymbol {\xi} _ {\mu}, \boldsymbol {\xi} _ {\mu} \rangle - \langle \boldsymbol {\xi} _ {\mu}, \boldsymbol {\xi} _ {\nu} \rangle \right] = \langle \boldsymbol {\xi} _ {\mu}, \boldsymbol {\xi} _ {\mu} \rangle - \operatorname * {M a x} _ {\nu , \nu \neq \mu} \left[ \langle \boldsymbol {\xi} _ {\mu}, \boldsymbol {\xi} _ {\nu} \rangle \right]. \tag {3.1}
$$
Similarly, the separation of  $\xi_{\mu}$  at a given  $\mathbf{x}$  from all memory patterns  $\Xi$  is given by
$$
\widetilde {\Delta} _ {\mu} := \operatorname * {M i n} _ {\nu , \nu \neq \mu} \left[ \langle \mathbf {x}, \boldsymbol {\xi} _ {\mu} \rangle - \langle \mathbf {x}, \boldsymbol {\xi} _ {\nu} \rangle \right]. \tag {3.2}
$$
and then the well-separation condition for a pattern being well-stored and retrieved.
Theorem 3.1 (Well-Separation Condition). Given the definition of stored and retrieved memories in Definition 2.2, suppose the memory patterns  $\{\pmb{\xi}_{\mu}\}_{\mu \in [M]}$  are located within the sphere  $S_{\mu} := \{\mathbf{x} \mid \| \mathbf{x} - \pmb{\xi}_{\mu} \| \leq R\}$ , where the radius  $R$  is finite and defined in Definition 2.2 for all  $\mu$ . Then, the retrieval dynamics  $\mathcal{T}$  maps the sphere  $S_{\mu}$  onto itself under the following conditions:
1. The initial query  $\mathbf{x}$  is located within the sphere  $S_{\mu}$ , i.e.,  $\mathbf{x} \in S_{\mu}$ .
2. The well-separation condition is satisfied, which is given by:
$$
\Delta_ {\mu} \geq m n + 2 m R - \left[ \boldsymbol {\Xi} ^ {\mathsf {T}} \mathbf {x} \right] _ {(\kappa)} - \frac {1}{\kappa} \left(\frac {R - m - m d ^ {1 / 2}}{m \beta d ^ {1 / 2}}\right).
$$
Corollary 3.1.1. Let  $\delta \coloneqq \| \mathcal{T}_{\mathrm{Dense}} - \pmb{\xi}_{\mu}\| -\| \mathcal{T} - \pmb{\xi}_{\mu}\|$ . The well-separation condition can be expressed as  $\Delta_{\mu}\geq \frac{1}{\beta}\ln \left(\frac{2(M - 1)m}{R + \delta}\right) + 2mR$ , which reduces to that of the dense Hopfield model when  $\delta = 0$ .
Proof Sketch. The proofs proceed by connecting  $\Delta_{\mu}$  with  $\| \mathcal{T}(\mathbf{x}) - \pmb{\xi}_{\mu}\|$ . To do so, we utilize Theorem 2.2 to incorporate the  $\Delta_{\mu}$ -dependent bound on the retrieval error of both sparse and dense Hopfield models [Ramsauer et al., 2021]. A detailed proof is shown in Appendix E.5.
Together with Lemma 2.2, the well-separated condition serves as the necessary condition for pattern  $\xi_{\mu}$  to be well-stored at the stationary points of  $\mathcal{H}$ , and can be retrieved with at most  $\epsilon = R$  by  $\mathcal{T}$ , as per Definition 2.2. We make the following three observations about the blessings from sparsity.
1. In general, to appreciate the blessings of sparsity, we rearrange the well-separation condition as
$$
\Delta_ {\mu} \geq 2 m R + \underbrace {\left(m n - \left[ \boldsymbol {\Xi} ^ {\mathrm {T}} \mathbf {x} \right] _ {(\kappa)}\right)} _ {:= \alpha n m \text {w i t h} 0 \leq \alpha \leq 2} - \frac {1}{\kappa} \left(\frac {R - m - m d ^ {1 / 2}}{m \beta d ^ {1 / 2}}\right), \tag {3.3}
$$
and observe the two competing terms,  $\alpha nm$  and  $(R - m - md^{1 / 2}) / (\kappa m\beta d^{1 / 2})$ . Sparsity proves advantageous when the latter term surpasses the former, i.e. the sparse well-separation condition is consistently lower than its dense counterpart. The condition under which sparsity benefits are more likely to emerge (i.e., when the well-separation condition is more readily satisfied) is thereby:
$$
\frac {1}{2} \operatorname * {M i n} _ {\mu , \nu \in [ M ]} \| \boldsymbol {\xi} _ {\mu} - \boldsymbol {\xi} _ {\nu} \| \geq m d ^ {1 / 2} (1 + \alpha \beta n m \kappa) + m, \quad \text {w i t h} 0 \leq \alpha \leq 2. \tag {3.4}
$$
Intuitively, the sparser  $\Xi^{\top}\mathbf{x}$  is, the easier it is for the above condition to be fulfilled.
2. Large  $M$  limit: For large  $M$ , the dense well-separation condition (Corollary 3.1.1) explodes while the sparse one (Theorem 3.1) saturates to the first three  $M$ -independent terms. This suggests that the hardness of distinguishing patterns can be tamed by the sparsity, preventing an increase of  $\Delta_{\mu}$  with  $M$  as observed in the dense Hopfield model. We numerically confirm this in Figure 1.
3.  $\beta \rightarrow \infty$  Limit: In the region of low temperature, where  $\beta \rightarrow \infty$  and hence all patterns can be error-free retrieved as per (2.7), we have  $\Delta_{\mu} \geq 2mR + \alpha nm$  with  $0 \leq \alpha \leq 2$ . Here, the second term on the RHS represents the sparsity level of  $\Xi^{\top}\mathbf{x}$ , i.e. a smaller  $\alpha$  indicates a higher degree of sparsity in  $\Xi^{\top}\mathbf{x}$ . Hence, the higher the sparsity, the easier it is to separate patterns.
For (B), equipped with Theorem 3.1 and Corollary 3.1.1, we provide a lower bound for the number of patterns being well-stored and can be at least  $R$ -retrieved in the next lemma<sup>5</sup>:
$^{5}$ Following the convention in memory capacity literature [Ramsauer et al., 2021, Demircigil et al., 2017, Krotov and Hopfield, 2016], we assume that all memory patterns  $\{\pmb{\xi}_{\mu}\}$  are sampled from a  $d$ -sphere of radius  $m$ .
Lemma 3.1 (Memory Capacity Lower Bound). Let  $1 - p$  be the probability of successfully storing and retrieving a pattern. The number of patterns randomly sampled from a sphere of radius  $m$  that the sparse Hopfield model can store and retrieve is lower-bounded by
$$
M \geq \sqrt {p} C ^ {\frac {d - 1}{4}}, \tag {3.5}
$$
where  $C$  is the solution to  $C = \frac{b}{W_0(\exp\{a + \ln b\})}$  with  $W_{0}(\cdot)$  being the principal branch of Lambert  $W$  function,  $a\coloneqq (4 / d - 1)\left(\ln \left[2m(\sqrt{p} -1) / (R + \delta)\right] + 1\right)$  and  $b\coloneqq 4m^2\beta /5(d - 1)$ . For sufficiently large  $\beta$ , the sparse Hopfield model exhibits a larger lower bound on the exponential memory capacity compared to its dense counterpart [Ramsauer et al., 2021]:  $M\geq M_{\mathrm{Dense}}$ .
Proof Sketch. Our proof is built on [Ramsauer et al., 2021]. The high-level idea is to utilize the separation of random patterns sampled from spheres [Cai and Jiang, 2012, Brauchart et al., 2018] and the asymptotic expansion of the Lambert  $W$  function [Corless et al., 1996]. Firstly, we link the well-separation condition to cosine similarity distance, creating an inequality for the probability of a pattern being well-stored and retrieved. Next, we identify and prove conditions for the exponential memory capacity  $M = \sqrt{p} C^{(d - 1) / 4}$  to hold. Finally, we analyze the scaling behaviors of  $C$  using its asymptotic expansion and show that  $M \geq M_{\mathrm{Dense}}$ . A detailed proof is shown in Appendix E.6.
Intuitively, the benefits of sparsity arises from the increased energy landscape separation provided by the sparse Hopfield energy function, which enables the separation of closely correlated patterns, resulting in a tighter well-separation condition for distinguishing such patterns and hence a larger lower bound on the memory capacity. Moreover, the sparse Hopfield model also enjoys the properties of fast convergence and exponentially suppressed retrieval error provided by the following corollary.
Corollary 3.1.2 (Fast Convergence and Exponentially Suppressed Retrieval Error). For any query  $\mathbf{x}$ ,  $\mathcal{T}$  approximately retrieves a memory pattern  $\xi_{\mu}$  with retrieval error  $\epsilon$  exponentially suppressed by  $\Delta_{\mu}:\| \mathcal{T}(\mathbf{x}) - \xi_{\mu}\| \leq 2m(M - 1)\exp \bigl \{-\beta \left(\Delta_{\mu} - 2m\operatorname {Max}\left[\| \mathbf{x} - \xi_{\mu}\| ,\| \mathbf{x} - \mathbf{x}_{\mu}^{\star}\| \right]\right)\bigr \}$ .
Proof. This results from Theorem 2.2, Lemma 2.2, and [Ramsauer et al., 2021, Theorem 4].
Corollary 3.1.2 suggests that, with a sufficient  $\Delta_{\mu}$ ,  $\mathcal{T}$  can approximately retrieve patterns after a single activation, allowing the integration of sparse Hopfield models into deep learning architectures similarly to [Hoover et al., 2023, Seidl et al., 2022, Fürst et al., 2022, Ramsauer et al., 2021].
4 Proof of Concept Experimental Studies
We demonstrate the validity of our theoretical results and method by testing them on various experimental settings with both synthetic and real-world datasets.
4.1 Experimental Validation of Theoretical Results
We conduct experiments to verify our theoretical findings, and report the results in Figure 1. For the memory capacity (the top row of Figure 1), we test the proposed sparse model on retrieving half-masked patterns comparing with the Dense (Softmax) and 10th order polynomial Hopfield models [Millidge et al., 2022, Krotov and Hopfield, 2016] on MNIST (high sparsity), Cifar10 (low sparsity) and ImageNet (low sparsity) datasets. For all Hopfield models, we set  $\beta = 1$ . A query is regarded as correctly retrieved if its cosine similarity error is below a set threshold. In addition, for the robustness against noisy queries (the bottom row of Figure 1), we inject Gaussian noises with varying variances  $(\sigma)$  into the images. Plotted are the means and standard deviations of 10 runs. The results show that the proposed sparse Hopfield model excels when memory patterns exhibit a high degree of sparsity and the signal-to-noise ratio in patterns is low, aligning with our theoretical results.
4.2 Multiple Instance Learning Tasks
Ramsauer et al. [2021] point out that the memory-enhanced Hopfield layers present a promising approach for Multiple Instance Learning (MIL) tasks. Multiple Instance Learning (MIL) [Ilse et al., 2018, Carbonneau et al., 2018] is a variation of supervised learning where the training set consists of labeled bags, each containing multiple instances. The goal of MIL is to predict the bag labels based on the instances they contain, which makes it particularly useful in scenarios where labeling individual instances is difficult or impractical, but bag-level labels are available. Examples of such scenarios include medical imaging (where a bag could be an image, instances could be patches of the
However, as pointed out in [Millidge et al., 2022], this is in fact not fair to compare modern Hopfield with  $\beta = 1$  with higher order polynomial Hopfield models.
Figure 1: Top: Memory Capacity measured by successful half-masked retrieval rates. Bottom: Memory Robustness measured by retrieving patterns with varying levels of Gaussian noise. For all Hopfield models, we set  $\beta = .01 / 0.1 / 0.1$  (for MNIST/CIFAR10/ImageNet) for better visualization. A query pattern is deemed correctly retrieved if its squared Euclidean distance is below a set threshold. For MNIST/CIFAR10/ImageNet datasets, we set the error thresholds to be  $10/20/20$  to cope with different sparse levels in data. Plotted are the means and standard deviations of 10 runs. The results suggest that the sparse Hopfield model excels when memory patterns exhibit a high degree of sparsity and the signal-to-noise ratio in patterns is low.
image, and the label could indicate the presence or absence of disease) and document classification (where a bag could be a document, instances could be the words or sentences in the document, and the label could indicate the topic or sentiment of the document). In this subsection, we implement our sparse Hopfield layers and applied them to MIL tasks on one synthetic and four real-world settings.
4.2.1 Synthetic Experiments
We use a synthetic MIL dataset, the bit pattern dataset, to demonstrate the effectiveness of the sparse Hopfield model. Each bag in this synthetic dataset contains a set of binary bit strings. The positive bag includes at least one of the positive bit patterns. We compare the performance of the SparseHopfield and SparseHopfieldPooling to their dense counterparts and vanilla attention [Vaswani et al., 2017]. We report the mean test accuracy of 10 runs. To demonstrate the effectiveness of sparse Hopfield model, we vary two hyperparameters of the bit pattern dataset corresponding to two perspectives: bag sparsity (sparsity in data) and bag size (number of memory patterns,  $M$ ). For bag sparsity, we fix the bag size as 200, and inject from 2 to 80 positive patterns in a positive bag, results in 1 to 40 percent of positive patterns in each positive bag. For bag size, we fix the number of positive pattern in a bag to be 1, and vary bag size from 20 to 300. We report results of SparseHopfieldPooling in Table 1, and implementation details in Appendix H.1.1. A more complete version of Table 1, including the results of Hopfield and attention, is in Appendix G. The sparse Hopfield model demonstrates a better performance across all sparsity and all bag sizes.
Table 1: Top (Bag Size): Accuracy comparison on bit pattern dataset for sparse and dense Hopfield model. We report the average accuracy over 10 runs. The results suggest that the sparse Hopfield model demonstrates a better performance when facing a bag size increase. Bottom (Bag Sparsity): Performance comparison on bit pattern dataset for sparse and dense Hopfield model with varying bag sparsity. We report the average accuracy over 10 runs. The results suggest that the sparse Hopfield model demonstrates a better performance across all sparsity.
Bag Size	20	50	100	150	200	300
Dense Hopfield Pooling	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00	76.44 ± 0.23	49.13 ± 0.01	52.88 ± 0.01
Sparse Hopfield Pooling	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00	99.76 ± 0.00	99.76 ± 0.00	99.76 ± 0.00
Bag Sparsity	1%	5%	10%	20%	40%
Dense Hopfield Pooling	49.20 ± 0.00	85.58 ± 0.10	100.0 ± 0.00	100.0 ± 0.00	99.68 ± 0.00
Sparse Hopfield Pooling	73.40 ± 0.06	99.68 ± 0.00	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00
Convergence Analysis. In Figure 2, we numerically examine the convergence of the sparse and dense Hopfield models, plotting their loss and accuracy for the bag size tasks in above on the bit pattern
dataset. We include multiple bag sizes to assess the effect of increasing memory patterns (i.e.  $M$ ) on the loss curve. The plotted are the loss and accuracy curves of SparseHopfieldPooling. We refer results of Hopfield and more details to Appendix G.3. The results (Figure 2) show that, sparse Hopfield model surpasses its dense counterpart in all bag sizes. Moreover, for the same bag size, the sparse Hopfield model always reaches the minimum validation loss faster than dense Hopfield model, validating our Theorem 2.2.
Sparsity Generalization. We also evaluate the models' generalization performance with shifting information sparsity, by training dense and sparse Hopfield models with a specific bag sparsity and testing them on the other. We report the results in Table 5 and refer more details to Appendix G.3.
Figure 2: Top: The training loss and accuracy curve of dense and sparse Hopfield models with different bag sizes. Bottom: The validation loss and accuracy curve of dense and sparse Hopfield models with different bag sizes. The plotted are the mean of 10 runs. The results indicate that the sparse Hopfield model converges faster than the dense model and also yields superior accuracy.
4.2.2 Real-World MIL Tasks
Next, we demonstrate that the proposed method achieves near-optimal performance on four realistic (non-sparse) MIL benchmark datasets: Elephant, Fox and Tiger for image annotation [Ilse et al., 2018], UCSB breast cancer classification [Kandemir et al., 2014]. We use Hopfield and SparseHopfield to construct a similar model architecture proposed in [Ramsauer et al., 2021] and a detailed description of this experiment as well as its training and evaluating process can be found in Appendix H.1.2. As shown in Table 2, both Sparse and Dense Hopfield achieve nearest results on Tiger, Elephant and UCSB datasets, despite the low sparsity in data. The sparse Hopfield model outperforms the dense Hopfield model by a small margin on three out of four datasets.