SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jerry Yao-Chieh Hu† Donglin Yang† Dennis Wu†
Chenwei Xu† Bo-Yu Chen‡ Han Liu†‡
$^{\dagger}$ Department of Computer Science, Northwestern University, Evanston, IL 60208 USA
$^{\ddagger}$ Department of Physics, National Taiwan University, Taipei 10617, Taiwan
$^{2}$ Department of Statistics and Data Science, Northwestern University, Evanston, IL 60208 USA
{jhu, dlyang, hibb, cxu}@u.northwestern.edu
b12202023@ntu.edu.tw, hanliu@northwestern.edu
We introduce the sparse modern Hopfield model as a sparse extension of the modern Hopfield model. Like its dense counterpart, the sparse modern Hopfield model equips a memory-retrieval dynamics whose one-step approximation corresponds to the sparse attention mechanism. Theoretically, our key contribution is a principled derivation of a closed-form sparse Hopfield energy using the convex conjugate of the sparse entropic regularizer. Building upon this, we derive the sparse memory retrieval dynamics from the sparse energy function and show its one-step approximation is equivalent to the sparse-structured attention. Importantly, we provide a sparsity-dependent memory retrieval error bound which is provably tighter than its dense analog. The conditions for the benefits of sparsity to arise are therefore identified and discussed. In addition, we show that the sparse modern Hopfield model maintains the robust theoretical properties of its dense counterpart, including rapid fixed point convergence and exponential memory capacity. Empirically, we use both synthetic and real-world datasets to demonstrate that the sparse Hopfield model outperforms its dense counterpart in many situations.
We address the computational challenges of modern Hopfield models by introducing a sparse Hopfield model. Our sparse continuous Hopfield model equips a memory-retrieval dynamics that aligns with the sparse-structured attention mechanism. By establishing a connection to sparse attention, the proposed model not only offers a theoretically-grounded energy-based model for associative memory but also enables robust representation learning and seamless integration with deep learning architectures. This approach serves as an initial attempt of pushing the correspondence between Hopfield models and attention mechanism [Ramsauer et al., 2021] toward sparse region, both theoretically and empirically, resulting in data-dependent sparsity for meaningful and robust pattern representations, and a focus on the most relevant information for each specific instance.
Hopfield models are classic associative memory models for both biological and artificial neural networks [Hopfield, 1982, 1984]. These models are designed to store and retrieve memory patterns. They achieve these by embedding the memories in the energy landscape of a physical system (e.g., the Ising model in [Hopfield, 1982, Peretto and Niez, 1986]; see Figure 3 for a visualization), where each memory corresponds to a local minimum. When a query is presented, the model initiates energy-minimizing retrieval dynamics at the query, which then navigate the energy landscape to find the nearest local minimum, effectively retrieving the memory most similar to the query.
Code is available at GitHub; future updates are on arXiv. [Version: November, 28, 2023]
<sup>1</sup>While this equivalence only holds when the retrieval dynamics is applied exactly once, as originally shown in [Ramsauer et al., 2021] and later emphasized in [Krotov and Hopfield, 2021], it allows us to view modern Hopfield models as generalized attentions with additional functionalities and hence opens new avenues for Hopfield-based architecture designs. See Appendix C for more discussions.
In the same vein, Ramsauer et al. [2021] propose the modern Hopfield model and integrate it into deep learning architectures via a strong connection with transformer attention, offering enhanced performance, theoretically guaranteed exponential memory capacity, and the ability to handle continuous patterns. In addition, the modern Hopfield models have found success in various applications, such as immunology [Widrich et al., 2020] and large language model [Fürst et al., 2022]. Apart from the elegant connection to attention, theoretical advantages and empirically successes, the modern Hopfield models have been shown to be computationally heavy and vulnerable against noisy queries [Millidge et al., 2022]. In particular, the dense output alignments of the retrieval dynamics in modern Hopfield models [Ramsauer et al., 2021] can be computationally inefficient, making models less interpretable and noise-sensitive by assigning probability mass to many implausible outputs (patterns/keys).
To combat above, incorporating sparsity is an essential and common strategy. While there is a vast body of work on sparsifying attention mechanisms [Tay et al., 2022, Beltagy et al., 2020, Qiu et al., 2019, Child et al., 2019, Peters et al., 2019, Martins and Astudillo, 2016], similar developments for the Hopfield models remain less explored. To bridge this gap, we present a sparse Hopfield model that corresponds to the sparsemax attention mechanism [Martins and Astudillo, 2016]. In this paper, we study the sparsification of the modern Hopfield model. The challenges are three-fold:
(C1) Non-Trivial Sparsification — Sparse Hopfield  $\leftrightarrow$  Sparse Attention: To enable the use of sparse Hopfield models as computational devices (DNN learning models) akin to [Ramsauer et al., 2021], it is essential to achieve non-trivial sparsifications that exhibit equivalence to specific sparse attention models. In other words, any meaningful sparsification should extend the established equivalence [Ramsauer et al., 2021] between modern Hopfield models and attention to encompass the sparse domain. While generalizing such equivalence is potentially impactful as it may lay the groundwork for future Hopfield-based methodologies, architecture designs and bio-computing systems (as in [Kozachkov et al., 2023]), the heuristic design of the modern Hopfield model poses great difficulty to developing desired sparse models.
(C2) Introducing Sparsity into Hopfield Models: Unlike attention mechanisms where sparsification is typically achieved either on the attention matrix (e.g., structured-sparsity [Tay et al., 2020, Child et al., 2019]) or on the element-wise normalization map (e.g., sparsity-inducing maps [Correia et al., 2019, Peters et al., 2019, Martins and Astudillo, 2016]), the sparsification of Hopfield models is applied to both the energy function and the memory-retrieval dynamics, where the latter monotonically decreases the Hopfield energy over time. Since attention mechanisms (transformers) are typically not equipped with such a dynamical description, introducing sparsity into Hopfield models while retaining the connection to attention is a less straightforward process.
(C3) Properties of the Sparse Hopfield Model: Further, it is unclear how the introduced sparsity may affect different aspects of the model, such as memory capacity, fixed point convergence, retrieval accuracy, and so on. Ideally, we are looking for sparsities that offer provable computational benefits, such as enhanced robustness and increased memory capacity, among others.
Challenges (C1) and (C2) are inherent in Hopfield model, and certain requirements on the design of energy function and retrieval dynamics are inevitable to obtain non-trivial sparse models. Hence, we suppose the sparsified models should satisfy some conditions and verify them accordingly. Concretely, a formulation for deriving desired sparse Hopfield energy via convex conjugation of entropic regularizers is proposed. Furthermore, by applying Danskin's theorem and convex-concave procedure [Yuille and Rangarajan, 2003, 2001] on the sparse Hopfield energy function, we obtain sparse retrieval dynamics linked to sparse attention. For (C3), the convergence of energy stationary points and retrieval dynamics fixed points are connected via Zangwill's method [Zangwill, 1969]. The sparse retrieval error bound is derived and used to determine the well-separation condition for successful memory storage and retrieval. Lastly, the fundamental limit of memory capacity is derived using the expected separation of random points on spheres [Cai and Jiang, 2012, Brauchart et al., 2018, Ramsauer et al., 2021].
In summary, this work handles sparsification of modern Hopfield models while linking them to sparse attention by addressing the following question:
Is it possible to develop a theoretically-grounded (non-trivial) sparse Hopfield model capable of storing information or learned prototypes throughout various layers of DNN models?
Contributions. We propose the Sparse Modern Hopfield Model. Our contributions are as follows:
- We propose a novel sparse Hopfield model whose retrieval dynamics corresponds to sparse max attention mechanism. It leads to sparse patterns by design, inheriting both noise robustness and potential computational efficiency from [Martins and Astudillo, 2016], compared to its dense counterparts. This work extends the theoretical understanding of the correspondence between artificial and biological neural networks to sparse region. In addition, the sparse Hopfield layer, a new deep learning component, is introduced with data-dependent sparsity.
- Theoretically, we establish provably advantages from sparsity and identify the conditions under which these benefits arise. We begin by deriving the closed-form sparse Hopfield energy from the convex conjugation of sparse entropic regularizer. Next, we demonstrate the correspondence between sparse Hopfield retrieval dynamics and sparsemax attention. In addition, we prove the fast convergence of the fixed points (also known as memory patterns, attractor states in literature) for the retrieval dynamics and establish the exponential (in pattern size) memory capacity lower bound with tighter retrieval error bound, compared with modern Hopfield models.
- Empirically, we conduct synthetic and realistic experiments to verify our theoretical results and proposed methodology. Specifically, the sparse Hopfield model outperforms the dense Hopfield model and machine learning baselines in sparse Multiple Instance Learning (MIL), time series prediction and neural machine translation problems. This is observed with both sparse synthetic and real-world datasets, where the baselines tend to fall short. Moreover, even in cases without data sparsity, our proposed model delivers performance on par with its dense counterpart.
To the best of our knowledge, we are the first to propose a sparse Hopfield model whose retrieval dynamics is equivalent to sparse attention mechanism with provably computational advantages. Methodologically, the proposed model complements existing Hopfield-based DNN architectures [Hoover et al., 2023, Paischer et al., 2022, Seidl et al., 2022, Fürst et al., 2022, Ramsauer et al., 2021] by introducing a sparse Hopfield layer into deep learning models.
Organization. In Section 2, the sparse Hopfield model is introduced. In Section 3, the memory capacity is discussed. In Section 4, experimental studies are conducted. In Section 5, concluding discussions are provided. Additionally, related works and limitations are discussed in Appendix C.
Notations. We write  $\langle \mathbf{a},\mathbf{b}\rangle \coloneqq \mathbf{a}^{\mathrm{T}}\mathbf{b}$  as the inner product for vectors  $\mathbf{a},\mathbf{b}\in \mathbb{R}^{d}$ . The index set  $\{1,\dots ,I\}$  is denoted by  $[I]$ , where  $I\in \mathbb{N}_{+}$ . The spectral norm is denoted by  $\| \cdot \|$ , which is equivalent to the  $l_{2}$ -norm when applied to a vector. Throughout this paper, we denote the memory patterns (keys) by  $\pmb {\xi}\in \mathbb{R}^{d}$  and the state/configuration/query pattern by  $\mathbf{x}\in \mathbb{R}^{d}$ , and  $\Xi \coloneqq (\pmb {\xi}_1,\dots ,\pmb {\xi}_M)\in \mathbb{R}^{d\times M}$  as shorthand for stored memory (key) patterns  $\{\pmb {\xi}_\mu \}_{\mu \in [M]}$ . Moreover, we set norm  $n\coloneqq \| \mathbf{x}\|$  be the norm of the query pattern, and  $m\coloneqq \operatorname {Max}_{\mu \in [M]}\| \pmb {\xi}_\mu \|$  be the largest norm of memory patterns. We also provide a nomenclature table (Table 3) in the appendix.
2 Sparse Hopfield Model
In this section, we introduce the sparse Hopfield energy from convex conjugate of entropic regularizer, and then the sparse retrieval dynamics. In this paper we only consider the Gini entropic regularizer corresponding to the sparsemax distribution [Martins and Astudillo, 2016].
Let  $\mathbf{x} \in \mathbb{R}^d$  represent the query pattern, and let  $\Xi := (\xi_1, \dots, \xi_M) \in \mathbb{R}^{d \times M}$  denote the memory patterns. The objective of the Hopfield models is to store the memory patterns  $\Xi$  and then retrieve a specific memory pattern  $\xi_\mu$  based on a given query  $\mathbf{x}$ . Consequently, any Hopfield model consists of two main components: an energy function  $\mathcal{H}(\mathbf{x})$ , encoding memories into its local minima, and a retrieval dynamics  $\mathcal{T}(\mathbf{x})$ , which retrieves a memory by iteratively minimizing  $\mathcal{H}(\mathbf{x})$  when initialized with a query. We provide a visualization of this procedure in Figure 3. The construction of the energy function  $\mathcal{H}(\mathbf{x})$  is straightforward. As emphasized in [Krotov and Hopfield, 2016], the memories can be easily encoded into  $\mathcal{H}(\mathbf{x})$  through the overlap-construction:  $\mathcal{H}(\mathbf{x}) = F(\Xi^\top \mathbf{x})$ , where  $F: \mathbb{R}^M \to \mathbb{R}$  is a smooth function. This ensures that the memories  $\{\xi_\mu\}_{\mu \in [M]}$  are located at the stationary points of  $\mathcal{H}(\mathbf{x})$ , since  $\nabla_{\mathbf{x}} F(\Xi^\top \mathbf{x})|_{\xi_\mu} = 0$  for all  $\mu \in [M]$ . Different choices of  $F$  lead to different Hopfield models, as demonstrated in [Krotov and Hopfield, 2016, Demircigil et al., 2017, Ramsauer et al., 2021, Krotov and Hopfield, 2021]. However, finding a corresponding
2Note that, the proposed model's sparsity falls under the category of sparsity-inducing normalization maps. Consequently, the forward pass still requires  $\mathcal{O}(n^2)$  space complexity. Here, "potential computational efficiency" refers that the computational efficiency can be enhanced if one employs efficient implementations that leverage sparsity, such as sort operations or median-finding algorithms, to circumvent unnecessary computations, see Appendix C and [Martins and Astudillo, 2016, Section 2] for more discussions.
retrieval dynamics,  $\mathcal{T}$ , for a given energy  $\mathcal{H}(\mathbf{x})$ , is generally more challenging. This is because  $\mathcal{T}$  needs to satisfy two conditions to ensure successful memory retrieval: (i) To ensure consistent retrieval, an appropriate  $\mathcal{T}$  should monotonically minimize  $\mathcal{H}(\mathbf{x})$  when iteratively applied. (ii) To ensure accurate retrieval, an appropriate  $\mathcal{T}$  should align its fixed points (the points where iterative application terminates) with the stationary points of  $\mathcal{H}(\mathbf{x})$ .
To this end, we introduce the sparse Hopfield model, providing a principled construction for  $\mathcal{H}$  and  $\mathcal{T}$ . This model not only fulfills the aforementioned desirable properties, but also enables more robust and faster memory retrieval compared to the modern Hopfield model [Ramsauer et al., 2021].
2.1 Sparse Hopfield Energy
Let  $\mathbf{x} \in \mathbb{R}^d$  be the query pattern, and  $\Xi := (\xi_1, \dots, \xi_M) \in \mathbb{R}^{d \times M}$  be the memory patterns. We introduce the sparse Hopfield energy as
$$
\mathcal {H} (\mathbf {x}) = - \Psi^ {\star} \left(\beta \boldsymbol {\Xi} ^ {\top} \mathbf {x}\right) + \frac {1}{2} \langle \mathbf {x}, \mathbf {x} \rangle , \tag {2.1}
$$
with  $\Psi^{\star}(\mathbf{z})\coloneqq \frac{1}{2}\| \mathbf{z}\|^2 -\frac{1}{2}\| \mathrm{Sparsemax}(\mathbf{z}) - \mathbf{z}\|^2 +\frac{1}{2}$ , where  $\mathrm{Sparsemax}(\cdot)$  is defined as follows. Let  $\mathbf{z},\mathbf{p}\in \mathbb{R}^{M}$ , and  $\Delta^M\coloneqq \{\mathbf{p}\in \mathbb{R}_+^M\mid \sum_\mu^M p_\mu = 1\}$  be the  $(M - 1)$ -dimensional unit simplex.
Definition 2.1 (Sparsemax in Variational Form [Martins and Astudillo, 2016], also see Remark F.1).
$$
\operatorname {S p a r s e m a x} (\mathbf {z}) := \underset {\mathbf {p} \in \Delta^ {M}} {\operatorname {A r g M i n}} \| \mathbf {p} - \mathbf {z} \| ^ {2} = \underset {\mathbf {p} \in \Delta^ {M}} {\operatorname {A r g M a x}} \left[ \mathbf {p} ^ {\top} \mathbf {z} - \Psi (\mathbf {p}) \right], \tag {2.2}
$$
where  $\Psi (\mathbf{p})\coloneqq -\frac{1}{2}\sum_{\nu}^{M}p_{\nu}(1 - p_{\nu})$  is the negative Gini entropy or Gini entropic regularizer.
Remark 2.1. Recall that, the variational form (2.2) is in fact general, that applies to various entropic regularizers, as discussed in [Peters et al., 2019, Wainwright et al., 2008]. The choice of  $\Psi$  determines the resulting sparse probability distribution. For instance, if we choose the Gibbs' entropic regularizer  $\Psi_{\mathrm{Gibbs}} = -\sum_{\nu}^{M}p_{\nu}\ln p_{\nu}$ , (2.2) reduces to the standard softmax distribution.
Overview of Theoretical Results. At first glance, the energy function (2.1) may seem peculiar. However, it indeed represents a non-trivial sparse Hopfield model with appealing properties, including:
(i) In response to challenge (C1) & (C2), as we shall see in Section 2.2, the energy (2.1) leads to a sparse retrieval dynamics that not only retrieves memory by monotonically decreasing (Lemma 2.1) to its stationary points (Lemma 2.2), but also associates with sparsemax attention through its single-step approximation (Remark 2.2);
(ii) In response to challenge (C3), as we shall see in Section 3, it indulges fast convergence of retrieval (Corollary 3.1.2), exponential-in-  $d$  memory capacity akin to modern Hopfield models (Lemma 3.1). Notably, it accomplishes these with a tighter retrieval error bound (Theorem 2.1).
We reveal each of these properties in the following sections.
2.2 Sparse Retrieval Dynamics and Connection to Sparse Attention
The optimization problem  $\operatorname{ArgMax}_{\mathbf{p} \in \Delta^M} \left[ \mathbf{p}^\top \mathbf{z} - \Psi(\mathbf{p}) \right]$  does not necessarily have a closed-form solution for arbitrary  $\Psi$ . However, a family of  $\Psi$  has been investigated in literature [Correia et al., 2019, Martins and Astudillo, 2016] with closed-form solutions derived, including the Sparsemax( $\cdot$ ).
Sparsemax in Closed-Form (Proposition 1 of [Martins and Astudillo, 2016]). Let  $\mathbf{z} \in \mathbb{R}^M$ . Denote  $[a]_+ := \operatorname{Max}\{0, a\}$ ,  $z_{(\nu)}$  the  $\nu$ 'th element in a sorted descending  $z$ -sequence  $\mathbf{z}_{\text{sorted}} := z_{(1)} \geq z_{(2)} \geq \ldots \geq z_{(M)}$ , and  $\kappa(\mathbf{z}) := \operatorname{Max}\left\{k \in [M] \mid 1 + kz_{(k)} > \sum_{\nu \leq k} z_{(\nu)}\right\}$ . The optimization problem(s) (2.2) has closed-form solution
$$
\operatorname {S p a r s e m a x} (\mathbf {z}) = \left[ \mathbf {z} - \tau (\mathbf {z}) \mathbf {1} _ {M} \right] _ {+}, \tag {2.3}
$$
where  $\tau : \mathbb{R}^M \to \mathbb{R}$  is the threshold function  $\tau(\mathbf{z}) = \left[\left(\sum_{\nu \leq \kappa(\mathbf{z})} z_{(\nu)}\right) - 1\right] / \kappa(\mathbf{z})$ , satisfying  $\sum_{\mu=1}^{M} [z_{\mu} - \tau(\mathbf{z})]_{+} = 1$  for all  $\mathbf{z}$ . Notably,  $\kappa(\mathbf{z}) = |S(\mathbf{z})|$  where  $S(\mathbf{z}) = \{\mu \in [M] \mid \operatorname{Sparsemax}_{\mu}(\mathbf{z}) > 0\}$  is the support set of  $\operatorname{Sparsemax}(\mathbf{z})$ .
In this case, we present the following theorem to derive the convex conjugate of  $\Psi$  in closed-form:
Theorem 2.1 (Convex Conjugate of Negative Gini Entropy). Let  $F(\mathbf{p}) \coloneqq \langle \mathbf{p}, \mathbf{z} \rangle - \Psi(\mathbf{p})$  with  $\Psi$  being the negative Gini entropy,  $\Psi(\mathbf{p}) = \frac{1}{2} \| \mathbf{p} \|^2 - \frac{1}{2}$ . The convex conjugate of  $\Psi(\mathbf{p})$  is
$$
\Psi^ {\star} (\mathbf {z}) := \underset {\mathbf {p} \in \Delta^ {M}} {\operatorname {M a x}} F (\mathbf {p}, \mathbf {z}) = \frac {1}{2} \| \mathbf {z} \| ^ {2} - \frac {1}{2} \| \mathbf {p} ^ {\star} - \mathbf {z} \| ^ {2} + \frac {1}{2}, \tag {2.4}
$$
where  $\mathbf{p}^{\star} = \mathrm{S}$  arsemax(z) is given by (2.3).
Corollary 2.1.1. By Danskin's Theorem,  $\nabla \Psi^{\star}(\mathbf{z}) = \operatorname{ArgMax}_{\mathbf{p} \in \Delta^{M}} F(\mathbf{p}, \mathbf{z}) = \operatorname{Sparsemax}(\mathbf{z})$ .
Proof. A detailed proof is shown in Appendix E.1.
Theorem 2.1 and Corollary 2.1.1 not only provide the intuition behind the sparse Hopfield energy (2.1) — the memory patterns are stored in local minima aligned with the overlap-function constructions (i.e.  $\left\| \boldsymbol{\Xi}^{\mathrm{T}}\mathbf{x}\right\|^{2} = \sum_{\mu = 1}^{M}\langle \boldsymbol{\xi}_{\mu},\mathbf{x}\rangle^{2})$  in [Ramsauer et al., 2021, Demircigil et al., 2017, Krotov and Hopfield, 2016] — but also prepare us for the following corresponding sparse retrieval dynamics.
Lemma 2.1 (Sparse Retrieval Dynamics). Let  $t$  be the iteration number. The energy (2.1) can be monotonically decreased by the following sparse retrieval dynamics over  $t$ :
$$
\left. \mathcal {T} \left(\mathbf {x} _ {t}\right) := \nabla_ {\mathbf {x}} \Psi (\beta \boldsymbol {\Xi} ^ {\top} \mathbf {x}) \right| _ {\mathbf {x} _ {t}} = \boldsymbol {\Xi} \operatorname {S p a r s e m a x} (\beta \boldsymbol {\Xi} ^ {\top} \mathbf {x} _ {t}) = \mathbf {x} _ {t + 1}. \tag {2.5}
$$
Proof Sketch. To show monotonic decreasing property, we first derive the sparse retrieval dynamics by utilizing Theorem 2.1, Corollary 2.1.1, along with the convex-concave procedure [Yuille and Rangarajan, 2003, 2001]. Then, we show the monotonicity of  $\mathcal{H}$  by constructing a iterative upper bound of  $\mathcal{H}$  which is convex in  $\mathbf{x}_{t+1}$  and thus, can be lowered iteratively by the convex-concave procedure. A detailed proof is shown in the Appendix E.2.
Remark 2.2. Similar to [Ramsauer et al., 2021], (2.5) is equivalent to sparsemax attention [Martins and Astudillo, 2016] when the  $\mathcal{T}$  is applied only once, see Appendix D for more details. Importantly,  $\beta$  acts as a scaling factor for the energy function, often referred to as the "inverse temperature". It influences the sharpness of energy landscape Equation (2.1), thereby controlling the dynamics. High  $\beta$  values, corresponding to low temperatures, encourage that the basins of attraction for individual memory patterns remain distinct, leading to easier retrieval.
Notably, since  $\left\| \boldsymbol{\Xi}^{\mathrm{T}}\mathbf{x}\right\|^{2} = \sum_{\mu = 1}^{M}\langle \pmb{\xi}_{\mu},\mathbf{x}\rangle^{2}$ , (2.5) implies that the local optimum of  $\mathcal{H}$  are located near the patterns  $\pmb{\xi}_{\mu}$ . Different from previous studies on binary Hopfield models [Demircigil et al., 2017, Krotov and Hopfield, 2016], for continuous patterns, we adopt the relaxed definition from [Ramsauer et al., 2021] to rigorously analyze the memory retrieval, and the subsequent lemma.
Definition 2.2 (Stored and Retrieved). Assuming that every pattern  $\pmb{\xi}_{\mu}$  surrounded by a sphere  $S_{\mu}$  with finite radius  $R\coloneqq \frac{1}{2}\mathrm{Min}_{\mu \neq \nu ;\mu ,\nu \in [M]}\| \pmb {\xi}_\mu -\pmb {\xi}_\nu \|$  we say  $\pmb{\xi}_{\mu}$  is stored if there exists a generalized fixed point of  $\mathcal{T}$ ,  $\mathbf{x}_{\mu}^{\star}\in S_{\mu}$ , to which all limit points  $\mathbf{x}\in S_{\mu}$  converge to, and  $S_{\mu}\cap S_{\nu} = \emptyset$  for  $\mu \neq \nu$ . We say  $\pmb{\xi}_{\mu}$  is  $\epsilon$ -retrieved by  $\mathcal{T}$  with  $\mathbf{x}$  for an error  $^a\epsilon$ , if  $\| T(\mathbf{x}) - \pmb{\xi}_{\mu}\| \leq \epsilon$ .
aThe retrieval error has a naive bound  $\epsilon \coloneqq \mathrm{Max}\left\{\| \mathbf{x} - \pmb {\xi}_\mu \| ,\| \pmb {\xi}_\mu -\mathbf{x}_\mu^\star \| \right\}$  by interpolating from  $\mathbf{x}$  to  $\pmb{\xi}_{\mu}$