STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction
Authors: Dennis Wu
Jerry Yao-Chieh Hu
Weijian Li
Bo-Yu Chen
Han Liu
Venue: ICLR 2024 poster (2024)
Keywords: Time Series Prediction; Multivariate Time Series; Modern Hopfield Networks; Sparse Hopfield Model; Hopfield Layer; Attention Mechanism
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce STanHop-Net, a novel time series prediction model, combines a Hopfield-based block with external memory modules, enhancing learning, rapid response to sudden events, and superior theoretical guarantees and empirical performance.

Abstract: We present **STanHop-Net** (**S**parse **Tan**dem **Hop**field **Net**work) for multivariate time series prediction with memory-enhanced capabilities. At the heart of our approach is **STanHop**, a novel Hopfield-based neural network block, which sparsely learns and stores both temporal and cross-series representations in a data-dependent fashion. In essence, STanHop sequentially learns temporal representation and cross-series representation using two tandem sparse Hopfield layers. Additionally, STanHop incorporates two external memory modules: **Plug-and-Play** and **Tune-and-Play** for train-less and task-aware memory enhancements, respectively. They allow StanHop-Net to swiftly respond to sudden events. Methodologically, we construct the STanHop-Net by stacking STanHop blocks in a hierarchical fashion, enabling multi-resolution feature extraction with resolution-specific sparsity. Theoretically, we introduce a unified construction (**Generalized Sparse Modern Hopfield Model**) for both dense and sparse modern Hopfield models and show that it endows a tighter memory retrieval error compared to the dense counterpart without sacrificing memory capacity. Empirically, we validate the efficacy of STanHop-Net on many settings: time series prediction, fast test-time adaptation, and strongly correlated time series prediction.

Published as a conference paper at ICLR 2024
STANHOP: SPARSE TANDEM HOPFIELD MODEL FOR MEMORY-ENHANCER TIME SERIES PREDICTION
Dennis Wu* Jerry Yao-Chieh Hu Weijian Li
Bo-Yu Chen
Han Liu
t Department of Computer Science, Northwestern University, Evanston, II 60208, USA
4 Department of Physics, National Taiwan University, Taipei 10617, Taiwan
Department of Statisties and Data Science, Northwestem University, Evanston, 60208, USA
{hibb, jhu, weijiamli]e nowtwweeterm.eduu b122020238ntu. banliufforthhwsseeen.co
ABSTRACT
We present STanHop-Net (Sparse Tandem Hopfield Network) for multivari- ate time series prediction with memory-enhanese capabilities. At the heart of our approach is STanHop, a novel Hopfield- based neural network block, which sparsely learns and stores both temporal and cross-series representations in a data- dependent fashion. In essence, STanHop sequentially learn temporal represen- tation and cross-series representation using two tandem sparse Hopfield layers. In addition, STanHop incorporates two external memory modules: Plug-and- Play and Tune-and-Play for train-less and task-aware memorry-enhancemenns, respectively. They allow StanHop-Net to swiftly respond certain sudden events. Methodologically, we construct the STanHop-Net by stacking STanHop blocks in hierarchical fashion, enabling multi-resolution feature extraction with resolution-specific sparsity. Theoretically, we introduce unified construction (Generalized Sparse Modern Hopfield Model) for both dense and sparse mod- ern Hopfield models and show that it endows tighter memory retrieval error com- pared to the dense counterpart without sacrificing memory capacity. Empirically, we validate the efficacy of STanHop-Net many settings: time series prediction, fast test-time adaptation, and strongly correlated time series prediction.
INTRODUCTION
In this work, we aim enhance multivariate time series prediction by incorporating relevant addi- tional information specific to the inference task at hand. This problem holds practical importance due its wide range of real-world applications. On one hand, multivariate time series prediction itself poses a unique challenge given its multi-dimeniionall sequential structure and noise-sensitivity (Masini al., 2023; Reneau al., 2023; Nie al., 2022; Fawaz et al., 2019). A proficient model should robustly not only discern the correlations between series within each time step, but also grasp the intricate dynamics of each series over time. On the other hand, many real-world prediction tasks, one significant challenge with existing time series models is their slow responsiveness to sud- den or rare events. For instance, events like the 2008 financial crisis and the pandemic-induee ket turmoil in 2021 (Laborda and Olmo, 2021; Bond and Dow, 2021; Sevim et al., 2014; Bussiere and Fratzscher, 2006), or extreme climate changes in weather forecasting (Le et al., 2023; Sheshadri al., 2021) often lead to compromised model performance. To combat these challenges, STanHop-Net (Sparse Tandem Hopfield Network), a novel Hopfield-based deep learning model, for multivariate time series prediction, equipped with optional memory-enhanced capabilities.
mar-
present
Our motivation comes from the connection between associative memory models of human brain (specifically, the modern Hopfield models) and the attention mechanism (Hu et al., 2023; Ramsauer et al., 2020). Based on this link, we propose to enhance time series models with external information (c.g., real-time or relevant auxiliary data) via the memory retrieval mechanism of Hopfield models. In its core, we utilize and extend the becp-leaminng-compatee Ramsauer et al., 2020). Differing from typical transformer-basse architectures, these layers not only replace the attention mechanisms (Ramsauer et al., 2020; Widrich et al., 2020) but also serve as differentiable memory modules, enabling integration of external stimuli for enhanced predictions.
Hopfield layers (Hu et al., 2023;
"qqual contribution. Code is available GitHuh; full version and future updates arXiv.

---

Published as a conference paper at ICLR 2024
In this regard, first introduce a set of generalized sparse Hopfield layers, extension of the sparse modern Hopfield model (Hu et al., 2023). Based on these layers, we propose a structure termed the STanHop (Sparse Tandem Hopfield layers) block. In STanHop, there are two sequen- tially joined sub-blocks of generalized sparse Hopfield layers, hence tandem. This tandem design sparsely learn and store temporal and cross-series representations in sequential manner.
Furthermore, we introduce STanHop-Net (Sparse Tandem Hopfield Network) for time series, con- sisting of multiple layers of STanHHp blocks to cater for multi-resolution representation learning. To be more specific, rather than relying only on the input sequence for predictions, each stacked Stan- Hop block is capable of incorporating additional information through the Hopfield models' memory retrieval mechanism from a pre-specified external memory set. This capability facilitates the injec- tion of external memory at every resolution level when necessary. Consequently, STanHop-Net not only excels at making accurate predictions but also allows users to integrate additional information they consider valuable for their specific downstream inference tasks with minimal effort.
We provide visual overviews of STanHop-Net in Figure I and STanHop block in Figure 2.
Contributions. We summarize our contributions as follows:
Theoretically, we introduce a unified sparsity- awwre modern Hopfield model, termed the gen- eralized sparse Hopfield model. We show that it not only offer tighter memory retrieval error bound compared to the dense modern Hopfield model (Ramsauer et al., 2020), but also retains the robust theoretical properties of the dense model, such as fast fixed point convergence and exponential memory capacity. Moreover, it serves as a unified model that encompasses both the sparse (Hu al., 2023) and dense (Ramsauer et al., 2020) models its special cases.
Computationally, we show the one-step approximation of the retrieval dynamics of the gener- alized sparse Hopfield model is connected to sparse attention mechanisms, akin to (Hu al., 2023; Ramsauer et al., 2020). This connection allows us to introduce the GSH layers featuring learnable sparsity, for time series representation learning. As result, these layers achieve faster memory-retrieval convergence and greater nobse-robus compared to the dense model.
Methodologically, with GSH layer, we present STanHop (Sparse Tandem Hopfield layers) block, hierarchical tandem Hopfield model design to capture the intrinsic multi-resolutie structure both temporal and cross-series dimensions of time series with resolution-speciff sparsity each level. In addition, we introduce the idea of pseudo-label retrieval, and de- but two external memory plugin schemes Plug-and-Play and Tune-and-Play memory plugin modules - for memory memory-enhanced predictions.
Experimentally, we validate STanHop-Ne in multivariate time series predictions, considering both with and without the incorporation of external memory. When external memory isn't 'tt uti- lized, STanHop-Net consistently matches or surpasses many popular baselines, across diverse real-world datasets. When external memory is utilized, STanHop-Net demonstrates further per- formance boosts in many settings, benefiting from both proposed external memory schemes.
Notations. We write (a, b) = a l as the inner product for vectors a, b. The index set {1, }} denoted by [4], where / N4. The spectral norm is denoted by |I-I], which is equivalent to the l2 12-norm when applied to a vector. Throughout this paper, we denote the memory patterns (keys) by & E Rd and the state/configuraaioonnneyy pattern by x E Ed with n == ||x||. and = = (E..... Ear) E Rdx M as shorthand for stored memoery (key) patterns {Ew]ne[M-- More- over, we set m == Max ||&,,]| be the largest norm of memory patterns.
2 BACKGROUND: MODERN HOPFIELD MODELS
Let E Rd be the query pattern and = = (E1, .MM) RdxM be the M memory patterns.
Hopfield Models. Hopfield models are associative models that store a set of memory patterns 3 in such way that stored pattern E,, can be retrieved based on partially known or contaminated version, a query x. The models achieve this by embedding the memories E in the energy landscape E(x) of a physical system (e.g., the Ising model in (Hopfield, 1982) or its higher-order general- izations (Lee et al., 1986; Peretto and Niez, 1986; Newman, 1988)), where each memory & corre- sponds local minimum. When query x is introduced, the model initiates energy minimizing retrieval dynamics T at the query's location. This process then navigate the energy landscape to locate the nearest local minimum É,,, effectively retrieving the memory most similar to the query x.
2

---

Published as a conference paper at ICLR 2024
Constructing the energy function, Exx). is straightforward. As outlined in (Krotov and Hopfield, 2016), memories get encoded into Exx using the overlap-construction, E(x) = F(=Tx), where F:R. + R is smooth function. This ensures that the memories [ej] sit at the stationary points E(x), given xx x)|{ k... for all μ E [M]. The choice of F results in different Hop- field model types, as demonstrated in (Krotov and Hopfield, 2016; Demircigil et al., 2017; Ramsauer al., 2020; Krotov and Hopfield, 2020). However, determining suitable retrieval dynamics, T, for given energy Exx) is more challenging. For effective memory retrieval, T must:
(T1) Monotonically reduce E(x) when applied iteratively.
(T2) Ensure its fixed points coincide with the stationary points of Exx) for precise retrieval.
Modern Hopfield Models. Ramsauer al. (2020) propose the modern Hopfield model with spe-
cific set of E and T satisfying above requirements, and integrate it into deep learning architectures via its strong connection with attention mechanism, offering enhanced performance, and theoreti- cally guaranteed exponential memory capacity. Specifically, they introduce
Exx) = 1, _.xxxx x) +
2 (x. x) + Const., and (Demse = Softmax (SE) x) = x='e,, (2.1)
where = = ((5),)),... ...., (Ear, x)) IR,, lse (3, x) == log (En- (xp( / is the log-
sum eum-eexpoential for any given vector z E R M and O. Their analysis concludes that:
Tensse converges well (Ramsauer et al., 2020, Theorem 1,2) and can retrieve patterns accurately in just one step (Ramsauer et al., 2020, Theorem 4), i.e. (T1) and (T2) are satisfied.
The modern Hopfield model 12.1) possesses an exponential memory capacity in pattern size d (Ramsauer et al., 2020, Theorem 3).
Notably, the one-step approximation of Dense mirrors the attention mechanism in transformers, leading to a novel deep architecture design: the Hopfield layers.
In related vein, Hu et al. (2023) introduce principled approach constructing modern Hopfield models using the convex conjugate of the entropy regularizer. Unlike the original modern Hopfield model (Ramsauer al., 2020), the key insight of (Hu et al., 2023) is that the convex conjugate of various entropic regularizers can yield distributions with varying degrees of sparsity. Leveraging this understanding, introduce the generalized sparse Hopfield model in the next section.
GENERALIZED SPARSE HOPFIELD MODEL
In this section, we extend the entropic regularizer construction of the sparse modern Hopfield model (Hu al. 2023) by replacing the Gini entropic regularizer with the Tsallis a-eenrrpy (Tsallis, 1988),
Ep=1 (pr
M
pi)
a + 1, for a 2 1,
V (p(p)
ala-1
(3.1)
>p=1 Pau Imp.
1,
thereby introducing the generalized sparse Hopfield model. Subsequently, we verify the connection between the memory retrieval dynamics of the generalized sparse Hopfield model and attention mechanism. This leads to the Generalized Sparse Hopfield (GSH) layers for deep learning.
3.1 ENERGY FUNCTION, RETRIEVAL DYNAMICS AND FUNDAMENTAL LIMITS
Let z, P E R and A = {p E E ΣM Pe = 1} be the (M- - -imeennnia unit simplex. Energy Function. We introduce the generalized sparse Hopfield energy function':
1
(x, x) Const., with (z) = f a-EntMax(z), (3.2)
H(x)
(5 xx)
2
where a-EntMax()) ) M + A M is a finite-domain distribution map defined as follows.
Definition 3.1. The variational form of a-EntMax is defined by the optimization problem a- a-EntMax(2) 2) == ArgMax((pp) z) (P)),
(3.3)
where (.) is the Tsallis entropic regularizer given by (3.1). See Remark G.1 for a closed form. T (p) is the convex conjugate of the Tsallis entropic regularizer (p(p) (Definition C.1) and hence
Thii energy function (3.2) is equivalent up to an additive constant.
3

---

Published as a conference paper at ICLR 2024
Lemma 3.1. V T (z) ArgMax peapaaxpesmllle ((,, z) (p)] = a- M-EnntMax(z).
Proof. See Appendix C.1 for detailed proof.
Retrieval Dynamics. With Lemma 3.1, it is clear to see that the energy function (3.2) aligns with
the overlap-function construction of Hopfield models, in (Hu et al., 2023; Ramsauer al., 2020). Next, we introduce the corresponding retrieval dynamics satisfying the monotinicity property (TI).
Lemma 3.2 (Generalized Sparse Hopfield Retrieval Dynamics). Let be the iteration number. The retrieval dynamics of the generalized sparse Hopfield model is 1-step update of the form
(xx) = x (5=) = Ma-EntMax (3E' x.) = X(+1
(3.4)
that minimizes the energy function (3.2) monotonically over t.
Proof. See Appendix C.2 for detailed proof.
To see how this model store and retrieve memory patterns, first introduce the following definition.
Definition 3.2 (Stored and Retrieved), Assuming that every pattern Eμ surrounded by sphere Sp with finite radius R == Mi ||5, Evll. we say Eu is stored if there exists a generalized fixed point of T, x E Sj to which all limit points x E Sμ. converge to, and S n, = for # v. We say E, is e-retrieved by T with x for an error if (x] - Eyll S.
To ensure the convergence property (T2) of retrieval dynamics (3.4), we have the next lemma.
Lemma 3.3 (Convergence of Retrieval Dynamics T). Given the energy function (3.2) and retrieval dynamics T(x) (3.4), respectively. For any sequence {x, ) generated by the iteration Xy++1 T(xe), all limit points of this sequence are stationary points of H.
Proof. See Appendix C.3 for detailed proof.
Intuitively, Lemma 3.3 suggests that for any query x, T (given by (3.4)) monotonically and itera- tively approaches stationary points of H. (given by (3.2)), where the memory patterns [Ea]ne[M] are stored. This completes the construction of a well-defined modern Hopfield model.
Fundamental Limits. To highlight the computational benefits of the generalized sparse Hopfield model, we analyze the fundamental limits of the memory retrieval error and memory capacity.
Theorem 3.1 (Retrieval Error). Let Temc be the retrieval dynamics of the dense modern Hopfield model (Ramsauer et al., 2020). Let z E A =(v) be the w'th element sorted descending z- sequence zunted == 2(1) 2 ... 2 2(M)). and x(z) == Max(k [M] + kz(k) > Ense 2(r) For all S,,, holds [[T(x) - E,.ll < Dense
E. and
for 2 2 a 1, [|T(x) E, S 2m(M - 1) 1)eepf -B ( (Ew: NE(M]
Max (E Eui )}
(3.5)
1
for a 2 2, TT(x) E. m d d
Max (Ev, - - (10)
+
(3.6)
VE(M)
Corollary 3.1.1 (Noise-Rooe Soaise-Robustness). In cases of noisy patterns with noise 77, i.c. x + 77 (noise in query) or &, E, + v (noise in memory), the impact of noise 7) on the sparse retrieval error [x((x) - Epl is linear for a 2 2, while its effect on the dense retrieval error I'Deeeee (x) - Eμ1 (or [x((x) Epll with 2 2 a 2 1) is exponential.
Proof. See Appendix C.4 for detailed proof.
Intuitively, Theorem 3.1 implies the sparse model converge faster to memory patterns than the dense model (Ramsauer et al., 2020), and the larger sparsity leads the lower retrieval error.
Lemma 3.4 (Memory Capacity Lower Bound). Suppose the probability of successfully storing and retrieving memory pattern is given by 1 p.p. The number of memory patterns sampled from sphere of radius that the sparse Hopfield model can store and retrieve has lower bound: M vpC d- where C is the solution for C = t/Wo(exp(a+tabb) b3) with Wo(-) being the principal branch of Lambert W function (Olver et al., 2010), a = 4/2-1 1f In (2ml//-11m///// (n 1} and 4m 8/5(d-1). 1). For
4

---

Published as a conference paper at ICLR 2024
sufficiently large 99, the sparse Hopfield model has a larger lower bound on the exponential-in-d memory capacity compared to that of dense counterpart (Ramsauer et al., 2020): M z MDesse
Proof. See Appendix C.5 for a detailed proof.
Lemma 3.4 offers lower bound on the count of patterns efffectiveyy stored and retrievable by T with a minimum precision of R, as defined in Definition 3.2. Essentially, the capacity of the generalized sparse Hopfield model to store and retrieve patterns grows exponentially with pattern size d. This mirrors findings in (Hu et al., 2023; Ramsauer et al., 2020). Notably, when a 2, the results of Theorem 3.1 and Lemma 3.4 reduce to those of (Hu al., 2023).
3.2
GENERALIZED SPARSE HOPFIELD (GSH) LAYERS FOR DEEP LEARNING
Now we introduce the Generalized Sparse Hopfield (GSH) layers for deep learning, by drawing the connection between the generalized sparse Hopfield model and attention mechanism.
Generalized Sparse Hopfield (GSH) Layer. Following (Hu al., 2023), we extend (3.4) multiple queries X == <xx From previous section, we say that the Hopfield model, as defined by (3.2) and (3.4), functions within the associative spaces X and =. Given any raw query R and memory Y that are input into the Hopfield model", we compute X and E as X = RR q == Q and ET YWK = K, using matrices WQ and WK. Therefore, we rewrite T in (3.4) as (Qnew) K" a-Enntax (BKQ"). Taking transpose and projecting to with Wv. we have Z = Q" = a- a-EnttMax (SQK?) KWv = a-EnttMax (SQK)) V,
(3.7)
which leads to the attention mechanism with a- Max activation function. Plugging back the raw patterns R. and Y, we arrive the foundation of the Generalized Sparse Hopfield (GSH) layer,
GSH (R, Y) = Z = Max (SRW Q W Y TW&W,, By (3.5), T retrieves memory patterns with high accuracy after single activation. This allows (3.8) to integrate with deep learning architectures just like (Hu et al., 2023; Ramsauer al., 2020).
(3.8)
Remark 3.1. a is learnable parameter (Correia et al., 2019), enabling GSH to learn input sparsity.
GSHPooling and GSHLayer Layers. Following (Hu et al., 2023), we introduce two more variants: the GSHPooling and GSHLayer layers. They are similar to the GSH, and only differ in how obtain the associative sets Q Y. For GHSPooling (Y), ), K YWR V = KWv. v. and Q is a learnable variable independent from any input. For GSHLayer (R, Y), have K V Y, and Q R. Note that GSHLayer can have Q as learnable parameter or as an input. Where if Q was served as an input, the whole GSHLayer has no learnable parameters and can be used as a lookup table. We provide an example of memory retrieval for image completion using GSHLayer in Appendix D.3.
4 STANHOP-NET: SPARSE TANDEM HOPFIELD NETWORK
In this section, we introduce a Hopfield-based deep architecture (STanHop-Net) tailored for memory-enhanced learning of noisy multivariate time series. These additional memory-enhanced functionalities enable STanHop-Nee to effectively handle the problem of slow response to sudden or rare events (c.g, 2021 pandemic meltdown in financial market) by making predictions using both in-context inputs (c.g.. historical data) and external stimuli (c.g., real-time or relevant past data). In the following, we consider multivariate time series X E RCxx'xd comprised of C univariate series. Each univariate series has T time steps and d features.
4.1 PATCHED EMBEDDING
Motivated by (Zhang and Yan, 2023), cent time steps into subseries patches. This method extends the input time horizon without altering token length, enabling us capture local semantics and critical information more effectively, which is often missed at the point-level. We define the multivariate input sequence as X E CxT xdl where C,T, d denotes the number of variates, number of time steps and the number of dimensions of each variate. Given a time series sequence X = {x1, .X., XT and a patch size P. the patching operation divides X into = {si, ST/P]; For each patched sequence S; E RCxPxd for [T/P], we define the patched embedding as EMB (s,) = Eemb s; + Epos (i) E TR. Panb, where Demb is the embed- ding dimension, Eemb E R Pems x and E T/ the positional encoding. When T is
a patching technique model input that groups adja-
"The raw query R and memory Y may originate from data, external sets, or hidden representations through- out given deep learning pipeline. They not necessarily usable as X and S. Therefore, to use (3.4), they must he mapped into d-dimensional associative spaces.
5

---

Published as a conference paper at ICLR 2024
Tandem Spane--kphhhieeaeee Sub-BBodk
Resollutio
Coarser Resolution
Resolution
TimeGSH SerlesGSH
TimeGSH
Opptonal
SeriesGSH
STantfop: RR
(Saanllop: Y)
STanilopoR. with Y [SaanHoyyRR Ri without
STannlop:RR R) without Y
[SauuHooe: RR, R) without
MLP
Fortical Goeding
It Hoptied Layer's Qvery Pettore Hipilied Liper's sieenoyy justern Sppccyy
TimeGSH SeriesGSH
OSSHR.
TimeGSH SerlesGSH
OSSUR. YI
TimedSH SerlesGSH
OSBIR.
Hierorchicallyy Stacked STanHop Blocks
Figure STanHop-Neet Overview. Patch Embedding: Given an input multivariate time series
RCXTx2 consisting C univariate series, T time steps and d features, the patch embedding aggregates temporal information for each univariate series, subsequently reducing temporal dimensionality from T P T/P for all features. STanHop Block: The STanHop block leverages the Generalized Sparse Hopfield (GSH) model (Section 3). It captures time series representations from its input through two tandem sparse-Hopfield layers sub-blocks (i.e. TimeGSH and SeriesGSH, see Figure 2), catering to both temporal and cross-series dimensions. STanHop-Net: Using stacked encoder-decoder structure, STanHop-Net facilitates hierarchical multi resolution learning. This design allows STa Net extract distill representations from both temporal and cross-rorrs series dimensions across multiple scales (multi resolution in a hardwired fashion via coarse coarse-graining layers, see Section 4.4). Moreover, each stacked block has optional external memory plugin functionalities for enhanced predictions (Section 4.3). These representations from all resolutions are then merged, providing holistic representation learning for downstream predictions specially tailored for time series data.
not divisible to P. assuming T P x Cn + with Cn, E N+, we pad the sequence by repeating the first e elements in the sequence. Consequently, this patching embedding significantly improves computational efficiency and memory usage.
4.2 STANHOP: SPARSE TANDEM HOPFIELD BLOCK
We introduce STanHop (Sparse Tandem Hopfield) block which comprises one GSHLayer- external memory plugin module, and two tandem sub-blocks of GSH layers to process both time and series dimensions, L.c. TimeGSH and SeriesGSH sub-blocks in Figure 2. In essence, STanHop not only sequentially extracts temporal and cross-series information of multivariate time series with (learnable) data-dependent sparsity, but also utilizes both acquired (in-context) representations and external stimulus through the memory plugin modules for the downstream prediction tasks.
Given hidden vector, R E CCTTxxau and its corresponding external memory set Y RMxXXTxDDase where C denotes the channel number and T denotes the number of time seg- ments (patched time steps), To clarify, the GSH layer only operates on the last two dimensions, i.e., t e T and d E Dmidden. Thus, the operation GSH (Z, Z) extracts information of the temporal dynam- ies of Z from the segmented time series. Here we define the dimensional transpose operation T. For given tensor X E Raxbxe we have TX*((X) (X) = X' Raxexb i.e. this operation rearranges the dimensions of the original tensor X from (a, b, e) to a new order (a, c, b). Given set of query
pattern Q E Rleng x Duata, we define single block of STanHop as
Z Memory (R, Y),
(Memory Plugin Module, see Section 4.3)
Z Teth (LayerNorm (Z+ FF (GSH 2)))
(Temporal GSH)
Z" GSHPooling (R', ZZ') E IR Txkng x Diates
(R. is learnable and randomly initialized)
Z CSH (Z', ZP) E Tx'xx Dawie
(Cross-series GSH)
Z LayerNorm Zz' FF (Z9)) ERR TXCXDDam
Zout = LayerNorm (Z* + FF(Z")) E TTCxxXXDDo Daaatn
where Memory(::)) ) is the external memory plugin module introduced the next section. Note that, if we choose turn off the external memory functionalities (or external memory is not available) during training, we set Y = R such that Memory(R. (R. R) R (see Section 4.3 for details). Here ESHPoolingle (R" Z') takes Z and a randomly initialized query R* as input, Importantly, R not only acts as learnable prototype patterns learned by pooling over Z', but also as a knob to control the computational complexity by picking the hidden dimension of R* We summarize the STanHop block as Zom STanHop (R, Y) E RT'X' xx'xx'xXx Drates
4.3 EXTERNAL MEMORY PLUGIN MODULE AND PSEUDO-LABEL RETRIEVAL
Here we introduce the external memory modules (i.e., Memory(-s) in Section 4.2 or Memory Plugin blocks in Figure 2) for external memory functionalities. These modules are tailored for time series modeling by incorporating -ase-iccc supplemental information (such as relevant historical data for sudden or rare events predictions) for subsequent inference. To this end, we introduce two

---

Published as a conference paper at ICLR 2024
SeriesGSH
STanHop Block
Plug and Play Memory Plugin
Tune and Play Memory Plugin
Input Sequence m (X]
ing
Memory Pattems
DHHLayerrR
DSHLayerck
Layerhoom
Memory Plugin
Coarse-Graining
LayerNorm
Series6SH
Layerform
Residual Layer
Residual Layer
Figure 2: STanHop Block. (Leff) Tandem Hopfield-Layer Blocks: TimeGSH and SeriesGSH. Notably, in the GSHPooling block of SeriesGSH, the learnable query R is initialized randomly and employed to store learned prototype patterns from temporal representations extracted during training. (Right) Plug-and-Play and Tune-and-Pla Memory Plugins.
memory plugin modules: Plug-and For query R and memory Y, we denote them by PlugMemory(R, Y) and TuneMemory(R, Y).
Memory Plugin and Tune-and-Play Memory Plugin.
Plug-and-Play Memory Plugin. This module enables performance enhancement utilizing external memory without any fine-tuning. Given a trained STanHop-Net (without external memory), we parameter fixed GSHLayer for memory retrieval. Explicitly, given input sequence R || Druse and a corresponding external memory set Y E RMx ji,xd, where [R| and Duidden are the sequence length and hidden dimension of R. respectively. We define the memory retrieval operation as Z PlugMemory (R. Y) LayerNorm (R + GSHLayer (R, Y)) with all parameters fixed.
Tune-and-Play Memory Plugin. Here we propose the idea of "pseudo-label retrieval" using GSHLayer for time series prediction. Specifically, we use modern Hopfield models' memory trieval mechanism to generate pseudo-labels for a given R from a label-included memory set Y. thereby enhancing predictions. Intuitively, this method suppiements predictions by learning from demonstrations and we use the retrieved pseudo-labels (i.e., learned preudddoreditiion)) as additional features. An illustration of this mechanism is shown in Figure 2. Firstly, we pre- pare the label-included external memory as Y Y + Yiabc,. where Y is the concatenation of memory sequences and their corresponding labels. Next, we denote the padded R as R, where R E T Y xd And we utilize the GSHLayer to retrieve the pseudo-label from the memory se- quences Zun- Then we concatenate R and the pseudo-label Zou and send it to a feed forward layer to encode the pseudo-label information: Zout = GSHLayer [R, Y). Zpscudo = R Zour and then Z = LayerNorm (FF (Zppeddo + Z. In other words, we first obtain a weight matrix from the association between R. and Y, and then multiply this weight matrix with Yaaell to obtain Z We summarize the Tune-and and-Play memory plugin as Ž TuneMemory(R, Y).
4.4 COARSE-GRAN GRAINING
To cope with the intrinsic multi-resolution inductive bias of time series, we introduce a coarse- graining layer in each STanHop block. Given an hidden vector output, Z E Cxxx Dastee, grain level A, and a weight matrix W E RR Diati x2D and @ denotes the concatenation operation. We denote Ze.r,d with e [C], t E (T], [Dmasen] as the element representing the c-th series, t-th time segment, and d-th dimension. The coarse- graining layer consists a vector concatenation and matrix multiplication: Ze.t.: Ze,t, + Ze,t+s. E 2 and then Ze,t,: WZe,t,; E IR Puates such that Ż E 2 and Z E XCxTTDauas similar to (Liu et al., 2021b; Zhang and Yan, 2023). Operationally, it first obtains the representation of smaller resolution, and then distills information via a linear transformation. We express this course- grausing layer as CoarseGrain (Z, = Z.
coarse-graining
4.5 MULTI-LAYER STANHOP FOR MULTI- COLTI-RESOLUTION LEARNING
Finally, we construct the STanHop-Net by stacking STanHop blocks in a hierarchical fashion, en- abling multi-resolution feature extraction with resolution-specific sparsity. Given a prediction win- dow size E I. number of layer L E R, and a learnable positional embedding for the decoder multi-layer STanHop
Euec. we construct consists of a course-gu course-graining operation first, following by an STanHop layer. The decoder follows the similar structure the standard transformer decoder (Vaswani al., 2017), but replace the cross-attention mechanism to a GSH layer, and self-attention layer as STanHop layer. We summarize the STanHop-Net network structure in Figure 1, and in Algorithm 2 in appendix.
autoencoder structure. The encoder structure
7

---

Published as a conference paper at ICLR 2024
Table I: Accuracy Comparison for Multivariate Time Series Predictions without External Memory. We implement 3 STanHop variants, STanHop-Net (D) with Dense Hopfield layer (Ramsauer et al., 2020). STanHop-Net (S) with Sparse SparseHopfield layer (Hu et al., 2023) and STanHop-Net with our GSH layer respectively. We report the average Mean Square Error (MSE) and Mean Absolute Error (MAE) metrics of 10 runs, with variance omitted as they are all 0.44%. We benchmark our method against leading transformer- based methods (FEDformer (Zhou et al., 2022). Informer (Zhou et al., 2021) and Autoformer (Wu et al., 2021), Crossformer (Zhang and Yan, 2022)) and a linear model with beasonal-trend decomposition (DLinear (Zeng al., 2023)). We evaluate cach dataset with different prediction horizons (showed in the second column). We have the best results bolded and the second best results underlined. In 47 out of 58 settings, STanHop-Nets rank either first or second, Our results indicate that our proposed STanHop-No Net delivers consistent top-tier
performance compared to all the baselines, even without external memory.
Models
FEDFomer
DLinear
Informer
Autoformer
Croesfomer
StanHop-Net
STunHep-Net (S)
STanIHop-Net
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MAE
MSE
24
0.318
0.384
0.312
0.355
0.577
0.549
0.439
0.440
0.305
0.367
0.301
0.363
0.298
0.360
0.294
0.351
48
0.342
0.35%
0.352
0,383
0.685
0.625
0.429
0.422
0.352
0.394
0.356
0.406
0.355
0.399
0.340
0.387
ETTM
168
0.412
0.449
0.416
0.430
0.931
0.752
0.495
0.499
0.40
0.411
0.398
0.440
0.419
0.458
0.398
0.437
336
0.456
0.474
0.450
0.452
1.128
0.873
0.509
0.492
0.440
0.41
0.488
0.472
0.484
0.50
0.472
720
0.521
0.515
0.484
0.501
1.215
0.896
0.539
0.537
0.519
0.524
0.516
0.522
0.541
0.533
0.112
0.511
24
0,290
0.364
0.217
0.289
0.325
0.369
0.410
0.428
0.211
0.293
0.205
0.278
0.191
0.270
0.195
0.273
ILI WTH HO. ETiml
18
0,342
0.396
0.278
0.350
0.191
0.503
0.483
0.161
0.500
0.352
0.303
0,310
0.293
0,311
0.270
0.353
0.366
0.412
0.310
0.354
0.678
0.614
0.502
0.476
0.320
0.373
0.325
0.377
0.322
0.362
0.286
288
0.398
0.433
0.369
0.386
1,056
0.786
0.604
0.522
0.404
0.427
0.410
0.429
0.395
0.413
0.366
0.399
672
0.155
0.161
0.116
0.417
1.192
0.926
0.607
0.530
0.569
0.528
0.574
0,516
0.556
0,510
0.400
0.131
18
0.229
0.338
0.155
0.258
0.344
0.393
0.241
0.351
0.156
0.255
0.159
0.261
0.170
0.273
0.152
0,252
0.263
0.361
0.195
0.368
0:424
0.299
0.387
0.231
0.309
0.296
0.368
0.288
0.373
0.227
0.304
336
0.305
0.386
0.238
0.316
0.381
0.431
0.375
0.428
0.323
0.369
0.326
0.374
0.317
0.375
0.317
0.361
720
0.372
0.134
0.272
0.346
0.105
0.143
0.577
0.134
0.104
0.123
0.412
0.428
0.440
0.450
0.105
0.416
960
0,393
0.449
0,299
0.367
0,460
0.548
0.366
0.426
0.433
0.438
0.446
0,447
0.467
0,463
0.430
0.431
0.357
0.412
0.357
0.39)
0.335
0.381
0.363
0.396
0.294
0.343
0.304
0.351
0.352
0.292
0.341
48
B.42X
(3-45%
D3425
0-444
0.395
0.459
0.45
0.462
0.370
0411
0.374
0.411
D.372
0.411
0.363
0.402
168
0.564
0.541
0.516
0.516
0.608
0.567
0.574
0.548
0.73
0.494
0.480
0.501
D.496
0.511
0.332
0.393
336
0.533
0,536
0.536
0.537
0,702
0.620
0.600
0.571
0.495
0.515
0.507
0.526
0.514
0.530
0.499
0,515
0.562
0.557
0.582
0.571
0,31
0.731
0.587
0.536
0.542
0.545
0.557
0.556
0.546
74
2.687
1.147
2,940
1.205
4.588
1.238
3,041
1.186
3.305
1.241
3.194
1.176
3.121
1.139
36
2,847
1.160
2.826
1.184
4.845
1,496
1.397
1.270
3,405
1.252
3.542
1.314
3.193
1.169
3.288
1.182
48
2.797
L155
2.677
L155
4,865
1.516
2.947
1.205
3.459
1.221
3,409
1,208
3.15
1,142
3.122
1.120
60
2,8099
1.163
1.011
5,212
1.576
3.019
3.640
1.305
1,269
1.196
3.416
74
B.562
0.375
00351
D.761
0.608
0.334
0.550
0.363
0.491
0.271
0.484
0.266
D.499
0.277
0.422
0.247
4H
0.567
0.374
0.370
0.230
0.644
0.359
0.595
0.376
0.519
0.295
0.516
0.293
D.516
0.290
0.315
0.261
Traftic
168
0.607
0.385
0.395
0,277
0.000
0.391
0.649
0.407
0.513
0.289
0.511
0.301
0.517
0.289
0.501
0.276
336
0.624
0.389
0.415
0.189
0.747
0.405
0.624
0.388
0.530
0.300
0.531
0.316
0.544
0.503
0.506
0.288
730
0.623
0.378
0.455
0.313
0.792
0.400
0.674
0.417
0.573
0.313
0.569
0.303
0.563
0.311
0.539
0.300
5
EXPERIMENTAL. STUDIES
We demonstrate the validity of STanHop-Net and external memory modules by testing them on various experimental settings with both synthetic and real-world datasets.
5.1 MULTIVARIATE TIME SERIES PREDICTION WITHOUT EXTERNAL MEMORY
Table includes the experiment results of the multivariate time series predictions using STanHop- Net without external memory. We implement three variants of STanHop-Net: StanHop-Net, StanHop-Net (D) and StanHop-Net (S), with GSH, Hopfield (Ramsauer et al., 2020) and Sparseloppf iel (Hu et al., 2023) layers respectively. Our results show that in 47 out of 58 cases, STanHop-Nets rank in the top two, delivering top-tier performance compared to all baselines.
Data. Following (Zhang and Yan, 2023; Zhou et al., 2022; Wu et al., 2021), we use 6 realistic datasets: ETThT (Flectricity Transformer Tlemperature-ourlly ETTml (Flectricity Transformer Temperature-minn minntely. WTH (Weather), ECL (Flectricity Consuming Load), ILI (Influenza-Like Illnes),, Traffic. The first four datasets are split into train/val/test ratios of 14/5/5, and the last two are split into 7/1/2. Metrics We use Mean Square Error (MSE) and Mean Absolute Error (MAE) as accuracy metrics. Setup. Here we use the same setting as in (Zhang and Yan, 2022): multivariate time series predictions tasks on 6 real-world datasets. For each dataset, we evaluate our models with several different prediction horizons. For all experiments, we report the mean MSE, MAE over 10 runs. Baselines. We benchmark our method against 5 leading methods listed Table 1. Baseline results are quoted from competing papers when possible and reproduced otherwise. Hy- perparameters. For each experiment, we optimize the hyperparameters using the "sweep" function from Weights and Biases (Biewald et al., 2020). We conduct 100 random search iterations for each setting, selecting the best set based on the validation performance.
For datasets, hyperparameter tuning, implementations and training details, please see Appendix F.
PREDICTION: MEMORY PLUGIN VIA HOPFIELD LAYER In Table 2 and Figure 3, we showcase STanHop- with external memory enhancements delivers performance boosts in many scenarios. The external memory enhancements support two plugin schemes, Mlug-and-Play and Tune-and-Play. They focus on different benefits. TuneMemory is es-
5.2
8

---

Published as a conference paper at ICLR 2024
10
HagMMrrrr No
Pluglcmaay
HugWerrrrr
233
PugMonorry
3.0
No Menory
No
09
1.0
2.6
0.8
2.0
1.5
0.7
0.8
155
1/
10
06
088
0.5
05
0.1
02
0.3
0.4
0.5 0.1
0.2
0.3
044
0.5
0.1
02
0.5
0.4
0.5 0.1
0.2
0.3
3.4
0.5
Noko Scale
Nass Ecolo
Nokia Soale
Kasso Booke
Figure 3: Visualization of Memory Plugin Scenarios Case
From Left to Right: MAE against
different noise levels with (1) ETThl prediction horizon 336; (2) ETTht + prediction horizon 168; (3) ETTml + prediction horizon 288; and (4) ETTml + prediction horizon 96. The results show the robustness of PlugMemory against different level of noise.
pecially useful for task-relevan knowledge incorporation by fine-tuning on an external task relevant memory set'. On the other hand, PlugMemory provides more robust representation of inputs with high uncertainty by doing a retrieval (Figure 2) on an external task-relevant memory set, without the work of any training or fine-tuning. Below we provide 4 practical scenarios to showcase the aforementioned benefits of TuneMemory and PlugMemory external memory modules. The detailed setups of each case can be found in the appendix.
Case (TuneMemory). We take the single variate, Number of Influenza incidence in week (denoted as ILI OT), from the ILI dataset as a straightforward example. In this dataset, we are aware of the existence of recurring annual patterns, which can be readily identified through visualizations in Fig- ure 18. Notably, the signal patterns around the spring of 2014 closely resemble past springs. Thus, in predictions tasks with input located in the yearly recurring period, we collect similar patterns from the past to form a task-relevant external memory set.
Case 2 (TuneMemory). In many sociological studies (Wang al., 2021a;b), electricity usage ex- hibits consistent patterns across different regions, influenced by the daily and weekly routines of residents and local businesses. Thus, collect sequences that match the length of the input quence but are from 1 to 20 weeks prior, obtaining a task-relevant external memory set of size 20.
In addition, we also include analysis of "bad" external memory sets, to verify the effectiveness of incorporating informative external memory sets. We construct the "bad" external memory sets by randomly selecting from dataset without any task-relevant preference, see Appendix F.2 for more details about such selection. The results indicate that, by properly selecting external memory sets, we further improve the models' performance. On the contrary, randomly chosen external memory sets can negatively impact performance. We report the results of Case and Case 2 in Table 2.
Case (PlugMemory). Through PlugMemory. informative patterns can be extracted from a memory set for the given noisy input. To verify this ability, we construct the external memory sets based on the weekly pattern spotted in ETThl and ETTml, and add noise of different scales into the input sequence. We add the noise following <- x scale- std(x)... For Case 3, we use the ETTh dataset. Case 4 (PlugMemory). For Case 4, we evaluate PlugMemory on the ETTml dataset.
Table 2: Performance Comparison of the StanHop Model with TuneMemory and Ablation Using Bad External Memory Sets (TuneMemory-(b). We report the mean MSF and MAF over runs with variances omitted as they are < 0.79%. For IIJ OT, we consider prediction horizons of 12, 24, and 60. For ETTh1, we choose prediction horizons of 24, 48, and 720, covering both short and long durations. The results indicate that using dataset insights and TuneMemory enhances our model'ss performance.
Case 3. (ETTN)
Defral
Ruatheccy-00
Defeal
Tuseheccy-h)
MSE
MAB
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
12
4011
1.701
3.05 (49%)
1,693 (4.%)
4,540 (85
1.759 (+0.55)
24
0.294
0.351
(3.4%)44%
0.351 cauP))
0 3,300 (o229))
0.361 (x2.4%)
24
4254
1771
3300 (695)
1.80 (466)
4.271 (e0.88)
1,276 (+0.9%)
0.340
0.387
0.303
(3.6%)
0.519 (2.19)
1.54 (60
U1388 (+0 (+.
69
3613
1683
387261115
(839))
A821 (65
1.735 (47.48
720
0.512
0.11
0.904
(1.6))
0.512 (4.2%)
0.514 (40
0.521 (+7.09)
6
CONCLUSION
We propose the generalized sparse modern Hopfield model and present STanHop-Net, Hopfield- based time series prediction model with external memory functionalities. Our design improves time series forecasting performance, quickly reacts to unexpected rare events, and offers both strong theoretical guarantees and empirical results. Empirically, STanHop-Nets rank in the top two in 47 out of our 58 experiment settings compared to the baselines. Furthermore, with PlugMemory and TuneMemory modules, it showcases average performance boosts of ~ 12% and ~3% for each.
Task-relevant means the relevance to the inputs of the time series forecasting. A task-relevant memory set could be set of some history time series segments that are relevant to the inputs of the prediction.
9

---

Published as a conference paper at ICLR 2024
Note Added [December 27, 2023]. During IC'R'24's rebuttal period, the authors learn of an up- coming work by Martins et al. (2023), addressing similar topics from the perspective the Fenchel- Young loss (Blondel et al., 2020). The authors would like to thank Martins et al. (2023) for reading our manuscript and testing our code.
ACKNOWLEDGMENTS
JH would like to thank Feng Ruan, Dino Feng and Andrew Chen for enlightening discussions, Pei- Hsuan Chang for pointing out typos proofs, the Red Maple Family for support, and Jiayi Wang for facilitating experimental deployments. The authors would also like to thank the anonymous reviewers and program chairs for their constructive comments.
JH is partially supported by the Walter P. Murphy Fellowship. BY is supported by the National Tai- wan University Fu Bell Scholarship. HL is partially supported by NIH ROILM1372201, NSF CA- REER1841569, DOE DE-AC02-07CHI 1359, DOE LAB 20-2261 and NSF TRIPODSI740735. This research was supported in part through the computational resources and staff contributions pro- vided for the Quest high performance computing facility Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Infor- mation Technology. The content solely the responsibility of the authors and does necessarily represent the official views of the funding agencies.
REFERENCES
Donald WK Andrews. Cross-section regression with common shocks. Econometrica, 73(5):5551- 1585, 2005. URL https:/www.jssorrrrrttttl333333
Dimitri P Bertsekas, W Hager, and O Mangasarian. Nonlinear programming. athena scientific bel- mont. Massachusets, USA, 1999. URL http://tteenaaccoo//npplll
Lukas Biewald et al. available from wandb. com, 2:233, 2020. URL https//www..aggle..om/o//ddaaaa
Experiment tracking with weights and biases. Software
experiment--racking wutt-weighhs-anne
Mathieu Blondel, André FT Martins, and Vlad Niculae. Learning with fenchel-young losses. The
Journal of Machine Learning Research, 21(1):1314 2020.
Philip Bond and James Dow. Failing forecast rare events. Journal of Financial Eco-
nomics, 140111:1001-1016, 2021. URL http://www.aaaceeeiecttoooc/eeee $0304405X210029811
Johann S Brauchart, Alexander B Reznikov, Edward B Saff, Ian H Sloan, Yu Guang Wang, and Robert Womersley. Random point sets on the sphere-hole radii, covering, and separation. Experimental Mathematics. 27(1):62-81, -1, 2018. URL http://wwww.anddoiiiieeoo/////// 1080/10586458.201... 1226209.
Matthieu Bussiere and Marcel Fratzscher. Towards new early warning system of financial crises. journal of International Money and Finance, 25(6)9953,973, 2006. URL https://www.
Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mip architecture for time series forecasting. arXiv preprint arXiv:2303.06053 2023.
Gonçalo M Correia, Vlad Niculae, and André FT Martins. Adaptively sparse transformers. arXiv preprint arXiv:1909.0000 00015, 2019. URL hutps://axxivorr/abbs. wawpaasss/999900oo 000000015.
John M Danskin. The theory of max-min and its application to weapons allocation problems, vol- /0.00//00/0000000/////
ume 5. Springer Science & Business Media, 2012. URL
978-3-642-46092-0.
Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-299 1057//1095---17
2017. URL https://ink.griiger..c////tiell/ 00tp
10

---

Published as a conference paper at ICLR 2024
Eugene F Fama and Kenneth R French. Comparing cross-section and time-series factor models. The Review of Financial Studies, (3(5):1811-192)) 2020. URL https://academmic.ouu oup. http:///mmmmm..ooooo/////ttttooo com/ffs/articcc///
33/5/1891/5555877
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alaie Muller. Deep learning for time series classification: review. Data mining and knowledge discov- ery, 3344)9177963,, 2019. URL
1007/s 0610000116600 8-019-0066 9-1.
Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner, Viet T Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp. Günter Klambauer, Angela Bitto, et al. Cloob: Modern hop- field networks with infoloob outperform clip. Advances in neural information processing sys- tems, 35:20450-20468, 2022. URL http::/poeeediigg.nuuui.p..c///p htthaapp...ee//ppppe2/22aaaaaal (1 -tw.:////oo.00022----aatt----ooo Cesstract-Conffeeenceehmt
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv: 2014. URL http://anriv.rr/ah//111....1
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińssa, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538
(7626):471-46,, 2016. URL http:://wwwwwaattteoooooo///////
Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisit- ing the channel independent strategy for multivariate time series forecasting. arXiv preprint 2rXiv:2344.05206 2023.
Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed J Zaki, and Dmitry Krotov. Energy transformer. arXiv preprint arXiv:2302.07753,, 2023. URL https://proceedings. wwwpediiinnnnnnussss..mm www.rips.ccppaperffieeeeeee wwtpppppee...iee/ppppe//22aaaaae 37486977776699622888888811111111 Abstratt Abstract-Conferrncc htmml..
John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(3):2554-2558, 1982. URL https:
wwww.pnas.org/dbi/abs/////////// 1ww.:/paa..9...2222
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10):30883302) 1984. URL https:/www..ppas..o//doooaaa wttp:///abbb....../ppao....mmmm 81, 10..88...
Jerry Yao-Chieh Hu, Donglin Yang, Dennis Wu, Chenwei Xu, Bo-Yu Yu Chen, and Han Liu. On sparse modern hopfield model, 2023. URL https://arxivvor//aas//300 12673.
Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Robin Luo, Hong-Yu Chen, Weijian Li, Wei-Po Wang, and Han Liu. Outlier-efficient hopfield layers for large transformer-based models. 2024a. URL http::/Aariivog//as//44000022
Jerry Yao-Chieh Hu, Bo-Yu Chen, Dennis Wu, Feng Ruan, and Han Liu. Nonparametric modern hopfield models. 2024b. URL https///aaiivoryyaa////...009
Jerry Yao-Chieh Hu, Thomas Lin, Zhao Song, and Han Liu. On computational limits of modern hopfield models: A fine-grained complexity analysis. arXiv preprint urXiv:2402.04520, 2024c. URL https://arxxv.on/aa/2244.....442
Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learn- ing. In International conference on machine learning, pages 2127-2136. PMLR, 2018. URL
Eukasz, Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. arXiv preprint arXiv: /703.03/2. 2017. URL. ww...hmps//arriiooaaa///11000111
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445) 2020. URL https//arriioorg/abb/0000.

---

Published as a conference paper at ICLR 2024
Leo Kozachkov, Ksenia V Kastanenka, and Dmnity Krotov. Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences, 120(3)):22211150120, 2023. URL
Dmitry Krotov, A new frontier for hopfield networks. Nature Reviews Physics, pages 1-2, 2023.
URL
Dmitry Krotov and John Hopfield. Large associative memory problem in neurobiology and machine learning. arXiv preprint arXiv:2008. 06996, 2020. URL https///arxiv.orr/bb/22000 06960
Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016. URL https://rooceedings http:///00000000004444400000000. -aaaaalaaa22211-----ttttttt.. www.:/iiinnnnnnissssi.mmmm heurips.cc/paper files/
Ricardo Laborda and Jose Olmo. Volatility spillover between economic sectors in financial cri-
sis prediction: Evidence spanning the great financial crisis and covid-19 pandemic. Research in International Business and Finance, 57:101402, 2021. URL https:/www.sccennceirrccccom
Phong VV Lc, James T Randerson, Rebecca Willett, Stephen Wright, Padhraic Smyth, Clement Guilloteau, Antonios Mamalakis, and Efi Foufoula-Georgion Climate-driven changes in the predictability of seasonal precipitation. Nature communications, 14(1):3822, 2023. URL http://aatttrreoooaarreees//aeee -23-3------------
YC Lee, Gary Doolen, HH Chen, GZ Sun, Tom Maxwell, and HY Lee. Machine learning using higher order correlation network. Technical report, Los Alamos National Lab,(LAN)), Los Alamos, NM (United States); Univ. of Maryland, College Park, MD (United States), 1986. URL
https//www..st..gvvbbaa /ow//ibbibili/76664.....
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019. URL https://arxiv.org/ Abss/190.00023.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021a. URL https: "igpenreiee..erfoommddddEExEEa
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers effective for time series forecasting. arXiv preprint arXiv:2310.0662. 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Bain- ing Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021b. URL https://openacces.. htwpppppewaceess.eeee..ooooo wtw.e//..oomcooo///////000 1tt/2002/hm//LLuSS// Swin Transformer. Hierarchical, Vision_Transforman
Windows ICCV _0l_2222_______ _paper.
Donghao Luo and Xue Wang. Modernten: A modern pure convolution structure for general time series analysis. The Twelfth International Conference on Learning Representations, 2024.
Andre Martins and Ramon Astudillo. From softmax to sparsemax: sparse model of attention and multi-label classification. In International conference on machine learning, pages 1614-1623. PMLR, 2016. URL
Andre F. T. Martins. Vlad Niculae, and Daniel McNamee. Sparse modern hopfield networks. Associative Memory Hopfield Networks 2023. NeurIPS 2023 workshop.. 2023. URL.
Ricardo P Masini, Marcelo C Medeiros, and Eduardo F Mendes. Machine learning advances for time series forecasting. Journal of economic surveys, 37(1):76-111, 2023, URL http://ooniiiiiibrrar 12429. 12429.
willey.com/do//bs/110.1111 1/joes.
12

---

Published as a conference paper at ICLR 2024
Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Uni- versal hopfield networks: A general framework for single-shot associative memory models. In International Conference Machine Learning, pages 15561-15583, PMLR, 2022. URL
http://proceddinss.l.preess hemps//ieiimmmiddee22aaama http:
Charles M Newman. Memory capacity in neural network models: Rigorous lower bounds. Neu- ral Networks, 1(3):223-22 23,2,, 1988. URL http://www..cccccccccccccooooooe
08936080889002887.
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.147.00 2022. URI
hutps//aaxxx.oor//bb//22 14730.
Frank WJ Olver, Daniel W Lozier, Ronald F Boisvert, and Charles W Clark. NIST handbook of mathematical functions hardback and CD-ROM. Cambridge university press, 2010. URL htips: //dImf. fillmf.nist.gov nist.gov/.
Fabian Paischer, Thomas Adler, Vihang Patil, Angela Bitto-Nemling, Markus Holzleitner, Sebastian Lehner, Hamid Eghbal-7adeh, and Sepp Hochreiter. History compression via language models in reinforcement learning. International Conference on Machine Learning, pages 17156-17185. PMLR, 2022. URL http::/proccecings..trrppp 122ppaischrr22a.ttm
Pierre Peretto and Jean-Jacques Niez. Long term memory storage capacity of multiconnected neural networks. Biological Cybernetics, 54(1):33-63, 1986. URI. https://link.sppiggerr sprrnger..omm//rrcc///// 1007/BF003371
Ben Peters, Vlad Niculae, and André FT Martins. Sparse preprint ar arXiv:1905.057000 2019. URL https//arrxv.or///a////5..5....
models. arXiv
Hubert Ramsauer, Bernhard Schafl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovic, Geir Kjetil Sandve, et al. Hopfield networks all you need. arXiv preprint arXiv:2008.080 02217, 2020. URL https://axxivoor//ab// 2008.02217.
Alex Reneau, Jerry Yao-Chieh Hu, Chenwei Xu, Weijian Li, Ammar Gilani, and Han Liu. Feature programming for multivariate time series prediction. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 29009-29029, PMLR, 23-29 Jul 2023. URL
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory- augmented neural networks. In International conference on machine learn- ing, pages 1850- PMLR, 2016. URL
1..
Johannes Schimunek, Philipp Seidl, Lukas Friedrich, Daniel Kuhn, Friedrich Rippmann, Sepp Hochreiter, and Günter Klambauer. Context-enriched molecule representations improve few-shot drug discovery. In The Eleventh International Conference on Learning Representations, 2023.
URL https://ppeueeeeeeeeeeeetoueeeee
Philipp Seidl, Philipp Renz, Natalia Dyubankova, Paulo Neves, Jonas Verhoeven, Jorg K Wegner, Marwin Segler, Sepp Hochreiter, and Gunter Klambauer. Improving few-and zero-shot reaction template prediction using modern hopfield networks. Journal of chemical information and mod- eling, (2(9):21 1-2120. 2022. URL https://pbb..ass.og//////t/t// 1ttp:////ffuul11.10222111 1w.11/0css..immm.. 1c0106..
Cuneyt Sevim, Asil Oztekin, Ozkan Bali, Serkan Gumus, and Erkam Guresen. Developing early warning system to predict currency crises. European Journal of Operational Re- search, 237(3):1095-1104 2014. URL 50377221714001822
Aditi Sheshadri, Marshall Borrus, Mark Yoder, and Thomas Robinson. Midlatitude error growth in atmospheric gems: The role of eddy growth rate. Geophysical Research Letters, 28(23): 2021. URL http://aaagpllsssalllllloayyyyaa 2021 0D21GL096110 (096106..... 26.
13

---

Published as a conference paper at ICLR 2024
Bharath K Sriperumbudur and Gert RG Lanckriet. On the convergence of the concave-convex proce-
dure. In Nips, volume 9. pages 1759-1767, 2009. URL https://proceeddigss httpr//ccddnngneuuuuppopoooppppp meuripp..c//paprr.iii files/ http:22000asss50000aaaaaa0206663 Absrb900tt--ttttttttllmmm..
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015. URL https://axxiv.rg/aa////// 550.08895.
Constantino Tsallis. Possible generalization of boltzmann-gibbe statistics. Journal of statistical physics, 52:479 487, 1988. URL https://innksspinggr.oommaaa
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa-
fion processing systems, 30, 2017. URL https:/proocedings.eeen www.ips..c/papppe.eeeeeeee files/paper/2017/ hash/3ifee244544dee 1fb03333 144335aa- 1448445a--bsttact.hmm Abstract.hmll.
Xinlei Wang. Caomingzhe Si, Jinjin Gu, Guolong Liu, Wenxuan Liu, Jing Qiu, and Junhua Zhao. Ellecricity-consumptinn data reveals the economic impact and industry recovery during the pandemic. Scientific Reports, 11(1):19960, 2021a. URL. http://www..atuur.coommaaaiaaaie -41598-021-982594
Zhe Wang, Tianzhen Hong, Han Li, and Mary Ann Piette. Predicting city-scale daily electricity consumption using data-driven models. Advances in Applied Energy, 2:100025, 2021b. URL
Jason Weston, Sumit Chopra, and Antoine Bordes, Memory networks.
arXiv preprint
1rXiv:1410.3910 2014. URL https//arxiv.or//abs/11000
Michael Widrich, Bernhard Schäff, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Hol- zleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern hopfield networks and attention for immune repertoire classification. Advances in Neural Infor- mation Processing Systems. 33:18832 118832-1845,, 18845, 2020. URL https//proceedings..eurrt wwwptiigsssuuuscc//ppapro. 4000hsshha44002000032282833796/3 Aebdee/f0 -bssrratt.tttmm.
Jeffrey M Wooldridge, Mokhtarul Wadud, and Jenny Lye. rics: Asia pacific edition with online study tools 12 months. Cengage AU, 2016.
Introductory economet-
URL https://books. https//bboks.google.cooo
Econometrics Pacific E.html?
(dw.X.LDAAAQBAAJrrdii..... esc=yy
Dennis Wu, Jerry Yao-Chieh Hu, Teng- Yun Hsiao, and Han Liu, Uniform memory retrieval with larger capacity for modern hopfield models. 2024. URL http://arriioorgaaa///0........
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. Advances in Neural Information Pro- cessing Systems, 212222211224300 2021. URL bec04400288893ee8dcdffcco 1aw.dc0cccc0 Absttbbtttttttttttmmm
http:://wnnnnuuuuisssc///pppppaa 1/hash/
Chenwei Xu, Yu-Chao Huang, Jerry Yao-Chieh Hu, Weijian Li, Ammar Gilani, Hsi-Sheng Goan, and Han Liu. Bishop: Bi-directional cellular learning for tabular data with generalized sparse
modern hopfield model. 2024. URL https//aarxivorr//b///40044
Guoqi Yu, Jing Zou, Xiaowei Hu, Angelica Aviles-Rivero, Jing Qin, and Shujun Wang. Revitaliz- ing multivariate time series forecasting: Learnable decomposition with inter-series dependencies and intra-series variations modeling. arXiv preprint arXiv:2402. 2024.
Alan L Yuille and Anand Rangarajan. The concave-convex procedure (ccep). Advances in neural information processing systems, 14, 2001. URL http://pproceedinsss.emiijj htt.spp/ppp.e.llsppppppppeel 2001/hash/a01126693 11010000011111144444440000053300
files/paper/
Alan Yuille and Anand Rangarajan. The concave-convex procedure. Neural computation, 15(4):
915-936, 2003. URL http:////eeepooo.....eoo///////e
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023. URI. https://ojs.aaal wttpaa/wwwggg.a.pp//////aaaaaaaa
14

---

Published as a conference paper at ICLR 2024
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2022. URL http:://pcccctteoooooooooo/s///s
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023. URL
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence. volume 35, pages 11106-11115, 2021. URL https://ojs.aeet wwwp:///nnw..ppp/AAA/Aaaaaaaaaaa
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pages 27268-27286 PMLR, 2022. URL http://pooceeddinng.mi..e.ee v 162//hhou22.html.
15

---

Published as a conference paper at ICLR 2024
Supplementary Material
A Broader Impacts
17
Related Works and Limitations
17
B.I Limitations
19
C Proofs of Main Text
20
C.I
Lemma 3.1
20
C.2 Lemma 3.2
20
C,3 Lemma 3.3
C.4 Theorem 3.1
22
C.5
Lemma 3.4
22
Methodology Details
26
D.I The Multi-Step GSH Updates
26
D.2 GSHPooling and GSHLayer
26
D.3 Example: Memory Retrieval for Image Completion
................................
26
D.4 Pseudo Label Retrieval
26
D.5
Algorithm for STanHop-Net
27
E
Additional Numerical Experiments
28
E.1
Numerical Verification's Theoretical Results Computational Cost Analysis of Memory Modules Ablation Studies .u.............
................................
28
E.2
29
E.3
29
E.4
The Impact of Varying
31
E.5
Memory Usage of STanHop and learnable a
32
E.6
Time Complexity Analysis of STanHop- -ett
32
E.7
Hyperparameter Sensitivity Analysis
32
E.8
Additional Multiple Instance Learning Experiments
34
E.9
STanHop-Nett Outperforms DLinear in Settings Dominated by Multivariate Corre-
lations: A Case Study
36
F Experiment Details
38
F.1
Experiment Details Multivariate Time Series Predictions without Memory En hancements
38
F.2
External Memory Plugin Experiment Details.
40
Additional Theoretical Background
40
16

---

Published as a conference paper at ICLR 2024
A BROADER IMPACTS
We envision this approach as a means to refine large foundation models for time series, through a perspective shaped by neuroscience insights. Such memory-enhanced time series foundation models are vital in applications like eco- and climatic-modeling. For example, with a multi-modal time series foundation model, we effectively predict, detect, and mitigate emerging biological threats associated with the rapid changes in global climate. To this end, the differentiable external memory modules become handy, they allow users to integrate real-time data into pre-trained foundation models and thus enhance the model's responsiveness in real-time scenarios. Specifically, one can this memory-enhanced technique to embed historical, sudden, or rare events into any given time series foundation model, thereby boosting its overall performance.
B RELATED WORKS AND LIMITATIONS
Transformers for Time Series Prediction. As suggested in Section and (Hu et al., 2023; Ram- sauer et al., 2020), besides the additional memory functionalities, the Hopfield layers act as promis- ing alternatives for the attention mechanism. Therefore, we discuss and compare STanHop-NNt with existing transformer-buet based time series prediction methods here.
Transformers have gained prominence in time series prediction, inspired by their success in Natural Language Processing and Computer Vision. One challenge in time series prediction is managing transformers' quadratic complexity due to the typically long sequences. To address this, many re- searchers have not only optimized for prediction performance but also sought to reduce memory and computational complexity. LogTrans (Li et al., 2019) proposes transformer-based neural network for time series prediction. They propose a convolution layer over the vanilla transformer to better capture local context information and sparse attention mechanism to reduce memory complexity. Similarly, Informer (Zhou et al., 2021) proposes convolutional layers in between attention blocks to distill the dominating attention and sparse attention mechanism where the keys only attend to subset of queries. Reformer (Kitaev et al., 2020) replaces the dot-product self-attention in the vanilla transformer with a hashing-based attention mechanism reduce the complexity. Besides directly time se-
feeding the raw time series inputs to the model, many works focus on transformer-base ries prediction by modeling the decomposed time series. Autoformer (Wu et al., 2021) introduces model to separately model the seasonal com-
series decomposition module to its transformer- ponent and the trend-cyclical of the time series. FEDformer (Zhou et al., 2022) also models the decomposed time series and they introduce a block to extract signals by transforming the time series to the frequency domain.
Compared to STanHop, the above methods do not model multi multi-resolution information. Besides, Reformer's attention mechanism sacrifices the global receptive field compared to the vanilla self- attention mechanism and our method, which harms the prediction performance.
Some works intend to model the multi-resolution or multi-scale signals in the time series with a dedicated network design. Pyraformer (Liu et al., 2021a) designs pyramidal attention module to extract the multi-scale signals from the raw time series. Crossformer (Zhang and Yan, 2022) proposes multi-scale encoder-decoder architecture to hierarchically extract signals of different resolutions from the time series. Compared to these methods. STanHop adopts a more fine- fine-grained multi-resolution modeling mechanism that is capable of learning different sparsity levels for signals in the data of different resolutions.
Furthermore, all of the above works on time series prediction lack the external memory retrieval module as ours. Thus, our STanHop method and its variations have a unique advantage in that we have a fast response to real-time unexpected events.
Hopfield Models and Deep Learning. Hopfield Models (Hopfield, 1984; 1982; Krotov and Hop- field, 2016) have garnered renewed interest in the machine learning community due to the connection between their memory retrieval dynamics and attention mechanisms in transformers via the Modern Hopfield Models (Wu et al., 2024; Hu et al., 2024b; 2023; Ramsauer et al., 2020). Furthermore, these modern Hopfield models enjoy superior empirical performance and possess several appeal- ing theoretical properties, such as rapid convergence and guaranteed exponential memory capacity. By viewing modern Hopfield models generalized attentions with enhanced memory functionali-
17

---

Published as a conference paper at ICLR 2024
ties, these advancements pave the way for innovative Hopfield-centiie architectural designs in deep learning (Hu et al., 2024a; Hoover et al., 2023; Seidl et al., 2022; Fürst et al., 2022; Ramsauer et al., 2020). Consequently, their upplicabilitty spans diverse areas like physics (Krotov, 2023). biol- ogy (Schimunek et al., 2023; Kozachkov et al., 2023; Widrich et al., 2020), reinforcement learning (Paischer et al., 2022). tabular learning (Xu et al., 2024), and large language models (Hu et al., 2024a; Fürst et al., 2022).
This work pushes this line of research forward by presenting Hopfield-based deep architecture (StanHop-Net) tailored for memory-enhanced learning in noisy multivariate time series. In partieu- lar, our model emphasizes in-context memorization during training and bolsters retrieval capabilities with an integrated external memory component.
Sparse Modern Hopfield Model. Our work extends the theoretical framework proposed in (Hu al., 2023) for modern Hopfield models. Their primary insight is that using different entropic reg- ularizers can lead to distributions with varying sparsity. Using the Gibbs entropic regularizer, they reproduce the results of the standard dense Hopfield model (Ramsauer et al., 2020) and further pro- pose a sparse variant with the Gini entropic regularizer, providing improved theoretical guarantees. However, their sparse model primarily thrives with data of high intrinsic sparsity. To combat this, we enrich the link between Hopfield models and attention mechanisms by introducing learnable spar- sity and showing that the sparse model from (Hu et al., 2023) is a specific case of our model when setting a 2. Unlike (Hu et al., 2023), our generalized sparse Hopfield model ensures adaptable sparsity across various data types without sacrificing theoretical integrity. By making this sparsity learnable, we introduce the GSH layers. These new Hopfield layers adeptly learn and store sparse representations in any deep learning pipeline, proving invaluable for inherently noisy time series data.
Memory Augmented Neural Networks. The integration of external memory mechanisms with neural networks has emerged as a pivotal technique for machine learning, particularly for tasks requiring complex data manipulation and retention over long sequences, such as open question an- swering, and few-shot learning.
Neural Turing Machines (NTMs) (Graves et al., 2014) combine the capabilities of neural networks with the external memory access of Turing machine. NTMs use a differentiable controller (typi- cally an RNN) to interact with an external memory matrix through read and write heads. This design allows NTMs to perform complex data manipulations, akin to computer with a read-write memory. Building upon this, Graves al. (2016) further improve the concept through Differentiable Neural Computers (DNCs), which enhance the memory access mechanism using differentiable attention process. This includes a dynamic memory allocation and linkage system that tracks the relationships between different pieces of data in memory. This feature makes DNCs particularly adept at tasks that require complex data relationships and temporal linkages.
Concurrently, Memory Networks (Weston et al.. 2014) showease the significance of external mem- ory in tasks requiring complex reasoning and inference. Unlike traditional neural networks that rely on their inherent weights store information, Memory Networks incorporate separate memory matrix that stores and retrieves information across different processing steps. This capability allows the network to maintain and manipulate a "memory" of past inputs and computations, which is par- ticularly crucial for tasks requiring persistent memory, such as question-answering systems where the network needs to remember context or facts from previous parts of a conversation text to accurately respond to queries. This concept is further developed into the End-to-End Memory Net- works (Sukhbaatar al., 2015), which extend the utility of Memory Networks (Weston et al., 2014) beyond the limitations of traditional recurrent neural network architectures, transitioning them into a fully end-to-end trainable framework, thereby making them more adaptable and easier to integrate into various learning paradigms.
A notable application of cemnory-augmented neural networks is in the domain of one-shot learning. The concept of meta-learning with memory-augmented neural networks, as explored by Santoro et al. (2016), has demonstrated the potential of these networks to rapidly adapt to new tasks by lever- aging their external memory, highlighting their versatility and efficiency in learning. They showcase the potential of these networks to adapt rapidly to new tasks, a crucial capability in scenarios where data availability is limited. Complementing this, Kaiser et al. (2017) foeus on enhancing the recall of
18

---

Published as a conference paper at ICLR 2024
rare events and is particularly notable for its exploration of eemory-aaggentter neural networks de- signed to improve the retention and recall of infrequent but significant occurrences, highlighting the potential of external memory modules in handling rare data challenges. This is achieved through unique soft attention mechanism, which dynamically assigns relevance weights to different memory entries, enabling the model to draw broad spectrum of stored experiences. This approach not only facilitates the effective recall of rare events but also adapts to new data, ensuring the memory's 's relevance and utility over time.
In all above methods, (Kaiser et al., 2017) is closest to this work. However, our approach diverges in two key aspects: (Enhanced Generalization): Firstly, our external memory enhancements are external plugins with option for fine-tuning. This design choice avoids over-specialization on rare events, thereby broadening our method's applicability and enhancing its generalization capabilities across various tasks where the frequency and recency of data are less pivotal. (Adaptive Response over Rare Event Memorization): Secondly, our approach excels in real-time adaptability. By integrating relevant external memory sets tailored to specific inference tasks, our method can rapidly respond and improve performance, even without the necessity prior learning. This flexibility contrasts with the primary focus memorizing rare events in (Kaiser et al., 2017).
B. LIMITATIONS
The proposed generalized sparse modern Hopfield model shares the similar inefficiency due to the (dd) complexity'. In addition, the effectiveness of our memory enhancement methods is contin- gent on the relevance of the external memory set to the specific inference task. Achieving a high degree of relevance in the external memory set often necessitates considerable human effort and domain expertise, just like our selection process detailed in Appendix F.2. This requirement could potentially limit the model's applicability in scenarios where such resources scarce unavail- able.
As side note, Hu et al. (2024c) provides a characterization of the computational efficiency of all possible efficient variants of the modern Hopfield model from the fine-grained complexity theory.
19

---

Published as a conference paper at ICLR 2024
C PROOFS OF MAIN TEXT
C.I LEMMA 3.1
Our proof relies on verifying that q meets the criteria of Danskin's theorem.
Proof of Lemma 3.1. Firstly, we introduce the notion of convex conjugate.
Definition C.I. Let (p, 2) == (p, 2) Toa (p). The convex conjugate of Vaa V takes the form:
(T"(z) Max (p, PEA
(p)
Max F(p, PEAM
(C.1)
By Danskin's theorem (Danskin, 2012; Bertsekas et al., 1999). the function V is convex and its partial derivative with respect to z is equal to that of F, i.e. 0v' /00/ = 3F/0,, if the following three conditions satisfied for T/* and F:
(i) F(p, 2) : R - is a continuous function, where P C IR M a compact set.
iii) F' is convex in i.c. for each given p P, the mapping F(p, z) is convex.
(iii) There exists an unique maximizing point such that F(p. (). Maxppp F(p,z).
Since both (p. z) and " are continuous functions and every component of is ranging from 0 to 1, the function F' is continuous and the domain P is a compact set. Therefore. condition (i) is satisfied.
Since we require E A (i.e. P = A to be probability distributions, for any fixed P' z) (P. z) y (p) reduces to an affine function depending only on input z. Due to the inner product form, this affine function is convex in and hence condition (ii) holds for all given P = A
Since, for any given a-EntMax only produces one unique probability distribution p", condition (iii) is satisfied. Therefore, from Danskin's 's theorem, it holds
ar θ ((P.2) v/((p) = - a-Fnttax(2))
(C.2)
Oz Oz
C.2 LEMMA 3.2
Our proof is built on (Hu al., 2023, Lemma 2.1). We first derive T by utilizing Lemma 3.1 and Remark G.I, along with the convex-concave procedure (Yuille and Rangarajan, 2003; 2001). Then, show the monotonicity of minimizing H with T by constructing itera- tive upper bound of which is convex X,+1 and thus, can be lowered iteratively by the convex-concave procedure.
Proof. From Lemma 3.1, the conjugate convex of V, Vr,, is always convex, and, therefore, T concave function. Then, the energy function H. defined in (3.2) is the sum of the convex function H.(x)) (x) = 1 (x, x) and the concave function H2 (x) = 4 (E"x).
Furthermore, by definition, the energy function H. is differentiable.
Every iteration step of convex-concave procedure applied on H. gives
Hi (x++1)= -
V H2 (x:),
(C.3)
which implies that
X4+1
Ex,) E Eo-Fnt (ET (ETx.).
(C.4)
20

---

Published as a conference paper at ICLR 2024
On the basis of (Yuille and Rangarajan, 2003; 2001), we show the decreasing property of (3.2) over t via solving the minimization problem of energy function:
Min [H(x)]
Min [Hi + H2(x)).
(C.5)
which, in coneex- procedure, is equivalent solve the iterative programming
X++1
ArgMin [Hi(x) (x. Xx (x))),,
(C.6)
for all L. The concept behind this programming is to linearize the concave function H2 around the solution for current iteration, X,, which makes H1 (Xx+1) (x1+1, vxH2(xx)) convex in X(+1-
The convexity of Hi and concavity of H2 imply that the inequalities
(x)x) 2 Hyly) + ((x-)) y), vxHi(y)), H2(x) < Hy + y) VxH2(y)),
(C.7)
(C.8)
hold for all y,y, which leads to
H(() = H1(x) + H2(x)
(C.9)
< Hx)x) + H2ly) + ((x y), xxH2(y)) = Ho (x,y),
(C.10)
where the upper bound of H is defined as Hu. Then, the iteration (C.6)
X(+1 E ArgMin [Hy (x, x;) = Arg.in [1 + (x, V H2(x)))
(C.11)
make Hy decrease iteratively and thus decreases the value of energy function H monotonically,
i.e.
H(x++1) S Hx((x,1, xx) S Hy (x1, x,) H(x:),
(C.12)
for all t. Equation (C.10) shows that the retrieval dynamics defined in (3.2) lead the energy function H. to decrease with respect to the increasing t.
C.3 LEMMA 3.3
To prove the convergence property of retrieval dynamics T, first we introduce an auxiliary lemma from (Sriperumbudur and Lanckriet, 2009).
Lemma C.1 (SSriperumbudur and Lanckriet, 2009), Lemma 5). Following Lemma 3.3, x is called the fixed point of iteration T with respect to H if = T(x) and is considered as a generalized fixed point of T if Txx). x is generalized fixed point of T, then, x" is stationary point of the energy minimization problem (C.5).
Proof. Since the energy function monotonically decreases with respect to increasing in Lemma 3.2, we can follow (Hu et al., 2023, Lemma 2.2) to guarantee the convergence property of T by checking the necessary conditions Zangwill's global convergence. After satisfying these conditions, Zangwill global convergence theory ensures that all the limit points of {x are gen- eralized fixed points of the mapping and it holds limg-+00 (xx) H (x*), where x* are some generalized fixed points of T. Furthermore, auxiliary Lemma C.1 implies that x* are also the sta- tionary points of energy function H. Therefore, we guarantee that can itcratively lead the query converge to the local optimum of H.
L
21

---

Published as a conference paper at ICLR 2024
C.4 THEOREM 3.1
Proof. Compared with Tease. T only sample partial memory patterns near Exx which gives the tighter bound:
[T((x) - Enul
|| Dennse (x) Exil.
(C.13)
For 2 2 1:
Then, we derive the upper bound on [denss (x) Eilll based on (Hu al., 2023,
Theorem 2.2):
M Σ
T (x) - -Eull- =
[Sotmmax
(B x)]uEu En
(BE
(C.14)
M
Softmax(EETx)]_ee x)]uEx - Softmax(8E"x)),,
(C.15)
p=1uzz
S 2
(C.16)
where ē = (M - 1) exp{ - - (M - 1) exp ((S,. x) Maxxe[M] (x, Ex)) J. Con-
sequently, (3.5) results from above and (Ramsauer et al., 2020, Theorem 4.5).
For 2 2. Following the setting of a-FatMax in (Peters et al., 2019), the equation
2-EntMax(3EF (3 x) Sparsomax($E"xx
(C.17)
holds.
According to the closed form solution of Sparsemax in (Martins and Astudillo, 2016), it holds
1 R [[ (x)
SSparsemax (BE x)
[SE x
(C.18)
for μ E [M]. Then, the sparsemax retrieval error is
[ Sparsemaax (x)
1
Sparsemax (BE
EE" Σ E(v) [Sparsemax (BE x)) (v) E"
E(v)
< m + m3
Σ
[ (v)
[='xx]aa (1s)
+
(By (C.18))
3x
m
1
S m + /m3
k
Max (Ev, x) PE[M]]
| [='x] (n))
(C.19)
B
By the first inequality of Theorem 3.1, for 2, have
Ix Max (Eu, VE (EE(M)
1
[x((x) - Epul S | Sparssemux (x) E m d/
[xTx
3
which completes the proof of (3.6).
C.5
LEMMA 3.4
Our proof, built on (Hu et al., 2023, Lemma 2.1), proceeds 3 steps:
(Step 1.) We establish a more refined well-separation condition, ensuring that patterns (E are well-stored in H and can be retrieved by T with an emror at most R.
22

---

Published as a conference paper at ICLR 2024
This condition is then related to the cosine similarity of memory patterns, from which we deduce an inequality governing the probability of successful pattern storage and retrieval.
(Step 2.)
(Step 3.) We pinpoint the conditions for exponential memory capacity and confirm their satisfaction.
Since the generalized sparse Hopfield shares the same well vell-separation condition (shown in below Lemma C.2), it has the same exponential memory capacity as the sparse Hopfield model (Hu et al., 2023, Lemma 3.1). For completeness, we restate the proof of (Hu al., 2023, Lemma 3.1) below.
Step To analyze the memory capacity of the proposed model, we first present the following two auxiliary lemmas.
Lemma C.2. [Corollary 3.1.1 of (Hu et al., 2023)] Let 5 Dense Eull 1 Eull Then, the well-separation condition can be formulated as:
2(M 1)m
In
+ 2mH.
(C.20)
R 8
Furthermore, 5 = 0, this bound reduces to well-separation condition of naftmax-basse Hopfield model.
Proof of Lemma C.2. Let Tense be the retrieval dynamics given by the dense modern Hopfield model (Ramsauer et al., 2020), and []]((x) Ex and DiTeee (x) - Enll be the retrieval error of generalized sparse and dense modern Hopfield model, respectively. By Theorem 3.1, have
[x(x) - Euul S [|T (x) - Eull-
(C.21)
By (Ramsauer et al., 2020, Lemma A.4), we have
|eeme Exll 22mm,
(C.22)
where == (M - 1) exp - = (M - 1) exp ((Ep. x) Maxxe[M] (E E. Then, by the Cauchy-Schwartz inequality
VE... (x. E.>I < ||E, x|| ||E,|| S ErE, x||m, Vu E [M],
(C.23)
we observe that A, can be expressed terms of A
A < A 2|| x x|Im Ap 2mR,
(C.24)
where R is radius of the sphere S,.. Thus, inserting the upper bound given by (C.22) into (3.5), we obtain
[T(x) Eaal 5 | (x) Euul < 2ēm
(C.25)
5 2(M - 1) exepp(-8 (A,, 2mR)
(C.26)
Then, for any given å = | (x) - E - [|T(x) - EE,ll S 0, the retrieval error []T(x) has an upper bound:
E,,
[x() Eاسرع S 2(M 1) expl -3 (A, - 2mR - 8) - 8 < | (x) E
(C.27)
Therefore, for to mapping T Sp + Sys we need the well-separation condition
2(M 1)m
A p >
+ 2m
(C.28)
3
R +
23

---

Published as a conference paper at ICLR 2024
Lemma C.3 ((Hu et al., 2023; Ramsauer et al., 2020)). If the identity
ac + cinc - b = O,
(C.29)
holds for all real numbers a, b E IR, then c takes a solution:
b
(C.30)
Wolexp(a + bn)
Proof of Lemma C.3. We restate the proof of (Hu et al., 2023, Lemam 3.1) here for completeness.
With the given equation ac + clnc e = O, we solve for c by following steps:
ac + clnc b 0, b e²
a + In
In
= a + Inb,
b
exp
expla + lnb),
Wolexp(a Inb)),
Wolexpla + bn))
Then, we present the main proof of Lemma 3.4.
Proof of Lemma 3.4. Since the generalized Hopfield model shares the same well-separation condi- tion the sparse Hopfield model (Hu al., 2023), the proof of the exponential memory capacity automatically follows that of (Hu et al., 2023). We restate the proof of (Hu et al., 2023, Corol- lary 3.1.1) here for completeness.
(Step 2.) & (Step 3.) Here we define Amin and fur Amin == Min Ap and the angle between two patterns &o and g", respectively. Intuitively, 0ur E [0, nT] represent the pairwise corre- lation of patterns the two patterns and hence
A min
Min INSUEM
[m (1 (=-oos(Axx)) m² [1 cos cosmmmaa)),
(C.31)
where Omin
Min Min Opz E [0,T]. T).
From the well-separation condition (C.2), we have
1 ln
2(M 1)m
A A min
+ 2mR
(C.32)
3
R
Hence, we have
2(M 1)m
m. cos(O
2 lm
+ 2m R.
(C.33)
R +
24

---

Published as a conference paper at ICLR 2024
Therefore, are able to write down the probability of successful storage and retrieval, i.. minimal separation Amin satisfies Lemma C.2:
2(M 1)m
P m 2 [1 - cos(@min)
+ 2mR = 1
(C.34)
R +
By (Olver al., 2010, (4.22.2)), it holds
8 min
cos(Omin) 5
for 0 S cos((mii)) $
(C.35)
and hence
V-M
2 - 1)m
+ 2mR 7 +) = - p.
P
M
Omnn
B In
(C.36)
m
R +
Here introduce M on both sides in above for later convenience.
Let wa
2xr be the surface area of d-dimensional unit sphere, where r(+) represents the
T( -
gamma function. By (Brauchart al., 2018, Lemma 3.5), holds
d-1
- 2--15"" " 2 (d-1)
2(M 1)m
2
1-p21-
+ 2mn
(C.37)
3
R +
where y is characterized as ratio between surface areas of the unit spheres (d 1) and dimensions, respectively: 7a == w2
Since M VDC d=1 is always true for d, M E N4, p E [0, 1] and some real values C E R, we have
(VDC== m
d1
5 C m- (d-1) In
11 1.
(C.38)
3
R+5
Then, we rearrange above as
5C
2
d-1 (/-*+111
VDC
-1)
In
+1 +1
15 0,
(C.39)
m
R+
and identify
2m /pp -1)
4m 23
a ==
In
+ +13.
=
(C.40)
d
R R+
3(d 1)
By Lemma C.3, we have
C= b
(C.41)
Wolexpla Inb))
where Wo(+) is the upper branch of the Lambert W function. Since the domain of the Lambert W function is x > (-1/e, oo) and the fact expfa + In b} > 0. the solution for (C.41) exists. When the inequality (C.38) holds, arrive the lower bound on the exponential storage capacity M:
M 2 =
(C.42)
25

---

Published as a conference paper at ICLR 2024
In addition, by the asymptotic expansion of the Lambert W function (Hu et al., 2023, Lemma 3.1), also holds M 2 Mpense. where MDnse is the memory capacity of the dense modern Hopfield model (Ramsauer al., 2020).
D METHODOLOGY DETAILS
D.1 THE MULTI-STEP GSH UPDATES
GSH inherits the capability of multi-step update for better retrieval aceuracy, which summarized in below Algorithm for given number of update steps In practice, we find that single update suffices, consistent with our theoretical finding (3.5) of Theorem 3.1.
Algorithm Multi-Step Generalized Sparse Hopfield Update for CSH
Require: A. E R 1, Q
Rllno Do K E Relen x Dx Wv E R Rxxd WK E PxxDa
for i + 1 to do
Q new a-EEntMax (BQ(KWk)T) K
Hopfield Update
Q end for Z QWv return
new
D.2
GSHPooling AND GSHLayer
Here we provide the operational definitions the GSHPooling and the GSHLayer.
Definition D.1 (Generalized Sparse Hopfield Pooling (GSHPooling)). Given inputs Y pleny x Dy, and lenq query patterns Q E x Di the 1-step Sparse Adaptive Hopfield Pooling update is
GSHPooling (Y) a-EntMax /QKT/ V,
(D.I)
Here we have K, V equal to V WW&WV. = YW K. and Wy E R Wx >Dx Where d is the dimension of K. And the query pattern is learnable variable, and is independent from the input, the size of lenq controls how many query patterns we want to store.
Definition D.2 (Generalized Sparse Hopfield Layer (GSHLayer)). Given inputs Y E Rleny Dr and lenq query patterns Q E Blong Dx the 1-step Sparse Adaptive Hopfield Layer update is
GSHLayer (R, Y) a-EntMax (RYT) / Ds) Y,
(D.2)
Here R. is the input and Y can be either learnable weights given an input.
D.3 EXAMPLE: MEMORY RETRIEVAL FOR IMAGE COMPLETION
The standard memory retrieval mechanism Hopfield Models contains two inputs, the query x and the associative memory set E. The goal is to retrieve associated memory & most similar to the query x from the stored memory set E. For example, in (Ramsauer et al., 2020), the query x is corrupted/noisy image from 10. and the associative memory set E\t is the CIFARI0 image dataset. All images are flattened into vector-valued patterns. This task can be achieved by taking the query R. = and the associative memory After steps of updates, we expect the output of the GSHLayer to be the recovered version of x.
Y E for GSHLayer with fixed parameters.
D.4 PSEUDO LABEL RETRIEVAL
Here, present the of the memory retrieval mechanism from modern Hopfield models to generate pscudo-labels for queries R, thereby enhancing predictions. Given a set of memory patterns Y and their corresponding labels Y tabel, concatenate them together form the label-included
26

---

Published as a conference paper at ICLR 2024
memory set Y. Take CIFARIO for example, one-hot encoded labels together as the memory set. For the query, we use the input with padded zeros concatenated the end of it. The goal here is to "retrieve" the padding part in the query, which is expected to be the retrieved "pseudo-label". Note that this pseudo-label will be a weighted all other labels in the associative memory set. An illustration of this mechanism can be found in Figure 2. For the retrieved pscudo-label, we can either use it as the final prediction, or use it pseudo-label to provide extra information for the model.
concatenate the flatten images along with their
D.5 ALGORITHM FOR STANHOP-NET
Here we summarize the STanHop-Nett as below algorithm.
Algorithm 2 STanHop-Neet
Require: L 2 1, Z RTxxXx D Piadden
for + to L do
Z STanHop (Coarse-Graining (Zer A))
encoder forward
end for Zo
Edoe
learnable positional embedding
for
+ to L do
decoder forward
z dee STanHop Z
2-11
dec
Z
dec GSH
(Z
Ze enc
Z
dec LayerNorm
El000 + ZLco
Z LayerNorm
2o00 MLP (Z
end for
return L E IR. 2xxxx xxxxDrxx
27

---

Published as a conference paper at ICLR 2024
E ADDITIONAL NUMERICAL EXPERIMENTS
Here we provide additional experimental investigations to back up the effectiveness of our method.
E.1 NUMERICAL VERIFICATION'S S OF THEORETICAL RESULTS
Faster Fixed Point Convergence and Better Generalization. In Figure 4, support our theo- retical results in Section 4, we numerically analyze the convergence behavior of the GSH, compared with the dense modern Hopfield layer Hopfield.
0:45
0.50
STon-lop
0.40
0:45
Dah kog (MSE)
(MS58)
0.85
0.48
0.30
0.25
0.42
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.
2.5
5.0
7.5
10.0 12.5
15.0
17.5
20.0
spoot
epach
Figure 4: The training and validation loss curves of STanHop (D), i.e. STanHop-NNt with dense modern Hopfield Hopfield layer, and STanHop-Net with GSH layer. The results show that the generalized sparse Hopfield model enjoys faster convergence than the dense model and also obtain better generalization.
In Figure 4, we plot the loss curves for STanHop-Net using both generalized sparse and dense modern models on the ETThl dataset for the multivariate time series prediction tasks.
The results reveal that the generalized sparse Hopfield model (GSH) converges faster than the dense model (Hopfield) and also achieves better generalization. This empirically supports our theoret- ical findings presented in Theorem 3.1, which suggest that the generalized sparse Hopfield model provides faster retrieval convergence with enhanced accuracy.
Memory Capacity and Noise Robustness. Following (Hu et al., 2023), we also conduct exper- iments verifying our memory capacity and noise robustness theoretical results (Lemma 3.4 and Theorem 3.1), and report the results in Figure 5. The plots present average values and standard deviations derived from 10 trials.
Memory Capacity (MNIST)
Momory Rob sstness (MNIST)
1.2
Spane
0.8
thet at Sucree
DVG
0.4
02
02
42
50
60 100
Niemory
Figure 5: Left: Memory Capacity measured by successful half-masked retrieval rates. Right: Memory Robustness measured by retrieving patterns with various noise levels. A query pattern considered accurately retrieved its cosine similarity enrrr falls below a specified threshold. We set error threshold 20% and $=0.01 for better visualization. We plot the average and variance from 10 trials. These findings demonstrate the generalized sparse Hopfield model's ability of capturing data sparsity. improved memory capacity and its noise robustness.
Regarding memory capacity (displayed the left side of Figure 5), we evaluate the generalized sparse Hopfield model's ability to retrieve half-masked patterns from the MNIST dataset, com- parison to the Dense modern Hopfield model (Ramsauer et al., 2020).
28

---

Published as a conference paper at ICLR 2024
Regarding robustness against noisy queries (displayed the right side of Figure 5), introduce Gaussian noises of varying variances (σ)) to the images.
These findings demonstrate the generalized sparse Hopfield model's ability of capturing data spar- sity, improved memory capacity and its noise robustness.
E.2 COMPUTATIONAL COST ANALYSIS OF MEMORY MODULES
Here we analyze the computational cost between the Plug-and-Plan memory plugin module and the baseline. We evaluate 2 matrices: (i) the number of floating point operations (flops) (ii) number of parameters of the model. Note that for Plug-and- affected by the size external memory set. The result can be found in Figure 6 and Figure 7.
module, the parameter amount will not be
20000
4000
25000
foating paint
2500
22000
Namterr www.://1000000-0000000
2000
30000
1500
34
Number memory
momory
Figure 6: The number of floating-point op- erations (flops) (in millions) comparison be- tween Plug-and-Play, Tune-and-Play and the baseline. The result shows that the Plug-and- Play. Tune-and-Play successfully reduce the required computational cost to process an in- creased amount of data.
Figure 7: The number of Multiply-accumulate operations (MACs) (in millions) comparison be- tween Plug-and-Play Tune-and-Play and the baseline. The result shows that both of our memory plugin modules face little MACs in- creasement while the baseline model MACs crease almost linearly w.r.t. the input size.
E.3 ABLATION STUDIES
Hopfield Model Ablation. Beside our proposed generalized sparse modern Hopfield model, also test STanHop-Net with 2 other existing different modern Hopfield models: the dense modern Hopfield model (Ramsauer et al., 2020) and the sparse modern Hopfield model (Hu et al., 2023).
We report their results in Table 1.
We terms them as STanHop-Net (D) and STanHop-Net (S) where (D) and (S) are for "Dense" and "Sparse" respectively.
Component Ablation. order to evaluate the effectiveness different components in our model, perform ablation study by removing Embedding as (PE), StanHop as (SH), Hopfield Pooling as (HP), Multi-Resolution as (MR). We also denote their removals with "w/o" (i.e., without.)
component time. In below, denote Patch
For w/o PE, we set the patch size P equals 1. For w/o MR, we set the coarse level A as 1. For w/o SH and w/o HP, replace those blocks/layers with an MLP layer with GELU activation and layer normalization.
The results are showed in Table 3. From the ablation study results, observe that removing the STanHop block gives the biggest negative impact on the performance. Showing that the STanHop block contributes the most the model performance. Note that patch embedding also provides notable improvement on the performance. Overall, every component provides a different level of performance boost.
29

---

Published as a conference paper at ICLR 2024
Table 3: Component Ablation. conduct component ablation by separately removing Patch Embedding (PE), STanHop (SII), Hopfield Pooling (HIP), and Multi-Resolution (MR). We report the mean MSE and MAE over 10 runs, with variances omitted as they are all S 0.15%. The results indicate that while every single component in STanHop provides performance boost, the impact of STanHop block model performance the most significant among other components.
Models
STanHop
w/o PE
w/o SH
w/o HP
w/o MR
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.294
0.360
0.318
0.368
0.305
0.365
0.306
0.363
0.307
0.363
48
0.340
0.387
0.357
0.389
0.352
0.393
0.346
0.387
0.348
0.385
168
0.420
0,452
0.454
0.476
0.480
0.500
0.434
0.455
0.447
0,464
336
0.450
0.472
0.501
0.524
0.530
0.535
0.462
0.473
0.482
0.486
720
0.512
0.520
0.540
0.538
0.610
0.581
0.524
0.526
0.537
0.531
24
0.292
0.341
0.318
0.365
0.335
0.375
0.340
0.374
0.325
0.373
48
0.363
0.402
0.386
0.421
0.414
0.439
0.385
0,420
0.391
0.427
168
0.499
0.515
0.504
0.521
0.507
0.519
0.503
0.525
0.520
0.509
336
0.499
0.515
0.514
0.529
0.532
0.541
0.513
0.528
0.533
0.542
720
0.548
0.556
0.570
0.565
0.569
0.565
0.539
0.548
0.555
0.557
30

---

Published as a conference paper at ICLR 2024
E.4 THE IMPACT OF VARYING
We examine the impact of increasing the value of on memory capacity and noise robustness. is known that as approaches infinity. the a-enimax operation transitions to hardmax operation (Peters al., 2019; Correia al., 2019). Furthermore, it is also known that memory pattern retrieval using hardmax is expected to exhibit perfect retrieval ability, potentially offering larger memory capacity than the softmax modern Hopfield model pure retrieval tasks (Millidge al., 2022). Our empirical investigations confirm that higher values of a frequently lead to higher memory capacity. We report results only up to a 5, as we observed that values of greater than 5 consistently lead to numerical errors, especially under float32 precision. It is crucial to note, that while the hardmax operation (realized when a + oo) may maximize memory capacity, its lack of diffcrentiability renders it unsuitable for gradient descent- based optimization.
Memory Capacity (CIFAR10)
Memory Robustness (CIFAR10)
0.8
acthe22
Retreval Success Rabe
0.8
Ratrival
0.4
0.4
0.0
0.0
Number
Variance
Figure 8: Left: Memory Capacity measured by successful half-masked retrieval rates w.r.t different values of CIFARIO. Right: Memory Robustness measured by retrieving patterns with various noise levels CIFARIO. A query pattern is considered accurately retrieved if its cosine similarity error falls below a specified threshold. We set error threshold of 20% and & 0.01 for better visualization. We plot the average and variance from 10 trials. We can see that using hardmax (argmax) normally gives the best retrieval result as it retrieves only the most similar pattern w.r.t. dot product distance. And setting 5 approximately gives the similar result while having a keeps the overall mechanism differentiable. Memory Robustness (MNIST)
Memory Capacity (MNIST)
1.0
Retree Retieva Success Rate
Batreva Success
8.8
0.4
0.4
0.0
0.0
Number
Variance (2))
Figure 9: Left: Memory Capacity measured by successful malf-masked retrieval rates w.r.. different values of MNIST. Right: Memory Robustness measured by retrieving patterns with various noise levels on MNIST. query pattern considered accurately retrieved set error threshold of 20% and B = 0.1 better visualization. We plot average and variance from 10 trials. We can see that using hardmax (argmax) normally gives the best retrieval result as it retrieves only the most similar pattern w.r.t. dot product distance. And setting approximately gives the similar result while having a 5 keeps the overall mechanism differentiable.
cosine similarity error falls below specified threshold.
31

---

Published as a conference paper at ICLR 2024
E.5 MEMORY USAGE OF STANHOP AND LEARNABLE
To compare the memory and GPU usage, benchmark STanHop with STanHop-(D) (using dense modern Hopfield layers). In below Figure 10 and Figure 11, we report the footprints of from the weight and bias (wandb) system (Biewald al., 2020). The figures clearly demonstrate that the computational cost associated with learning an additional a for adaptive sparsity is negligible.
GPU Hemory Allocated (50)
GPU Ucllaaattn PS Mocner Stantip
300
Tineteretal
Tincfrrntel
Figure 10: The GPU memory allocation be- tween STanHop-Net with and without learnable alpha (STanHop-Net (D)). We that with learnable alpha does not significantly increase reuired GPU memory.
Figure 11: The percentage of gpu utilization be- tween STanHop-Net with and without learnable alpha (STanHop-Net (D)). We can that with learnable alpha does not significantly increase the GPU utilization.
E.6 TIME COMPLEXITY ANALYSIS OF STANHOP-NET
Here we use the same notation as introduced in the main paper. Let 7 be the number of input length on the time dimension, Duidden be the hidden dimension, P be patch size, Dout be the prediction horizon, lenq be the size of query pattern in GSHPooling.
Patch embedding: (DDiideen T) OTT).
Temporal and Cross series GSH: O hidden
x T P O(T - = O(T3)
Coarse graining: ( hidden x T) OZ)
GSHPooling ODDnidoon x lenq x 2 x P-2) (lenq x T
PlugMemory: O(T2)
TuneMemory: O(TT + Dmmi)?
Additionally, the number parameters of STanHop-Net 0.78 million. As a reference, with batch size of 32, input length of 168, STanHop-Net requires minutes per epoch for the ETTh1 dataset. Meanwhile, STanHop-Net (D) also requires minutes per epoch under same setting.
E.7 HYPERPARAMETER SENSITIVITY ANALYSIS
We conduct experiments exploring the parameter sensitivity of STanHop-Net on the ILI dataset. Our results (in Table 5 and Table 6) show that STanHop-Net is not sensitive to hyperparameter changes.
The way that conduct the hyperparameter sensitivity analysis is that to show the model's sensi- tivity to a hyperparameter h, we change the value of in each run and keep the rest of the hyperpa- rameters' values default and we record the MAE and MSE the test set. We train and evaluate the model 3 times for 3 different values for each hyperparameter. We analyze the model's sensitivity hyperparameters respectively. We conduct all the experiments the ILI dataset.
32

---

Published as a conference paper at ICLR 2024
Table 4: Default Values of Hyperparameters in Sensitivity Analysis
Parameter
Default Value
seg_len (patch size)
window_siz (coarse level)
2
layes (number of encoder layers)
model
32
(feedforward dimension)
64
heads (number of heads)
2
Table 5: MAEs for each value of the hyperparameter with the rest the hyperparameters as default. For each hyperparameter, each row's MAE score corresponds to the hyperparameter value inside the parentheses the same order. For example, when 1e-3, the MAE score is 1.313.
seg
window size (2,4,8)
layers (3,4,5)
model
d ff (64, 128, 256)
heads (2,4,8)
(Ie-3, lc-4, Ic-5)
(6, 12, 24)
(32, 64, 128)
1.313
1.313
1.313
1.313
1,313
1.313
1.313
1.588
1.311
1.285
1.306
1.235
1.334
1.319
1.673
1.288
1.368
1.279
1.372
1.302
1.312
Table MSEs for each value of the hyperparameter with the rest of the hyperparameters as default. For each hyperparameter, each row's MSE score corresponds to the hyperparameter value inside the parentheses the same order. For example, when 1e-3, the MSE score 3.948.
seg
window size (2,4,8)
e layers (3,4,5)
d model (32, 64, 128)
n heads (2,4,8)
(Ie-3, Ic-4, le-5)
(6, 12, 24)
(64, 128, 256)
3.948
3.948
3.948
3,948
3,948
3,948
3.948
5.045
3,968
3.877
3,915
3.566
3,983
3.967
5.580
3.865
4.267
3.834
4.078
3.866
3.998
33

---

Published as a conference paper at ICLR 2024
E.8 ADDITIONAL MULTIPLE INSTANCE LEARNING EXPERIMENTS
We also evaluate the efficacy of the proposed GSH layer on Multiple Instance Learning (MIL) tasks. In essence, MIL is type of supervised learning whose training data are divided into bags and labeling individual data points is difficult or impractical, but bag-level labels are available (Ilse et al., 2018). We follow (Ramsauer et al., 2020; Hu et al., 2023) to conduct the multiple instance learning experiments on MNIST.
Model. We first flatten each image and use fully connected layer to project each image to the embedding space. Then, we perform GSHPooling and use linear projection for prediction.
Hyperparameters. For hyperparameters, dropout as 0.3, training epoch as 100, optimizer as AdamW, initial learning rate as le - 4, and we also the cosine annealing learning rate decay.
hidden dimension of 256, number of head as 4,
Baselines. We benchmark the Generalized Sparse modern Hopfield model (GSH) with the Sparse (Hu al., 2023) and Dense (Ramsauer al., 2020) modern Hopfield models.
Dataset. For the training dataset, we randomly sample 1000 positive and 1000 negative bags for each bag size. For data, we randomly sample 250 positive and 250 negative bags for each bag size. We set the positive signal to the images of digit 9, and the rest as negative signals. We vary the bag size and report the accuracy and loss curves of 10 runs on both training and test data.
Results. Our results (shown in the figures below) demonstrate that GSH converges faster than the baselines most settings, as it can adapt to varying levels of data sparsity. This is consistent with theoretical findings in Theorem 3.1, which state that GSH achieves higher accuracy and faster fxxed-point convergence compared to the dense model.
Figure 12:
The MIL experiment with bag size From right: (1) Training data accuracy curve (2) Training data loss curve (3) Test data accuracy curve (4) Test data accuracy curve
Figure 13: The MIL experiment bag size From left to right: (1) Training data accuracy curve Training data loss curve (3) Test data accuracy curve (4) Test data accuracy curve
Figure 14: The MIL experiment with bag size From left to right: (1) Training data accuracy curve (2) Training data loss curve (3) Test data accuracy curve (4) Test data accuracy curve
34

---

Published as a conference paper at ICLR 2024
Figure 15: The MIL experiment with bag size 30. From left to right: (1) Training data accuracy curve (2)
Training data loss curve (3) Test data accuracy curve (4) Test data accuracy curve
Figure 16: The MIL experiment with bag size From left right: (1) Training data accuracy curve Training loss curve (3) Test data accuracy curve (4) Test data accuracy curve
Figure 17: The MIL experiment with bag size 100. From left right: (1) Training data accuracy curve (2)
Training data loss curve (3) Test data accuracy curve (4) Test data accuracy
35

---

Published as a conference paper at ICLR 2024
E.9 STANHOP-NET OUTPERFORMS DLINEAR IN SETTINGS DOMINATED BY MULTIVARIATE CORRELATIONS: CASE STUDY
The performance of STanHop-Net in the main text, presented in Table 1, does not show a clear su- periority over DLinear (Zeng et al., 2023). We attribute this to the nature of the common benchmark datasets in Table 1, which not dominated by multivariate correlations.
To verify our conjecture, we employ a strongly correlated multivariate time series dataset as a test bed, representing a practical scenario where multivariate correlations are the predominant source of predictive information in input features. In such scenario, following the same setting in Section 5. 1, our experiments show that STanHop-Ner consistently outperforms DI We also compare with another top-tier channel-independent method, PatchTST (Nie et al., 2022), and results show that we outperform it.
Specifically, we follow the experimental settings outlined in Section 5.1, but with specific focus on cases involving small lookback windows. This emphasis aims to reduce autoregressive correlation in data with smaller lookback windows, thereby increasing the dominance of multivariate correlations. We refer just scenario as Cross-Sectional Regression' (CSR) in time series modelling (Reneau et al., 2023; Fama and French, 2020; Andrews, 2005) and (Wooldridge et al., 2016, Part 1).
Dataset. We evaluate our model on the synthetic dataset generated the ICML.2023 Feature Programming paper (Reneau et al., 2023). Feature programming is automated, programmable method for feature engineering. produces a large number of predictive features from any given input time series. These generated features, termed extended features, are by construction highly correlated. The synthetic dataset, containing 44 extended features derived from the taxi dataset (see (Reneau al., 2023, Section D.I) for the generation details), is thereby strongly correlated multivariate time series dataset.
Baseline. We mainly compare our performance with DLinear (Zeng al., 2023) as it showed comparable performance in Table 1. We also include an additional comparison with PatchTST (Nie al., 2022) to further showcase our method's advantage in capturing multivariate correlations.
Setting. For both STaHop-Nee and DLinear, we use the same hyperparameter setup we used in the ETTh1 dataset. For PatchTST, we used the same hyperparameter setup its original paper. We also conduct two ablation studies with varying lookback windows. We report the mean MAE, MSE and R2 score over runs.
Results. Our results (Table 7) demonstrate that STanHop-Ne consistently outperforms DLinear and PatchTST when multivariate correlations dominate the predictive information in input features. Importantly, our ablation studies show that increasing the lookback window size, which reduces the dominance multivariate correlations, results in DLinear's performance becoming comparable to, rather than being consistently outperformed by, STanHop-Net. This explains why DLinear hibits comparable performance to STanHop-Net in Table 1, when the datasets are not dominated by multivariate correlations.
Channel-Indee Independent and Channel-Dependent Methods. Recent studies have compared Ghannel-Independeee (CI) (Zeng al., 2023; Nie et al., 2022) and Deannel-Dependent (CD) (Yu et al., 2024; Liu et al., 2023; Chen et al., 2023; Luo and Wang, 2024) methods for time series prediction. CI methods, like DLinear and PStchTST, treat multivariate time series as multiple in- dependent uni-variate tasks, often using simpler architectures like linear layers (Zeng et al., 2023). Conversely, CD methods leverage operations like convolution and Channel-wisc attention to capture inter-channel relationships and enhance performance.
'From technical standpoint, this setting doesn't t fall under CSR, yet refer it CSR emphasize its utilization of one time slice for predicting the next. Cross-sectional regression is valuable for understanding the simultancous impact variables across different entities (i.c., multivariate correlations), providing insights that are complementary to those obtained from autoregressive time-series analysis.
whtps://wwww.rroooo.cco http://wwwww....oooooocooo//otta
36

---

Published as a conference paper at ICLR 2024
Surprisingly, CI methods perform well with minimal model complexity and parameters, and some- times even outperform CD methods, as noted in (Zeng et al., 2023; Nie et al., 2022). However, (Han et al., 2023) points out that while CD benefits from increased capacity, CT's simple architecture and limited samples in current benchmarks can significantly boost performance. Existing datasets are often too small for CD models to showcase their full potential, leading potential underestimation of CD model effectiveness on popular benchmarks like ETTh1, ILI etc.
Table 7: Comparison between DLinear, PStchTST and StanHop-Ner on the synthetic dataset gener- ated in (Reneau al., 2023). This dataset is by construction strongly correlated multivariate time series dataset. We report the mean MAE, MSE and R2 score over 5 runs. The A + B denotes the input horizon and prediction horizon B. CSR (Cross-Sectional Regression): Focuses on using single time step information from multivariate time series for predictions, Ablation1: With a prediction horizon of the lookback window increasing. thereby the dominance multivari- ate correlations is decreasing. Ablation2: With prediction horizon of 2, the lookback window increasing, thereby the dominance of multivariate correlations is decreasing. Our results aligns with expectations: STanHop-Net uniformly beats DLinear (Zeng et al., 2023) and PatchTST (Nie et al., 2022) in the Pross-Sectional Regression (CSR) setting. Importantly, our ablation studies show that increasing the lookback window size, which reduces the dominance of multivariate relations, results DLinear's performance becoming comparable to, rather than being consistently outperformed by, STanHop-Net. This explains why DLinear exhibits comparable performance to STanHop-Net in Table 1, when the datasets are not dominated by multivariate correlations.
lookback + horizon
DLinear
PStchTST
STanHop-
MSE
MAE
R²
MSE
MAE
R2
MSE
MAE
R
1 1-1
0.896
0.615
0.256
0.332
0.341
0.728
0.329
0.375
0.633
CSR
1 1-2 2
1.193
0.794
0.001
0.460
0.417
0.625
0.417
0.428
0.552
1
1.211
0.806
-0.002
0.734
0.546
0.401
0.592
0.522
0.383
1 1 8
1.333
0.868
-0.100
1.227
0.744
-0.001
0.812
0.636
0.182
1 16
1.305
0.846
-0.069
1.755
0.947
-0.409
1.028
0.734
-0.058
Ablation2 Ablation
2-1
0.514
0.504
0.573
0.332
0.355
0.729
0.328
0.366
0.710
4-11
0.373
0.417
0.690
0.332
0.357
0.729
0.328
0.364
0.712
8 $-1 1
0.328
0.380
0.727
0.322
0.361
0.737
0.327
0.367
0.715
16 1
0.319
0.372
0.736
0.324
0.371
0.735
0.323
0.361
0.717
2
0.771
0.632
0.359
0.458
0.426
0.626
0.424
0.425
0.630
4 2
0.423
0.439
0.645
0.460
0.429
0.624
0.410
0.415
0.643
8
0.647
0.441
0.646
0.439
0.429
0.642
0.402
0.412
0.655
16 2
0.419
0.435
0.652
0.447
0.446
0.635
0.435
0.433
0.626
37

---

Published as a conference paper at ICLR 2024
F EXPERIMENT DETAILS
Here present the details of experiments in the main text.
F.1 EXPERIMENT DETAILS OF MULTIVARIATE TIME SERIES PREDICTIONS WITHOUT MEMORY ENHANCEMENTS
Datasets. These datasets, commonly benchmarked in literature (Zhang and Yan, 2022; Wu al., 2021; Zhou et al., 2021).
ETT (Electricity Transformer Temperature) (Zhou et al., 2021): ETT records 2 years of data from two counties in China. We use two sub-datasets: ETThl (hourly) and ETTml (every 15 minutes). Each entry includes the "oil temperature" target and six power load features.
ECL (Electricity Consuming Load): ECL records electricity consumption (Kwh) for 321 customers. Our version, sourced from (Zhou al., 2021), covers hourly consumption over years, targeting "MT 320".
WTH (Weather): WTH records climatological data from approximately 1,600 U.S. sites be- tween 2010 and 2013, measured hourly. Entries include the "wet bulb" target and 11 climate features.
ILI (Influenza-Like Ilnness:: ILI records weekly data on influenza-like illness (ILI) patients from the U.S. Centers for Disease Control and Prevention between 2002 and 2021, depicts the ILI patient ratio against total patient count.
Traffic: Traffic records hourly road occupancy rates from the California Department of Trans- portation, sourced from sensors San Francisco Bay area freeways.
Table 8: Dataset Sources
Dataset
URL
ETThl & ETTml
ECL WTH
http://ww......ccaaaaaaae/eeeeee http://aaahiiie..iss....ddd///ls bumps
ILI Traffic
http://wwww..kkggg...c/ss//ttnne modelling
Training. We use Adam optimizer to minimize the MSE Loss. The coefficients of Adam opti- mizer, betas, are set to (0.9, 0.999). We continue training till there are Patience consecutive epochs where validation loss doesn't decrease or we reach 20 epochs. Finally, we evaluate our model on test set with the best checkpoint validation set.
Hyperparameters. For hyperparameter search, for each dataset, we conduct hyperparameter opti- mization using the "Sweep" feature of Weights and Biases (Biewald et al., 2020), with 200 iterations of random search for each setting to identify the optimal model configuration. The search space for all hyperparameters are reported in Table 9.
38

---

Published as a conference paper at ICLR 2024
Table 9: STanHop-Net hyperparameter space.
Parameter
Distribution
Patch size P
[6, 12, 24]
FeedForward dimension
[64, 128, 256]
Number of encoder layer
[1, 3]
Number of pooling vectors
[10]
Number of heads
[4, 8]
Number of stacked STanHop blocks
[1]
Dropout
[0.1, 0.2, 0.3]
Learning rate Input length on ILI Input length ETTml
[5e-4, le-4, 1e-5, le-3]
[24, 36, 48, 60)
[24, 48, 96, 192, 288, 672]
Input length on other dataset
[24, 48, 96, 168, 336, 720] [2, 4] [0.0, 0.0005, 0.001]
Course level Weight decay
39

---

Published as a conference paper at ICLR 2024
166
Time Series: OT
1.6
1.2
0.6
M M
0.2
2002
2004
2006
2008
2010
2012
2014
2016
2018
2020
Figure 18: The visualization of ILI dataset "OT" variate.
F.2 EXTERNAL MEMORY PLUGIN EXPERIMENT DETAILS
The hyperparameter of the external memory plugin experiment can be found in Table 9. For OL, we set the input length as 24, feed forward dimension as 32 and hidden dimension as 64. For prediction horizon 60, we the input length 48, feed-forward dimension 128 and hidden dimension as 256. For ETThl, we use the same hyperparameter set found via random search in Table
For the "bad" external memory set intervals, we pick 40 and 200 for ILIOOT and ETThI, which represents 40 timesteps (weeks) earlier and 200 timesteps (hours) earlier. For ILI dataset, set the memory set size as 15 and for ETTh1, we set it as 20.
For Case 3 (ETThI), we select construct the external memory pattern with interval 168 timesteps earlier (equivalent to week). For Case (ETTm1), we select construct the external memory pattern with interval 672 timesteps earlier (equivalent week).
G ADDITIONAL THEORETICAL BACKGROUND
Remark G.I. Peters al. (2019) provide closed-form expression for Max as
a-EntMax(2) = (aa - 1)z --)
(G.1)
where we denote [1]+ == max (1. 0}. and 7 the threshold function RM + R such that M 1)z- (2)) - satisfies the normalization condition probability distribution.
40


Reviews:
[{"review_id": "eV0ANDjs4u", "reviewer": "Reviewer_BpQa", "summary": "This paper extends the recent Sparse Modern Hopfield Network construction of Hu et al (2023) to use the alpha-entmax family of sparse probability mappings (alpha=1 giving Ramsauer et al softmax MHNs, alpha=2 giving the Hu et al (2023) sparsemax HSN), and prove some nice theoretical advantages of this construction.\n\nFurther, the paper constructs some neural network layers based on the proposed Generalized Sparse MHN for time series prediction and demonstrates its empirical performance in some benchmarks, with competitive performance.", "questions": "Some experiments and results that I would have liked to see and found valuable:\n - You treat $\\alpha$ as a learnable parameter: how is it parametrized and what do the learned values converge to?\n - The case $\\alpha \\to \\infty$ corresponds to using an argmax instead of softmax, i.e., retrieving the most compatible pattern in one step. How would such a \"argmax\"-based lookup memory perform like in the experiments? (Some of the gradients will be zero, but the same happens some of the times with high alpha too, even if not always.)\n\nSome theoretical issues in definitions:\n\n- in the definition of the Tsallis entropy (3.1), the bottom branch for alpha=1 seems wrong, as it gives the negative of what would be the top branch for alpha=2. I expected by continuity to define alpha=1 as the Shannon entropy -sum p log p. Am I missing something?\n\n- In equation (3.2), the definition of $\\Psi_{\\alpha}^\\star$ seems surprising: $\\alpha$-entmax is a vector-value function, thus so should be its integral, but the energy H(x) should be scalar-value. I expect (as stated also elsewhere in the paper) that $\\Psi_\\alpha^\\star$ should be the Fenchel convex conjugate of $\\Psi_\\alpha$, i.e. $\\Psi_\\alpha^\\star(z) = \\sup_{z^\\star \\in \\operatorname{dom}{\\Psi_\\alpha}} \\langle z^\\star, z \\rangle - \\Psi_\\alpha(z^\\star)$. Could you please clarify?\n\nOther questions:\n\n - It was not clear to me how the memories Y are constructed; in section 4.3 it seems like the memories must be the same length as the input sequence R. Is this a strong requirement or could it be avoided? A nice property of attention models is that they should support variable length data.\n- The qualitative change at alpha=2 in Theorem 3.1 seems surprising and interesting. Could you discuss the difference a bit, especially the difference between the Max terms that show up, and give some intuition about why we see this change? Is one of the bounds always tighter than the other or does this depend on choices of (m, M, d, beta, etc?) I did not have the time to read the proof.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The GSH construction is a nice and intuitive extension of Sparse Modern Hopfield Networks\n- Quite nice theoretical results about the GSH construction  and the impact of alpha.\n- Good empirical performance.", "weaknesses": "- Some theoretical inaccuracies (perhaps typos?) casting a bit of doubt.\n- Missing a few comparisons and reports that I would be very interested in (details below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gjCFvS8elL", "reviewer": "Reviewer_ntR2", "summary": "The work introduces a novel neural network model called STanHop-Net, which is based on the Hopfield model and offers memory-enhanced capabilities for multivariate time series prediction. The model incorporates sparsity and external memory modules, which enable it to respond quickly to sudden events and achieve good results in both synthetic and real-world settings.", "questions": "1. How does the sparsity of STanHop-Net affect its performance and memory usage?\n2. Can the author provide the time complexity and number of the parameters of the model?\n3. Can the author provide the limitation section?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is is well-written and comprehensive.\n2. The authors present case studies demonstrating the effectiveness of the model in practical applications.", "weaknesses": "1. The paper does not provide available code source to reproduce the experiments.\n2. The paper's contributions are limited in scope.\n3. The model is too complex, it's may difficult to optimize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u6Iq38X4UW", "reviewer": "Reviewer_u61L", "summary": "In this work, authors augmented hopefield network with external memory. They introduce the Sparse Tandem Hopfield Network which was tested on  multivariate time series prediction task and the proposed model exhibits improved memory capabilities. To be specific the memory module has Plug-and-Play module and a Tune-and-Play module for train-less and task-aware memory improvements. The model is theoretically motivated and series of simulation studies show proposed model achieves consistent gain compared to other transformer-based models.", "questions": "Adding memory to NNs is not a new concept, it has been out since early 90’s [1], and are even extended to modern NNs [2-4]. Similar to Hopfield networks these networks are shown to reach stable point [5-6]. Thus it is important to mention these relevant work.\n\nSecond, improving the memory capability of Hopefield network is widely studied these days [7-9], thus comparison should be done with these relevant approaches, especially 8 and 9.\n\nI would like to see a comparison against RNNs and memory-augmented RNNs, given that the proposed model is focused on time-series which is a stateful problem.\n\nFinally, what is the memory footprint? How much time model takes per epoch? Model size?\nDo you observe faster convergence? How stable is the model? All these questions should be addressed.\n\nI would like to see the variance of the model, including baseline models.\n\n\n\n1.\tDas, S., Giles, C. and Sun, G.Z., 1992. Using prior knowledge in a NNPDA to learn context-free languages. Advances in neural information processing systems, 5.\n2.\tJoulin, A. and Mikolov, T., 2015. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in neural information processing systems, 28.\n3.\tGraves, A., Wayne, G. and Danihelka, I., 2014. Neural turing machines. arXiv preprint arXiv:1410.5401.\n4.\tWeston, J., Chopra, S. and Bordes, A., 2014. Memory networks. arXiv preprint arXiv:1410.3916.\n5.\tStogin, J., Mali, A. and Giles, C.L., 2020. A provably stable neural network Turing Machine. arXiv preprint arXiv:2006.03651.\n6.\tMali, A.A., Ororbia II, A.G. and Giles, C.L., 2020. A neural state pushdown automata. IEEE Transactions on Artificial Intelligence, 1(3), pp.193-205.\n7.\tMillidge, B., Salvatori, T., Song, Y., Lukasiewicz, T. and Bogacz, R., 2022, June. Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning (pp. 15561-15583). PMLR.\n8.\tHillar, C.J. and Tran, N.M., 2018. Robust exponential memory in Hopfield networks. The Journal of Mathematical Neuroscience, 8(1), pp.1-20.\n9.\tOta, T., Sato, I., Kawakami, R., Tanaka, M. and Inoue, N., 2023, April. Learning with Partial Forgetting in Modern Hopfield Networks. In International Conference on Artificial Intelligence and Statistics (pp. 6661-6673). PMLR.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-written paper\n2. Proofs are incremental, mainly based on Hu's et al work in 2023, however, the presentation is neat.\n3. good set of experiments\n4. Hypothesis is backed by ablation study", "weaknesses": "1. Comparison against stateful models such as RNNs is missing, also, the proposed work focuses on external memory, a comparison against memory-augmented is needed.\n2. Explanations for Lemma 3.1, 3.2, and other lemmas in the main paper, can be written in a better way. Rather than stating to refer to proof, you should try and provide the simplest explanation of what each proof talks about.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BQBej3fdsf", "reviewer": "Reviewer_xuHy", "summary": "The paper presents a new transformer-like architecture (STANHOP) that uses a new form of sparse Hopfield layers. The layers use a form of Tsallis α-entropy regularization so as to induce sparse encoding. The authors provide several theoretical results on the capacity and convergence speed of new Hopfield model. Besides the use of this layer, the STANHOP architecture adopts several new solutions, such as the use of Plug-and-Play and Tune-and-Play memory plugin modules.\n\nThe experiments are solely focused on timeseries prediction tasks and compare several versions of the STANHOP architecture with several existing Transformers baselines.", "questions": "What are the advantages of using the alpha-entropy regularization instead of the Gini entropy used in the original sparse Hopfield network paper?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The introduction and analysis of alpha-entropy regularized Hopfield models and of the associated Transformer layers is interesting and potentially very useful.\n- The design of the proposed architecture has several interesting components.\n- The paper is well written.\n- The experiments compare the results with a large number of relevant baseline models.", "weaknesses": "The main issue with this work is that it tries to introduce too many innovations packed together in a single specialized architecture. This results in a paper that lacks a cohesive narrative, as it is unclear why these different novel parts should fit together. As a consequence, it is difficult for the reader to properly evaluate the merits of the different contributions. In my opinion, the main contribution is the introduction of the alpha-entropy regularized sparse Hopfield layers and their analysis. However, it is unclear to me why these layers should only be validated in multivariate timeseries prediction problems.  All in all, the specialized nature of the application does not match well with the general nature of the analysis in the first half of the paper. \n\nWhile the experimental analysis on timeseries data is rigorous, the results are rather disappointing since the main focus of the paper was to solve this specialized problem. The proposed architecture performs worse than at least one baseline model (DLinear) and in general it performs very similarly to the other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)