Predictive Attractor Models
Authors: Ramy Mounir
Sudeep Sarkar
Venue: NeurIPS 2024 poster (2024)
Keywords: Sequential Memory
Predictive Models
Fixed-point Attractors
Associative Memory Models
State Space Models
Biologically plausible
Hebbian Plasticity Rules
Local Computations
Hierarchical Temporal Memory
Continual Learning
Multiple Possibilities Generation
Noise Tolerance
Primary area: neuroscience_and_cognitive_science
TL;DR: A biologically plausible sequential memory model that (1) represents and generates multiple possibilities by learning fixed-point attractors, (2) learns continually while avoiding catastrophic forgetting, and (3) is robust to noise.

Abstract: Sequential memory, the ability to form and accurately recall a sequence of events or stimuli in the correct order, is a fundamental prerequisite for biological and artificial intelligence as it underpins numerous cognitive functions (e.g., language comprehension, planning, episodic memory formation, etc.) However, existing methods of sequential memory suffer from catastrophic forgetting, limited capacity, slow iterative learning procedures, low-order Markov memory, and, most importantly, the inability to represent and generate multiple valid future possibilities stemming from the same context. Inspired by biologically plausible neuroscience theories of cognition, we propose Predictive Attractor Models (PAM), a novel sequence memory architecture with desirable generative properties. PAM is a streaming model that learns a sequence in an online, continuous manner by observing each input only once. Additionally, we find that PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns, which prevents new memories from overwriting previously learned knowledge. PAM generates future predictions by sampling from a union set of predicted possibilities; this generative ability is realized through an attractor model trained alongside the predictor. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework. Other desirable traits (e.g., noise tolerance, CPU-based learning, capacity scaling) are discussed throughout the paper. Our findings suggest that PAM represents a significant step forward in the pursuit of biologically plausible and computationally efficient sequential memory models, with broad implications for cognitive science and artificial intelligence research. Illustration videos and code are available on our project page: https://ramymounir.com/publications/pam.

Predictive Attractor Models
Ramy Mounir Sudeep Sarkar
Department of Computer Science and Engineering, University of South Florida, Tampa {ramy, sarkar) @usf .dd
Abstract
Sequential memory, the ability to form and accurately recall sequence of events or stimuli in the correct order, a fundamental prerequisite for biological and artificial intelligence as it underpins numerous cognitive functions (e.g., language compre- hension, planning, episodic memory formation, etc.) However, existing methods of sequential memory suffer from catastrophic forgetting, limited capacity, slow iterative learning procedures, low-order Markov memory, and, most importantly, the inability to represent and generate multiple valid future possibilities stemming from the same context. Inspired by biologically plausible neuroscience theories of cognition, we propose Predictive Attractor Models ( (PAM). a novel sequence mem- ory architecture with desirable generative properties. PAM is streaming model that learns sequence in online, continuous manner by observing each input only once. Additionally, we find that PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns, which prevents new memories from overwriting previously learned knowledge. PAM generates future predictions by sampling from union set of predicted possibilities; this generative ability is realized through an attractor model trained alongside the predictor. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework. Other desirable traits (e.g.. noise tolerance, CPU-based learning, capacity scaling) are discussed throughout the paper. Our findings suggest that PA.M represents a significant step forward in the pursuit of biologically plausible and computationally efficient sequential memory models, with broad implications for cognitive science and artificial intel- ligence research. Illustration videos and code are available on our project page: https: / http:///mmmmmoiir.mmm com/ httppuulllicc publicatton catiiaattt/niiiiocoooooooo /pam
Introduction
Modeling the temporal associations between consecutive inputs in sequence (i.e., sequential memory) enables biological agents to perform various cognitive functions, such as episodic memory formation [53, 64, 15], complex action planning [22] and translating between languages [3]. For example, playing a musical instrument requires remembering the sequence of notes in a piece of music; similarly, playing game of chess requires simulating, planning, and executing sequence of moves in a specific order. While the ability to form such memories of static, unrelated events has been extensively studied [56, 55, 63, 69, 66), the ability of biologically-plausihe artificial networks to learn and recall lemporally-depende
patterns has not been sufficiently explored in literature [62]. The task of sequential memory is considered challenging for models operating under biological constraints (i.c., local synaptic computations) for many reasons, including catastrophic forgetting, ambiguous context representations, multiple future possibilities, etc.
In addition to the biological constraint, we impose the following set of desirable characteristics as learning constraints sequence memory models.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The learning of one sequence does not overwrite the previously learned sequences. This prop- erty is defined under the continual learning framework as avoiding catastrophic forgetting and evaluated with the Backward Transfer (BWT) metric [38].
The model should uniquely encode inputs based on their context in a sequence. Consider the sequence of letters "EVER"; the representation of "E" at position should be different from "E" at position 3, thus resulting in different predictions: "V" and "R". Moreover, the representation of "E" at position 3 in "EVER" should be different from "E" at position 3 in "CLEVER". Therefore, positional encoding is not valid solution.
When presented with multiple valid, future possibilities, the model should learn to represent each possibility separately yet stochastically sample a single valid possibility. Consider the two sequences "THAT" and "THEY"; after seeing "TH", the model should learn to generate either "A" or "E", but not an average [37] or a union of both [25].
The model should be capable of incrementally learning each transition without seeing the whole sequence or revisiting older sequence transitions that are previously learned. This property falls under online learning constraints, also called stream learning [45, 25J.
The learning algorithm should be tolerant and robust significant input noise. A model should continuously clean the noisy inputs using learned priors and beliefs, thus performing future predictions based the noise-free observations [25].
We propose Predictive Attractor Models (PAM), which consists of state prediction model and generative attractor model. The predictor in PAM is inspired by the Hierarchical Temporal Memory (HTM) [25] learning algorithm, where a group of neurons in the same cortical minicolumn share the same receptive feedforward connection from the sensory input on their proximal dendrites. The depolarization of the voltage of any neuron in a single minicolumn (i.e., on distal dendrites) primes this depolarized neuron to fire first while inhibiting all the other neurons in the same minicolumn from firing (i.e., competitive learning). The choice of which neurons fire within the same minicolumn is based on the previously active neurons and their trainable synaptic weights to the depolarized neurons, which gives rise to unique context representation for every input [20]. The sparsity of representations (discussed later in Section 3.2) allows for multiple possible predictions to be represented as a union of individual cell assemblies. The Attractor Model learns to disentangle possibilities by strengthening the synaptic weights between active neurons of input representations and inhibiting the other predicted possibilities from firing, effectively forming fixed point attractors during online learning. During recall, the model uses these learned conditional attractors to sample one of the valid predicted possibilities or uses the attractors as prior beliefs for removing noise from sensory observations.
PAM satisfies the above-listed constraints for sequential memory model, whereas the current ofte-off-theart models fail in all or many of the constraints, as shown in the experiments. Our contributions can be summarized as follows: (1) Present the novel PAM learning algorithm that can explicitly represent context in memory without backpropagation. avoid catastrophic forgetting, and perform stochastic generation of multiple future possibilities. (2) Perform extensive evaluation of PAM on multiple tasks (e.g., sequence capacity, sequence generation, catastrophic forgetting, noise robustness, etc.) and different data types (e.g., protein sequences, text, vision). (3) Formulate PAM and its learning rules State Space Model grounded in variational inference and the Free Energy Principle [17].
2 Background and Related Works
2.1 Predictive Coding
Predictive coding proposes a framework for the hierarchical processing of information. It was initially formulated as time series compression algorithm to create more efficient coding system [16, 46]. A few decades later, PC was used to model visual processing in the Retina [59, 27] as an inference model. the seminal work of Rao and Ballard [52], PC was reformulated as a general computational model of the cortex. The main intuition is that the brain continuously predicts all perceptual inputs, resulting in a quantity of prediction error, which can be minimized by adjusting its neural activities and synaptic strengths. In-depth variational free energy derivation is provided in Appendix A.

---

PC defines two subgroups of neurons: value 2 and error. Each neuron contains a value node sending its prediction to the lower level & = fi+1 (2(+1) through learnable function f. and error node propagating its computed error to the higher level. The total prediction error is computed as == - L0]. which is minimized by first running the network value nodes to equilibrium through optimizing the value nodes {=)) At equilibrium, the value nodes are fixed, and inferential optimization is performed by optimizing the functions {fi]/= Both optimizations aim to minimize the same prediction enror over all neurons. This propagation of error to equilibrium is shown to be equivalent to backpropagation but using only local computations [66]. The PC formulation has shown success in training on static and i.i.d data [66, 69, 57, 23]. More recently [62], Temporal Predictive Coding (IPC) has also shown some success in sequential memory tasks by modifying error formulation to account for a one-step synaptic delay through interneurons, thus modeling temporal associations between sequence inputs. In the experiments, we compare our model to tPC and its 2-layer variant [62]. A few PC-based models [47, 48] have shown promise for class-incremental continual learning tasks, other PC-inspired models, such as [39, 23, 24], have diverged from the biologically plausible constraints by training through backpropagation through time.
2.2 Fixed-Point Attractor Dynamics
Attractor dynamics refer to mathematical models that describe the behavior of dynamical systems. In our review, we focus on fixed point attractors, specifically Hopfield Networks [26] (HN), which an instance of associative memory models [32, 29, 30]. Consider, an ordered sequence of T + consecutive patterns
[ where D; {-1, 13.N. We refer to the Universal Hopfield Networks (UHN) framework [41] to describe all variants of HN architecture using similarity (sim) function and a separation (sep) function, as shown in equation 1. This family of models can be viewed as memory recall function, where a query (i.e., noisy or incomplete pattern) is compared to the existing patterns to compute similarity scores using the "sim" function. These scores are then used to weight the projection patterns after applying "sep" function to increase the separation between similarity scores. The classical HN uses symmetric weights to store the patterns; therefore, it cannot be used to model temporal associations in sequence. The asymmetrie Hopfield Network (AHN) [58] uses asymmetric weights to recall the next pattern in the sequence for given query &.
x++1 sep (sim(x,, {)) sep (sim(x,, E))
Asymmetric Weights Symmetric Weights
{ = P
sep
(sim(A)
(1)
Projection
Separation
Similarity
When dot product "sim" function and identity "sep" function used, get the classical HN [26] and AHN [58]. A few variants have been proposed to increase the capacity of the model. Recently. [12] has extended AHN by using a polynomial (with degree d) or a softmax function (with temperature 9) as the "sep" function. HN can also be applied to continuous dense patterns [34, 51, 12]. and extended to sparse modern Hopfield models [28, 67].
2.3 Predictive Learning
Predictive learning takes a more general form of minimizing the prediction error between two views of the same input to improve representations. Many backpropagation-base approaches to predictive learning have been proposed: most recently, JEPA [37] and its variants [8, 10, 70), learn useful dense representations from images and videos using the predictive objective. Other models, such as [11, 21, 13] to list a few, use a similar methodology of predicting distorted versions of the same input to learn good feature representations. been used to segment videos into events temporally [2, 42] and spatially [44, 43, 1]. More recently, STREAMER [45] used predictive learning approach achieve hierarchical segmentation and representation learning from streaming egocentric videos, where a high prediction error is used as an event boundary. While these biologically implausible approaches show impressive results on their respective tasks, they still suffer from deep learning known limitations, such as catastrophic forgetting and the inability to generate multiple possibilities in regression-based predictive tasks. Hierarchical Temporal Memory (HTM) [25] is predictive approach that is heavily inspired by neurobiology. HTM relies on lateral inhibition between neurons of the same minicolumn and sparsity of input representations (i.c.. SDR) to learn temporal context and associations using only local Hebbian rules.
inliiction-based approaches have also
3

---

HTM can be applied to online tasks, such as anomaly detection [6] and object recognition [36], but it currently incapable of generating future predictions auto-regressive prediction tasks.
Predictive Attractor Models
3.1 State Space Model (SSM) formulation
p(2/2,,1)
Midiille Frosch
Single Prosebillt Posterer
Boog
Posterior
Grber
Red
Update Frior
p(
g
g
G Bingle Possicilly
Altroctor
N
Figure I: State Space Model. (Left): Dynamical system represented by first-order Markov chain of latent states z with transition function f and an emission function g which projects to the observation states æ. (Right): Gaussian form assumptions for the prior 2 and posterior latent states, and the Mixture of Gaussian model representing the conditional probability multiple possibilities p(æ: 2x)
PAM can be represented as a dynamical system with its structure depicted by a Bayesian probabilistic graphical [35, 31] model, more specifically, State Space Model, where we can perform Bayesian inference on the latent variables and derive learning rules using Variational Inference. Formally, define state space model tuple X2 f.g), where Z is the latent state space, X' is the observation space, and f and g are the transition and emission functions respectively (similar to HMM [50]). We consider Gaussian form with white Gaussian noise covariance Σ, for the latent states. However, we assume a latent state z can generate multiple valid possibilities. Therefore, we model the conditional probability p( az, [z,) each mode is considered a possibility or a fixed-point attractor in an associative memory model. The GMM has components with means (z,,,, covariances E, and component weights w. The SSM dynamics can be formally represented with the following equations:
Multivariate Gaussian Mixture Model (GMM), where
2, 2,12,-1 N(f(2111), E.), and Z./Z, Σ We ((ga(z))) E.)
(2)
where E Z and I, E X. From the Bayesian inference viewpoint, we are interested in the posterior p(2, (2,, 2:-1). Since the functions f and y are non-linear, the computation of this posterior in- tractable (unlike an LG-SSM, such as Kalman Filter [65]). Therefore, we utilize variational inference to approximate the posterior by assuming the surrogate posterior q(2) has a Gaussian form, and mini- mize the Variational Free Energy (VFE) [18]. The minimization of VFE (in equation 3) minimizes the KL-divergence between the approximate posterior ((x) ) and the true posterior p(ze 2;-1). Derivation of Variational Free Energy is provided in appendix A.l
g(zz)
T
Eg log p(a,, 2e zt-
Σ E
Ey
log (pg(p(ze 2 x1-1)
log(p( (xx 1x))
Variational Free Energy
Latent State Error
Observation Eiror
where F. = Ez,,)((.)) and Hy is the Entropy of the approximate posterior (=)). The assumption of Gaussian forms for the latent and observable states can further simplify the negative likeliieldoodd

---

terms (i.e., Latent State Accuracy and Observation Accuracy) to prediction errors. This learning objective encourages the approximate posterior ((2) assign a high probability to states that explain the observations well and follow the latent dynamics of the system. We minimize the prediction errors (i.e., learn the transition and emission functions) through Hebbian rules as shown in equations? and 8.
Theorem Assume the likelihood p(æ (2)) in eqn 3 represents multiple possibilities using a Gaus- sian Mixture Model (GMM) conditioned on the latent state 2,, shown in eqn 2. The maximization of such log-likelihood function (i.e., B. Dh log logp(ar (2x (e)) w.r. query observation state æ is equivalent to the Hopfield recall function (i.e., eqn 1) with the means of the GMM representing the attractors of a Hopfield model. Formally, the weighted average of the GMM means (i.e., (pe]e=)) with the weights being a similarity measure, maximizes the rngliikelhood of x under the mixture model.
We Na Me Dx) Σe'
x
He projection
(4)
We NZ:; pey En)
E
cimilority score
Proof:
See derivation 3 in appendix A.2 for the full proof.
3.2 Preliminaries and Notations
Sparse Distributed Representation (SDR) Inspired by the sparse coding principles observed in the brain, SDRs encode information using small set of active neurons in high dimensional binary representation. We adopt SDRs as a more biologically plausible cell assembly representation [30]. SDRs have desirable characteristics, such low chance of false positives and collisions between multiple SDRs and high representational capacity [5] (More on SDRs Appendix G). An SDR is parameterized by the total number of neurons N and the number of active neurons W. The ratio = W/N denotes the SDR sparsity. A 1-dimensional SDR æ can be indexed as x² E {0, 1}. whereas 2-dimensional SDR
be indexed as 2 E {0, 1}. To identiff the active neurons, define the function 7 : {0, 1 + NW to represent the indices of the active neurons in an SDR æ as ((x) (i = 1).
Context Orthogonal Dimension We transform the high-order Markov dependencies between observation states into a first-order Markov chain of latent states by storing context information in those latent states. The latent states SDRs, {0, 1 N,XNN, are represented with two orthogonal dimensions, where content information about the input is stored in one dimension with size No. while context related information is stored in an orthogonal dimension with size No- Therefore, the projection of the latent state z on the first dimension (i.e., + 2) removes all context information from the state. In contrast, adding context information dimensionality of the state (i.e., 1 x) such that context can be encoded without affecting its content. the context dimension through lateral inhibition, effectively minimizing the collisions between contexts of multiple SDRs. We define projection operator 1: (0, N,,, + (0. 13 N.. Additionally, we define a projection operator f: {0, 13 + {0, 10 N. for I-dimensional SDRs (i.c., x) as shown in equation 5. A detailed list of notations is provided in the supplementary material (Table 1).
an observation state æ expands the
Competitive learning is enforced
if s.l. 2t =1, (1 (1x)²= {
if of 1, otherwise,
(1 x))
(5)
otherwise,
3.3 Sequence Learning
Given sequence of T + SDR patterns [x] where x; {0, 17Ne, the sequence can be learned by modeling the context- dependt- transitions between consecutive inputs within the sequence. We
define learnable weight parameters for transition and emission functions, A E D.N.NNxNN.. B R.N.xNN respectively. A single latent state transition is defined as 2, 8(A A 2t-1) 5(ar). where 5 is threshold function transforming the logits a, to the predicted SDR state 2, The full sequence learning algorithm is provided in algorithm I.
5

---

Sequence Learning. Given a sequence of T+1 patterns, this algorithm learns the Transition and Emission synaptic weights (A and B). Fixed start context m(ag) initialized for all learned sequences.
Algorithm 1
Algorithm Sequence Generation. Given a noisy sequence (i.e., online), or the first input in sequence (i.e., offline). The model uses the learned functions A and B to generate the full sequence. ~ denotes sampled from (eqn 9).
procedure TRAIN(#))
1: procedure GENERATE(Z2 or
2: Zo = (1 at)) C m((s)
b Eqn. 6
2: Za = (1 2t) m((a))
Eqn. 6
3: for 1 = to I + do
3: for t = 1 to I + do
4: a: A.21-1
4: a. A-21-1
21 (f (:) ( (m(a.)
2t 8(a:)
A AA
b Update Eqn. 7
æt
online
7: & =
~ (1) offline
Equ. 9
8
B B + +BB
Update B via Egn. 8
9:
end for
for iters
D Attractors iterations
10: end procedure
8:
ã = 6(B T1) (4
9:
end for
10:
(1 #
end for
12: end procedure
Context Encoding through Competitive Learning Every observation 2; contains only content information about the input; embed the observation with context by expanding the state with an orthogonal dimension (i.e., 1 xy), which activates all neurons the minicolumns at the indices I (xx)). Then, for each active minicolumn, the neuron in predictive state (i.e., higher than the prediction threshold) fires and inhibits all the other neurons in the same minicolumn from firing (i.e., lateral inhibition), shown in Equation 6. If none more than one of the neurons are in predictive state, random Gaussian noise (e) acts as a tiebreaker to select a context neuron. We do not allow multiple neurons to fire in the same minicolumn, which different from HTM [25], where multiple cells can fire in any minicolumn (e.g., bursting).
{ if a max((5(a))) ) 2, otherwise.
m(a:)
xx.) mm(a))
(6)
State Transition Learning The transition between latent states learned through local computa- tions with Hebbian-basee rules. We modify the synaptic weights A to model the transition between pre-synaptic neurons 2;-1 and post-synaptic neurons 2,. Only the synapses with active pre-synaptic neurons are modified. The weights operate on context-dependant latent states (i.e., Σ₁-1 - zz); thus, the learning of one transition does not overwrite previously learned transition of different contexts, regardless of the input contents (i.e., 11-1)) We use the learning constant coefficients n and 7 to independently control learning and forgetting behavior, shown in equation 7. A lower 7/ encourages learning multiple possibilities by slowing down the forgetting behavior.
AA
21-1 21
"AA 2t-1 (1))
2.)
A
à A
Contrastive Attractors Formation The attractors are formed in an online manner by contrasting the input observation to the predicted union set of possibilities + 2,. The goal is to learn excitatory synapses between active neurons of H,, and bidirectional inhibitory synapses between æ, and the union set of predicted possibilities excluding the æ, possibility (i.e., (1 2c) \ xx)), shown in equation
AB
E. Et T + Mn
[x. (() 2,) xz 5 ((1 2,) xx) x
(8)
A Biuseas
Bassaas
3.4 Sequence Generation
After learning one or multiple sequences using algorithm 1, we use algorithm to generate sequences. First, define two generative tasks: online and offline. In online sequence generation, noisy

---

Generation
Online Generation
4
DOG0
000
00000
DOC
O- COO 2
0900
Union of Possibilies
O 0O0
Markov Blanket
Afrozter Besite
Adrrocto Pheerw
Nokey Choerradion B
Figure 2: Sequence Generation. (Left): Offline generation by sampling single possibility (i.e., attractor point) from a union of predicted possibilities. (Right): Online generation by removing noise from an observation using the prior beliefs about the observed state. Markov Blanket separates the agent's latent variables from the world's observable states.
version of the sequence is provided as input, and the model is expected to generate the original learned sequence. offline sequence generation, the model is only provided with the first input, and it is expected to generate the entire sequence auto-regressively. For cases with equally valid future predictions (e.g., "a" and "c" after "TH" in "THAT" and "THEY"), the model expected to stochastically generate either one of the possibilities (i.e., "THAT" or "THEY"), The online generation task more challenging extension of the online recall task [62], where the noise-free inputs are provided, and the model only makes a 1-step prediction into the future. During offline sequence generation, the model randomly samples from union set of predictions 4 single SDR with W active neurons (equation 9) to initialize the iterative attractor procedure. π denotes random permutation function. This random permutation function allows the model randomly generate different sequence with every generation.
x = {
i E (T(7() 3)) otherwise
(9)
4 Experiments
4.1 Evaluation and Metrics
Metrics To evaluate the similarity of two SDRs, we use the Jaccard Index (i.e., IoU)!, which focuses on the active bits in sparse binary representations. Since the sparsity the representations can change across experiments and methods, we normalize the IoU by the expected IoU (Derived Theorem in Appendix A.3) of two random SDRs at their specified sparsities. The normalized IoU is computed as [oU-B]loU] [[E[[o We use the Backward Transfer [38] metric in evaluating catastrophic
forgetting. Mean Squared Error (MSE) is used to compare images for vision datasets.
Datasets We perform evaluations synthetic and real datasets. The synthetic datasets allow to manually control variables (c.g., sequence length, correlation, noise, input size) to better understand the models' behavior across various settings. Additionally, we evaluate real datasets of various types (e.g., protein sequences, text, vision) to benchmark PAM's performance relative to other models more challenging and real sequences. For all vision experiments, learn a mapping between images and SDRs (Details on the autoencoder are provided in Appendix C). We all experiments for 10 different seeds and report the mean and standard deviation in all the figures. More experimental details and results are provided in Appendices and E.
SDR autoencoder to
In the context of SDRs. IoU refers to the number of active bits in the intersection SDR divided by the number of active bits in the union SDR
7

---

AND MiD1A
Setiiee Wnd
Sequence Copacity rpat
B
Sequence Copac ation
Vi. Nebi
Langth
1.0
221
6d
Normateed
6-4
TOWo 1S
0.2
0.0
-02
35"1
Cametic on
9. active biss changed
Sequercee Length
F
Offine
Online
1VM 15
2181 841
Figure 3: Quantitative results on (A-B) Offline sequence capacity, (C) Noise robustness, and (D) Time of sequence learning and recall. Qualitative results on highly correlated CIFAR sequence in (E) offline and (F) online settings. The mean and standard deviation of 10 trials are reported for all plots.
4.2 Results
We align our evaluation tasks with the desired characteristics of biologically plausible sequence memory model, as listed in the introduction. We show that PAM outperforms current predictive coding and associative memory SoTA approaches on all tasks. Most importantly, PAM capable of long context encoding, multiple possibilities generation, and learning continually and efficiently while avoiding catastrophic forgetting. These tasks pose numerous significant challenges to other methods.
Offline Sequence Capacity We evaluate the models' capacity to learn long sequences by varying the input size No. model parameters (e.g., N2), and sequence correlation. The correlation increased by reducing the number of unique patterns (i.e., vocab) used to create sequence of length T. T
Correlation is computed as 1.0 vocab In Figure: A, we vary the input size N₂ and ablate the
models find the maximum sequence length to be encoded and retrieved, in an offline manner, with Normalized loU higher than 0.9. We set the number of active bits W to be 5 unless otherwise specified. Results show that Hopfield Networks (HN) fail to learn with sparse representations; therefore, W of 0.5N2, only for HN and normalize the loU metric accordingly. PAM's capacity significantly increases with context neurons N₁. expected. HN's capacity also increases with the polynomial degree d of its separation function; however, as shown in Figure 3 B, the capacity sharply drops as correlation increases. PAM retains its capacity with increasing correlation, reflecting its ability to encode context in long sequences (i.e.. high-order Markov memory). This context encoding property also demonstrated in the qualitative CIFAR [33] results in Figure's 3 E and F, where short sequence of images with high correlation is used. The model must uniquely encode the context to correctly predict at every step in the sequence. While PAM correctly predicts the full context, single-layer tPC learns to indefinitely alternate between the patterns, while two-layered tPC attempts to average its predictions. AHN shows similar low performance and failure mode as in [62].
Catastrophic Forgetting To assess the model's performance in a continual learning setup, we sequentially train each model on multiple sequences and compute the Backward Transfer (BWT) [38]
8

---

-ee PAN 1
FAH
PAY A
PAR PA
PAM
MC
AHR ONL Wew Sic
JMN de2, WwO.NN
fockword Wranfferis, Comeiition
Backward Torcler VS Protch Soquenoes
Wi generctee worls Mumber worts
Nutseer Reca 128e wonds Deneradaes
Thenter (ter
04
5.
Comeodinn
Number of Priten Sequences
Numbero WAVES
Number Oceergeee
Mamorins
Mamorisa
4
R
48
6
PC1dt
80
PCL
0
0
Figure 4: Qualitative results on (A) synthetic and (B) protein sequences backward transfer, and (C-D) multiple possibilities generation on text datasets. Qualitative results (E) noise robustness on CLEVRER sequence, and (F) catastrophic forgetting on Moving MNIST dataset. a highlights the first frame with significant error. The mean and standard deviation of 10 trials reported for all plots.
metric by reporting the average normalized IoU on previously learned sequences after learning a new one. In Figure 4 A, we report BWT for 50 synthetically generated sequences with varying correlations. AHN can avoid catastrophic forgetting when the patterns are not correlated, whereas tPC fails to retain previous knowledge regardless of the correlation value. PAM, with high enough context Ni. does not overwrite or forget previously learned sequences after learning new ones but performs poorly when more challenging sequences from Protein [7], which contains highly correlated (> 0.9) and long sequences (details in appendix). The results in Figure 4 B show similar trend with PAM requiring more context neurons N₂ to handle the more challenging data. Qualitative results on moving -MNIST [60] in Figure F further demonstrate the catastrophic forgetting challenge where the learning of the second sequence overwrites the learned sequence. PAM successfully retrieves the previously learned sequence while the other models fail.
N& 1, expected. We repeat the experiment
Multiple Possibilities Generation In addition accurately encoding input contexts, PAM is designed to represent multiple valid possibilities and sample a single possibility. We perform evaluation dataset of four-letter English words (details in appendix). which includes many possible future completions (e.g., "th[is]", "th[at]", "th[em"", etc.) We train PAM on the list of letters sequentially (i.e., one word at a time); the other methods are trained in simpler batched setup as in [62] because they suffer from catastrophic forgetting. This puts PAM at a disadvantage, but as shown in Figure C, PAM still significantly outperforms the other methods in accurately generating valid words (high average IoU) in an offline manner. Both tPC and AHN fail to generate meaningful words when trained on sequences with multiple future possibilities. Figure D further demonstrates the stochastic generative property of PAM. We show PAM's ability to recall more of the dataset as it repeats the generation process, whereas PC and AHN fail in the dataset recall task.
Online Noise Robustness The online generation ability of PAM shown in Figure 2 allows the model use the learned attractors clean the noisy observations before using them as inputs the predictor. This step allows the model to use its prior belief about future observations to modify the noisy inputs. In Figure C, we evaluate the models' performances by changing a percentage of the active bits during online generation. PAM is able to completely replace the noisy input with its prior belief it does not exist the predicted set of possibilities ž. In contrast, the other methods use the noisy inputs, thus hindering their performances. We provide qualitative results on CLEVRER [68]

---

in Figure 4 E; PAM retrieves the original sequence despite getting noisy inputs (40% noise), and outperforms the other models. Interestingly. tPC performs reasonably well on this task despite the added noise.
Efficiency We report, in Figure D. the time each model requires to learn and recall sequence. For this study, we use input size Nr = 100 and vary the sequence length. PAM operates entirely on CPU. The results show that a single-layer tPC model requires more time than all PAM variants (N₂ < 24). Additionally, two-layered tPC requires two to three orders of magnitude more time than PAM or single-layered tPC, significantly limiting scalability and practicality when applied to real data with long sequences.
5 Limitations
PAM requires that its representations be sparse and binary (i.e., SDRs) in order to represent multiple possibilities union of SDRs with minimal overlap. Therefore, PAM cannot be directly applied to images in the input space like Dense Hopfield Networks. We argue that the neocortex encodes sensory input into SDRs for processing and does not operate directly the input in its dense representation. Methods that operate directly on dense representations (c.g.. images) are arguably less biologically plausible as the neocortex uses sparse binary representations (i.e., cell assemblies with small fraction of firing neurons) to store and manipulate information. This paper focuses on learning multiple sequences without catastrophic forgetting and stochastically generating multiple possibilities efficiently, and we assume the sequence is provided in SDR format. Additionally, methods that operate on the input directly face challenges when the input is naturally sparse (see Figure 3 A). Therefore, it is useful to encode the input into representation with fixed sparsity before applying sequential memory learning algorithms. In future work, we plan to investigate how encode high dimensional complex inputs (e.g., images) in compositional part-whole structure of SDRs where we can apply PAM at different levels of abstraction.
Conclusion
We proposed PAM,, a biologically plausible generative model inspired by neuroscience findings and theories of cognition. We demonstrate that PAM is capable of encoding unique contexts with tremendous scalable capacity that is not affected by sequence correlation noise. PAM is a generative model; it can represent multiple possibilities as a union of SDRs (a property of sparsity) and sample single possibilities, thus predicting multiple steps in the future despite multiple possible continuations. We also show that PAM does not suffer from catastrophic forgetting as it learns multiple sequences. PAM is trained using local computations through Hebbian rules and runs efficiently CPUs. Future directions include hicrarchical sensory processing and higher-order sparse predictive models.
10

---

Acknowledgements
This research was supported by the US National Science Foundation Grants CNS 1513126 and IIS 1956050.
References
Sathyanarayanan Aakur and Sudeep Sarkar. Action localization through continual predictive learning. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings. Part XIV 16, pages 300-317. Springer, 2020.
[2] Sathyanarayanan N Aakur and Sudeep Sarkar, A perceptual prediction framework for self supervised event segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1197-1206, 2019.
[3] Valentin Afraimovich, Xue Gong, and Mikhail Rabinovich. Sequential memory: Binding dynamics. Chaos: An Interdisciplinary Journal of Nonlinear Science, 25(10), 20155
[4] Subutai Ahmad and Jeff Hawkins. Properties of sparse distributed representations and their application hierarchical temporal memory. arXiv preprint (rcXvv15033.04550 2015.
[5] Subutai Ahmad and Jeff Hawkins. How do neurons operate on sparse distributed representations? mathematical theory of sparsity, neurons and active dendrites. arXiv preprint arXiv:1601.0072 10, 2016.
[6] Subutai Ahmad, Alexander Lavin, Scott Purdy, and Zuha Agha. Unsupervised real-time anomaly detection for streaming data. Newrocomputing, 262: 143444,,455 2017.
[7] Mohammed AlQuraishi. Proteinnet: a standardized data set for machine learning of protein structure. BMC bioinformatics, 20:1-10, 2019.
[8] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture, Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 15619-15629, 2023.
[9] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv 244.00400, 2024.
[10] Adrien Bardes, Jean Ponee, and Yann LeCun. Mc-jepa: joint-embedding predictive architecture for self-supervised learning of motion and content features, arXiv preprint vrXiv:2307.1226% 2023.
[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IFFF/CVF international conference on computer vision, pages 950-9660, 2021.
[12] Hamza Chaudhry, Jacob Zavatone-Vent Veth, Dmitry Krotov, and Cengiz Pehlevan. Long sequence hopfield memory. Advances in Neural Information Processing Systems, 36, 2024.
[13] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17750-1558.. 2021.
[14] Nikolaos Chrysanthidis, Florian Fiebig, and Anders Lansner. Introducing double bouquet cells into modular cortical associative memory model. Journal of computational neuroscience, 47(2):223-236 230. 2019.
[15] Martin Conway, Episodic memories. Neumpsychologia, (7(11))2305-2313 2009.
[16] Peter Elias. Predictive coding-i. IRE transactions on information theory, 1(1):16-24, 1955.
[17] Karl Friston. The free-energy principle: unified brain theory? Nature reviews neuroscience, 11 138, 2010.
[18] Karl Friston, Jérémie Mattout, Nelson TrujilloBarrreto John Ashburner, and Will Penny. Variational free energy and the laplace approximation. Neuroimage,
234, 2007.
[19] Karl J Friston, N Trujillo-Barreto, and Jean Daunizeau, Dem: variational treatment of dynamic systems. Neuroimage, 41(3):849 885, 2008.

---

[20] Dileep George, Rajeev V Rikhye, Nishad Gothoskar, J Swaroop Guntupalli, Antoine Dedieu, and Miguel MiizaroGGredille Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps. Nature communications, 12(1):2332, 2021.
[21]
Saan-Battien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond. Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances neural information processing systems, 00:21271-21280 2020.
[22] Patrick Haggard. Planning of action sequences. Acta Psychologica, 99(2):001-215, 1998.
[23] Kuan Han, Haiguang Wen, Yizhen Zhang, Di Fu, Eugenio Culurciello, and Zhongming Liu. Deep predictive coding network with local recurrent processing for object recognition. Advances neural information processing systems, 31, 2018.
[24] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation leamingg by dense predictive coding. In Proceedings of the IEEE/CVE International Conference on Computer Vision Workshops, pages 0-0, 2019.
[25] Jeff Hawkins Subutai Ahmad. Why neurons have thousands of synapses, theory of sequence memory in neocortex. Frontiers in neural circuits, 10: 174222, 2016.
[26] John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554 255822555-,,, 1982.
[27] Toshihiko Hosoya, Stephen A Baccus, and Markus Meister. Dynamic predictive coding by the retina. Nature, 436(7047):71-77, 2005.
[28] Jerry Yao-Chich Hu, Donglin Yang, Dennis Wu, Chenwei Xu, Bo-Yu Chen, and Han Liu. On sparse modern hopfield model. Advances in Neural Information Processing Systems, 36, 2024.
[29] Hiroshi Kage. Implementing associative memories by echo state network for the applications of natural language processing. Machine Learning with Applications, 11:100449, 2023.
[30] Pentti Kanerva. Sparse distributed memory. MIT press, 1988.
[31] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009)
[32] Bart Kosko, Bidirectional associative memories. IEEE Transactions on Systems, man, and Cybernetics, 18(1):49-60, 1988.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] Dmitry Krotov John Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
[35] Steffen L Lauritzen. Graphical models, volume 17, Clarendon Press, 1996.
[36] Niels Leadholm, Marcus Lewis, and Subutai Ahmad, Grid cell path integration for movement-based visual object recognition. arXiv preprint arXiv: 1rXiv:2102.090 100.09076 2021.
[37] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. 2022.
[38] David Lopez-Paz and Marc' Aurelio Ranzato, Gradient episodic memory for continual learning. Advances in neurul information processing systems, 30, 2017.
[39] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08100 2016.
[40] Henry Markram, Maria Toledo ModdooRodriguez, Yun Wang, Aninudh Gupta, Gilad Silberberg, and Caizhi Wu. Interneurons of the ncocortical inhibitory system. Nature reviews neuroscience, 5(10):793-80) 807, 2004.
[41] Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewiez, Rafal Bogacz, Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning, pages 15561-15583, PMLR, 2022.
[42] Ramy Mounir, Sathyanarayanan Aakur, and Sudeep Sarkar. self-super temporal event segmentation inspired by cognitive theories. In Advanced Methods and Deep Learning in Computer Vision, pages 405 448. Elsevier, 2022.
12

---

[43] Ramy Mounir, Roman Gula, Jöm Theuerkauf, and Sudeep Sarkar. Spatio-temporal event segmentation for wildlife extended videos. In International Conference on Computer Vision and Image Processing, pages 48-59. Springer. 2021.
[44] Ramy Mounir, Ahmed Shahabaz, Roman Gula, Jörn Theuerkauf, and Sudeep Sarkar. Towards auto- mated ethogramming: Cognitively-inspired event segmentation for streaming wildlife video monitoring. International journal of computer vision, 13 131(9):2267 2297, 2023.
[45] Ramy Mounir, Sujal Vijayaraghavan and Sudeep Sarkar. Streamer: Streaming representation learning and event segmentation a hierarchical manner. Advances in Neural Information Processing Systems, 36, 2024.
[46] O'Neal. Entropy coding in speech and television differential pem systems (corresp.). IEEE Transactions on Information Theory, 17(6):55--76,, 1971.
[47] Alex Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Lifelong neural predictive coding: Learning cumulatively online without forgetting. Advances in Neural Information Processing Systems, 35:5867- 5881, 2022.
[48] Alexander Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Continual learning of recurrent neural networks by locally aligning distributed representations. IEEE transactions on neural networks and learning systems, 31100)4467-44278, 2020.
[49] A Peters and C Sethares. The organization of double bouquet cells in monkey striate cortex. Journal of neurocytology, 26(12):779-777, 1997.
[50] Lawrence Rabiner and Biinghwang Juang. An introduction hidden markov models. ieee assp magazine, 3(1):411 16, 1986.
[51] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, et al. Hopfield networks is all you need. In International Conference on Learning Representations, 2020.
[52] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1))79-87, 1999.
[53] Edmund T Rolls. computational theory of episadic memory formation the hippocampus. Behavioural brain research, 215(2):180-19)) 2010.
[54] Tommaso Salvatori, Ankur Mali, Christopher L Buckley, Thomas Lukasicwicz, Rajesh PN Rao, Karl Friston, and Alexander Ororbia. Brain-inspires computational intelligence via predictive coding. arXiv preprint 23Xiv:2308.07870 2023.
[55] Tommaso Salvatori, Luca Pinchetti, Beren Millidge. Yuhang Song. Tianyi Bao, Rafal Bogacz, and Thomas Lukasiewicz. Learning on arbitrary graph topologies via predictive coding. Advances in neural information processing systems, 0233222238244- 2022.
[56] Tommaso Salvatori, Yuhang Song. Yujian Hong, Lei Sha, Simon Frieder, Zhenghua Xu, Rafal Bogacz, and Thomas Lukasiewicz. Associative memories via predictive coding. Advances in Neural Information 2021.
Processing Systems,
[57] Tommaso Salvatori, Yuhang Song. Beren Millidge, Zhenghua Xu, Lei Sha, Cornelius Emde, Rafal Bogacz, and Thomas Lukasiewicz. Incremental predictive coding: A parallel and fully automatic learning algorithm. arXiv preprint arXiv:2212.00722 2022.
[58] Haim Sompolinsky and Ido Kanter. Temporal association in asymmetric neural networks. Physical review lettery, (2(2))28861, 1986.
[59] Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: fresh view of inhihition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, 216(1205):427-55) 1982.
[60] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised lcarning of vidco representa- tions using Istms. In International conference on muchine learning, pages 843-852. PMLR, 2015.
[61] Dan D Stettler, Aniruddha Das. Jean Bennett, and Charles D Gilbert. Lateral connectivity and contextual interactions in macaque primary visual cortex. Neuron, 36(4):739- 750, 2002.
13

---

[62] Mufeng Tang, Helen Barron, and Rafal Bogacz. Sequential memory with temporal predictive coding. Advances in Neural Information Processing Systems, 36, 2024.
[63] Mufeng Tang, Tommaso Salvatori, Beren Millidge, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Recurrent predictive coding models for associative memory employing covariance learning. PLoS computational biology. 1914)1 2023.
[64] Endel Tulving al. Episodic and semantic memory. Organization of memory, 1(381-403):1, 1972.
[65] Greg Welch, Gary Bishop, et al. An introduction to the kalman filter. 1995.
[66] James CR Whittington and Rafal Bogacz. An approximation the error backpropagation algorithm in predictive coding network with local hebbian synaptic plasticity. Neural computation, 122-11222-, 2017.
[67] Dennis Wu, Jerry Yao-Chich Hu, Weijian Li, Bo- Yu Chen, and Han Liu. Stanhop: Sparse tandem hopficld model for memory-enhanced time series prediction. arXiv preprint arX1v:2312.17346, 2023.
[68] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference Learning Representations, 2019.
[69] Jinsoo Yoo and Frank Wood. Bayespen: A continually learnable predictive coding associative memory. Advances Neural Information Processing Systems, 35:29903-22911 14, 2022.
[70] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Sclf-supervised learning via redundancy reduction. In International conference on machine learning, pages 12310-12320. PMLR, 2021.
14

---

Supplementary Material
A Theorems and Derivations
A.1 Variational Free Energy
Predictive Coding Consider hierarchical generative model with hidden states {z) where 1 & I. denotes the level in the hierarchy. The conditional probability p(z) is assumed to be a multivariate Gaussian distribution with its mean calculated as function fi+1 of the higher-level hidden representation #141 and covariance Σ, as shown in equation 10.
p(2)|2141) (N ((44))) Ei)
(10)
The goal is to calculate the posterior of hidden states given observation (i.c., ((( Since the prediction function f contains a non-linear activation, we cannot analytically compute the posterior and we have approximate it with surrogate posterior (i.e., {({))/n) by maximizing the Evidence Lower Bound (ELBO). We apply the mean field approximation to factorize this joint posterior probability into conditionally independent posteriors {q(21))<u) and apply the Laplace approximation to use Gaussian forms for the approximate distribution [19, 18, 54]. Through these approximations, we can maximize the ELBO, or equivalently minimize the Variational Free Energy, in equation 11.
Eg Eg~ [log( ((2)
E Ezng(z) [log( ((2) p(z) )] E Ex logg( p(z|z)
(11)
p(x, 2)
Variational Free Energy
Dxc(a (p)
Accuracy
The variational free energy can be reduced to minimizing the negative log likelihood (Accuracy term), which is simply the prediction eπrоr when the likelihood is assumed to take a Gaussian Form. Therefore, minimizing the prediction error reduces the sum the squared prediction error of every neuron.
Derivation Variational Free Energy derivation for the predictive coding objective function in equation 11. We approximate the true posterior p(z(z) with surrogate posterior q(z). The objective is to minimize the reverse KL divergence Dx Dkilg(>))lp(z(z))
g(2)
DkI(g())[p(</)z) = Exng(z)
(KL Divergence definition)
p(21
4( (z())())
Reng(t)) flo
(Bayes Theorem)
p(x }p(z)
q(2)
E E2ng(z) flog
+ E Ezng(z) (lggp(())
(Linearity of expectations)
p( [)(])
q(z)
(x=((())|p))e))) Exng(z)log log p( [00]z)10 ()))=)
log)(x)
(Evidence does not depend on q(x))
Variational Free Energy
To minimize the KL divergence, can minimize the Variational Free energy instead because the
Evidence term (()) is a constant negative term. The Variational Free Energy can be further simplified as follows:
15

---

g(z)
E x~~g(2) (()) (log {(2)
B [log
E (z) log
(Linearity of Expectations)
p(æ) z [)())))
p( zz)
p(z)
g(2)
[x)a(()) Dog
E (x) log
DDz(((<))lpl)
(KI. Divergence definition)
p z z)p(z)
p 2)
Variational Free Energy
We arrive at equation 11. Minimizing the enror term (i.e., negative log-likelihood) equivalent to minimizing the Sum of Squared Error (SSE) when a Gaussian form is assumed for the likelihood p(æ[lz)
Derivation Variational Free Energy derivation for a State Space Model (SSM) in equation 3. Latent states are denoted with z, whereas observations are denoted with a. We assume non non-linear transition and emission function (i.e., and g); therefore a variational approximation is needed to approximate the true posterior p(ze|zt-1.2)) ②z) with surrogate posterior q(z)). As derivation the goal is to minimize the divergence between the true posterior and the approximate posterior (ie., DEL
Note that, for notation brevity, E. IF [gggg(=))
((z))
DEL
Bog
(KL Divergence definition)
p 21 - x1)
a ((z))p( )p(p 2 2
Eog
(Bayes Theorem)
p 22, 2 )p((
9 z )p( x 2/
=Eylogg
(Conditional Independence)
p( zt, 3 pp ze
1( (2t)
DxL(g(z))ll((e/ke-..o = Eng llg
+ Eoglogg(azlzz--1 (Linearity of expectations)
p(a; ze )p( ze 21
Variational Free Energy
We can minimize the Variational Free Energy term, which reduces to the log-likelihood of two prediction enror terms and the negative entropy of the approximate posterior q(zi) as shown below.
g(2))
Eg flog p(zz)zz)p(z)(zz
Ey llog
+ E.g log p(x, (z,)
E.g log
pl z(
q
Latent State Evror
Observation Eroor
entmpy Hig
Minimizing the Variational Free Energy above forces g(zz) better approximate the true posterior.
A.2 Gaussian Mixture Model and Hopfield Recall
Derivation We derive theorem which states that the maximization of the log-likelihood of p(ax, (2.) in the form of a Gaussian Mixture Model is equivalent to the recall function in Hopfield networks (Le., eqn 1), where the means of the GMM (i.e,, { represents the attractors of Hopfield model. To maximize the log-lokkellnn lilkeeiiehodd compute its partial derivative with respect to
16

---

p(gp(z) = Ox
log Σ we (2: He: E.)
dæ
0
We (x;; Me: De)
Ees1 We Nix Ene
dx
i
W dæ
Nax Mo: En)
2 We (ax Hos Ene
Wc 1 ---- (x (++) (a: Ee)
Ees We Nax;; Mo: Ee)
De We (N Ma. En)
Dj (x ma)
Een We N(x; Mo En)
By setting the partial derivative of the log-likelihood to O, we can estimate the value of x which maximizes the function logp(æ(zz).
0
log log p(x]2)) =0
dx
Eg We -N Mo: E.)
E,
(x 11)) =0
Ee=1 We NNxx; Mes Ex)
E === We N2x:; M4: Ee)
Dot0 x
Ee We
N(x; Mo: Ee) D 1e
Ee Een We Mx Moo Exc
E W--NW Mo: Ec)
Finally, rearrange the equation in terms of x and show that it is equivalent to the Hopfield recall function where the recall value x equals a weighted average of the attractors (i.e., means of GMM (u with the weights being similarity score function.
x We N(:: M. Ee) Σ. He
Eg We N(x; p4: E.) Σ
We Nix; Mey Ee) E-1
@
P projection
we Nix; p Ec) E
similarity score
A.3 Expected IoU of Random SDRs
Theorem Consider two SDRs with sparsity defined as random variables p u(0, 1) and q ~ u(0,) 1), the expected Jaccard Index (i.e., IoU) of the two random SDRs follows:
pq
p p+a-1 -+4-PM PI
Given the sparsity random variables of both SDRs (i.e., and q) and the SDR size n. the number of active bits at the same location in both SDRs equal to the joint probability both SDRs being active multiplied by the SDR size (i.e., npy). The union of both SDRs is the total number of active bits minus the active bits in both SDRs, which is equal to np + ng npg. Therefore, the py
Proof:
expected intersection over union is mp+ng pop-ng-roou hpg
P+4 Pq
17

---

Analytical Vs. Empirical Expected loU
0.5
Analytical p=0.1
Empirical p=0.1
0.4
Analytical p=0.5
Empirical p=0.5
Expected loU
0.3
0.2
0.1
0.0
0.0
0.2
0.4
0.6
0.8
1.0
q
Figure 5: Empirical Validation of Theorem
Empirical Validation: We perform empirical validation of the above theorem shown in figure 5. The sparsity of the first SDR fixed at 0.1 and 0.5. We vary the sparsity the second SDR between 0.0 and 1.0 in steps of 0.1 and calculate the average IoU over population of 1000 pairs of SDR for every setting. Empirical results agree with the derived formulation in theorem
B Notations
The notations used in our paper are summarized in Table
C
Datasets
Synthetic For synthetic experiments, we generate SDRs with the specified size N, and uniformly initialized active bits W to match the required sparsity S. many of the experiments, No set to 100 with active bits unless otherwise specified. For Hopfield experiments, we set the sparsity to 50% improve its performance.
Protein Sequences We use the dataset ProteinNet 7 [7] to extract protein sequences. Each sequence consists of chain of Amino Acids. In the dataset, there are only 20 different types of Amino Acids (i.e., vocabulary), creating long protein sequences with hundreds of Amino Acids. The dataset is reported in the fasta format, where each Amino Acid is represented with single-letter code. We create dictionary mapping from the Amino Acid types to random SDRs with N, = 100 and W = train the models. When choosing the sequences, for all the dataset sequences to avoid ambiguous predictions in the continual learning evaluation. A sample of the protein sequence is provided below:
that the starting Amino Acid is unique
Text To evaluate the generative ability of PAM, we use dataset of the most frequently used English words. For preprocessing, we extract one hundred 4-letter words from the dataset and create mapping dictionary from all the unique letters in the dataset to random SDRs with N. 100 and
18

---

Table 1: Table of Notations
Symbol
Description
A
Learnable transition weight matrix
B
Learnable emission weight matrix
Observation time
Noisy observation at time during recall Posterior Latent state after observing æ, (i.e., single possibility)
2t
21
Prior Latent state before observing æ, (i.e., multiple possibilities)
Predicted latent logits at time (i.c., A - before applying threshold
Threshold function: NRNx ~ {0, [0,11NxXN5 NexN, or RR.. + {0, 1
Hebbian learning strength for adjusting the synaptic weights
Projection operator adds context Id observation state Projection operator removes context from a 2d latent state
r
Function for computing the indices of active bits in an SDR
Random permutation function
Ne
Input size pattern
N
Number of neurons per minicolumn for context encoding
W
Number of active bits in Sparse Distributed Representation (SDR)
S
Sparsity of SDR, calculated as W/N
T
Number of patterns in sequence (i.e., sequence length)
Degree of polynomial in Hopfield separation function
L
Number of Layers used in temporal Predictive Coding (IPC)
W 5 (except for AHN; W 0.5.N)). The dataset contains many words with ambiguous future possibilities. The selected words are provided below:
that with they have this from word what some were
when your said each time will many then them like
long make look more come most over know than call
down side been find work part take made live back
only year came show good give
very just form
help line turn much mean move same tell does want
well also play home read hand port even land here
must high such went kind need near self head page
grow food four keep last city tree farm hard draw
left late real life open seem next walk ease both
Vision In our experiment, we evaluate on sequences extracted from Moving MNIST [60], CLEVRER [68] well as synthetically generated sequences of CIFAR [33] images. In order to convert images to SDRs and SDRs back to images while encoding semantics into the SDRs, we design an SDR AutoEncoder. The goal is to force the bottleneck representation of the autoencoder to become a sparse binary representation, where two visually similar images would result in two SDRs with high overlap of active neurons. We simply design CNN autoencoder with 3-layer CNN encoder and 3-layer CNN decoder and apply top K binarization operation on the bottleneck embedding during training. The full architecture of the SDR autoencoder is shown in Figure 6.
In practice, we use weighted average of the SDR and Dense representation to allow gradients of the reconstruction loss to freely propagate into the encoder. The weight of the SDR (i.c., a) is gradually and linearly increased (from 0.0 1.0) with the number of training epochs. This gradual increase fundamental to the training of the SDR Autoencoder as it smooths the loss landscape and allows the model distribute the active bits on the full SDR. The total mse loss becomes Lenc + Lrecon- We use Adam optimizer with learning rate of x 10-4 For Moving MNIST, we use bottleneck embedding
19

---

recon
SDR
Top K binarization
CNN Encoder
Sparse
CNN
Rop
0.23
0.05
0.79
x -0) a)
Dense
Lene
Figure 6: Architecture of the SDR Autoencoder
NN. of size 100 with 5 active bits, whereas, for more complex datasets (i.e., CLEVRER, CIFAR). use SDR of size 200 with 10 active bits. We show examples of the autoencoder reconstruction with full binary SDR (i.c., & = 1) for all three datasets Figure 7.
Moving MNIST
CLEVRER
CIFAR
Image
Reconstruction
Image
Reconstruction
Image
Reconstruction
3
3
3
7
5/
5/
4
4
5/
5/
5
5
4
4
Figure 7: Examples Autoencoder reconstructions from SDRs for all three datasets
20

---

D Implementation Details
In this section, we describe the implementation details and hyperparameters of each method. For each model, optimize single set of hyperparameters for all the experiments.
The neurons in both, transition and emission, functions are fully connected. We do not assume any of the weight matrices are symmetric. All synaptic weights are initialized by sampling from normal distribution with mean of 0.0 and standard deviation of 0.1. All n)* values in Equations 7 & 8 are set to 0.1. dg is set to -0.1, while V/4 is set to 0.0 to avoid forgetting previous possibilities when learning new transitions; PAM learns union of possibilities. The threshold for the 5 function is set as a function of the SDR sparsity. For the transition function, we use threshold of 0.8W, where W is the active number of bits in the latent state (z) SDR. For the emission function, we use a threshold of 0.1W, where W is the active number of bits in the observation state (x) SDR. During offline generation we sample an initial 2 from with W generation, we set the maximum number of attractor iterations to 100 but stop iterating when the energy of the state converges iteratively until the transition is learned before learning the next transition. This iterative weight update makes the model insensitive to the hyperparameter values 7, Both A and B are always clamped in the range [-1, The states z are flattened into a single dimension before applying the learning rule in Equation Binary representations (i.e., (0, 1)) are used as inputs.
PAM
active neurons. During
local minimum. During sequence learning, we update A and B
Temporal Predictive Coding For the tPC architecture, we use learning rate of le-4 for 800 learning iterations. When 2-layer tPC model is used, the inference learning rate is set to le-2 for 400 inference iterations. Also, the hidden size is set to twice the input size. We found that these parameters work best for all of the experiments and allow the model to fully converge. Bipolar representations (i.c., {-1, 1)) are used as inputs.
Asymmetric Hopfield Network The Hopfield model does not require hyperparameters other than the ablated separation function. In many experiments, degree set to or 2. Bipolar representations (i.e., {-1, 1)) are used as inputs.
polynomial separation function with
E Experiments
In this section, we describe the setup of each figure in the main paper and provide additional quantitative and qualitative results for each task. All experiments are run for 10 different seeds/trials. We report the mean and standard deviation in all the figures and tables.
E.1 Sequence Capacity
Figure 3 A This experiment plots the maximum offline sequence length (i.e., sequence capacity, Tmaz) at different input sizes. The input size N₂ is varied from 10 to 100 while the number of active bits W is fixed 5. We compare variants of model with N₂ coding (tPC) and Asymmetric Hopfield Network (AHN). We observe that AHN completely fails as the sparsity S of the pattern decreases. Therefore. we also compare AHN with the sparsity set to 50% (i.e., W = 0.5NN). For AHN models, we experiment with a polynomial exponential function with degrees and 2, recently proposed [12] and used for evaluation in recent papers [62]. All models in this experiment are set to provided. PAM outperforms all other methods and has the potential improve further by expanding the context neurons Ng. The patterns in this experiment are uncorrelated, such that each pattern has active bits that are uniformly initialized.
4 and 8 to temporal predictive
in an offline manner, where only the first input is
Figure 3 B This experiment plots the effect of sequence correlation on the maximum offline capacity. The higher the correlation value, the more exact repetitions of patterns are available in the sequence. We enforce correlation by limiting the number of unique patterns (i.e., vocab) used to create the sequence. All patterns in this experiment are set to a size of N, = 100 and W = 5 (except for AHN, which set at W 0.5.N). Results show that the capacity of all other methods sharply drops when correlation is introduced. PAM retains most of its original capacity.
21

---

Figure 3 & In this experiment, we provide qualitative example of short sequence (T = 10) with high correlation (0.8). The sequence learned by all the methods; then we perform offline (E) and online (F) recall on the sequence. We use the SDR autoencoder to create SDRs from these CIFAR images for training and recall. The SDRs have a size N₂ of 200 and W 10. In the offline recall, only the first input is provided and the model auto-regressively generates the full sequence using its own predictions at every time step. online recall, the models perform a single-step prediction and always use the ground truth input at every time step perform predictions. Results show that only PAM can retain a context of correlated sequence and accurately predict into the future based on this context.
Figure A In this experiment, we show the effect of scaling the model context memory beyond simple N₂- 4 and N = 8. We show that when using Ng. = 16 and N₂- = 24, PAM can model much longer sequences. We vary the input size No from 50 and report the offline sequence capacity of the model ablations.
PAM N₂
PAM N2 =8
PAM N2 24
AHN dm1
AHN dm1, W=0.5N, AHN d W-0.5M,
PAM N2
PAM N2 16
tPC
AHN dm2
A Offline Sequence Capacity Input Size
B
Online Sequence Capacity Vs. Input Size
1400
1200
10a
1000
mma
800
Tmex
10a
600
400
10a
200
0
10
20
30
40
50
10
20
30
40
50
60
70
80
90 100
N
No
Figure 8: Additional sequence capacity experiments. A: scaling of the offline sequence capacity with context memory size N₁ and input size No. B: Online sequence capacity of various methods.
Figure 8 B Similar to the experiment plotted in Figure 3 A. we report the sequence capacity with input size Ne. However, this experiment evaluates the online generation capacity, where the model uses the correct pattern at every prediction time step instead of using its own prediction from the previous time step. Results show that PAM significantly increased in capacity (three times in some cases), whereas the other methods have not increased as much modeling longer sequences.
Figure 9
We provide additional qualitative example on different highly correlated sequence. The result shows a different failure mode for AHN, whereas PAM still performs well.
E.2
Catastrophic Forgetting
Figure A We benchmark the performance of difference models in the challenging continual learning setup. The models are expected to avoid catastrophic forgetting by not overwriting previously learned sequences. In this experiment, we use 50 sequences, each with size N. 100 and length 10. We vary the correlation of the sequences from 0.0 to 0.5 and compute the backward transfer metric with the normalized IoU as the measure of similarity. Results show that AHN can avoid catastrophic forgetting when the sequence is uncorrelated but quickly drops in performance with correlation. tPC fails to retain learned sequences regardless of correlation. PAM performs well with more context neurons Ng. When setting N₂ to 1, the model fails to retain its knowledge due to the decreased context modeling capability with single context neuron.
22

---

Memories
Offline
Online
Input
Hidden
L-2
AHN
Figure 9: Additional qualitative example of correlated sequential memory with CIFAR images.
Figure In this experiment, we report the performance the models on protein sequences. This is a more challenging setup due to the long sequence (a few hundred on average) with high correlation (only 20 unique Amino Acids). We show similar trend where the other methods fail due to high correlation or sequence lengths. PAM outperforms the other methods when using context memory N& of 16 or 24. All Amino Acid types are converted to fixed and randomly initialized SDRs with Ne 100. The sparsity is set similar to sequence capacity experiments (i.e., W 5 and W 0.5N,).
Figure F We provide qualitative results on a simple experiment with 2 sequences from moving MNIST. The models learn the first sequence and then learn the second sequence. The models are not allowed train on the first sequence after they have trained the second sequence. We then perform online generation on the first sequence with all models. We use the SDR autoencoder to generate SDRs for all images in the sequences; the SDRs have N. = 100 with W (for all methods except full sequence even after being trained on another sequence. Other methods fail in this simple task, even in online recall setup.
AHN). The results show that PAM can recall the
Figure 10
We provide continual learning results similar Figures 4 A & B; however, instead of offline generation, we perform the evaluation in online manner.
We provide additional qualitative results on Moving MNIST, as well as quantitative results averaged over trials of MNIST sequence pairs. These quantitative results are reported as the Mean Squared Error of the reconstructed image. Results show that PAM reports the lowest error with much smaller variance.
Figure 11
Tables 3, 4, & The Backward Transfer metric (BWT) to evaluate catastrophic forgetting is computed by taking the average of the performance on previously learned sequences after training on new sequence.
addition to plotting this average previous experiments, we provide
full tables for one of the experiments as an example. All tables show results on 10 sequences and N. 100. The BWT metric is calculated as the average of the similarity metric reported these tables.
23

---

N2
d=1, W c
PAM My 44
PAM N₁ =
16
tPC
AHN d=2, W= 5N.
Backward Transfer Vs. Correlation
Backward Transfer Vs. Protein Sequences
1.0
Backward Transfer (BWT)
Backward Transfer (BWT)
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
10
15
20
Correlation
Number of Protein Sequences
Figure 10: Catastrophic forgetting experiments in an online manner. Results are shown on synthetic dataset of SDR sequences with different correlations and on protein sequences.
Numeina
Moving MNIST Image Reconstruction Error
+7
0.040
Thm
0.035
0.030
0.025
#7
0.020
0.015
0.005
7
#
7
PAM
LPC
AHN d=
IPC -
AHN d-2, W=0.5M:
-V-
W-0,5M,
Figure 11: Additional qualitative catastrophic forgetting visualization Moving MNIST and quantitative results on the reconstruction eтrоr of 10 Moving MNIST random examples
Table 2: Catastophic forgetting experiment results 10 sequences for PAM with N₂ 1. The table shows the mean normalized loU and standard deviation of previous learned sequences after training on new sequences. The Backward Transfer metric is the average of all the shown numbers. Results are averaged over 10 trials.
Test
Sequence ID
Train
10
0.992 0.220
0.654 0.252
0.586 + 0.3314
Segueme
0.634 0.256
0.0
0.553 0.348
0.619
0.259 0.229
0.817 0.200
0,823 0.215
0.817 0.22
0.501 0.322
0.895 0.289
0.566 à 0.329
0.637 0.209
0:239)
0.253
0.11
0.29
0.527 0.236
0.571 0.297
0.204
01645 (1:266
0.605 0.284
0.267
0 0.311
0.498 (1266)
0.257
24

---

Table Catastophic forgetting experiment results on sequences for PAM with N₂ 4. The table shows the mean normalized IoU and standard deviation of previous learned sequences after training on new sequences. The Backward Transfer metric is the average of the shown numbers. Results are averaged over 10 trials.
Separnce ID
0.000
0.000
0.000
1.000
0.000
=:
=:
1.000
1.000
(1.01)
1,000
0.000
0.000
1.000 0:000
0,000
Table 4: Catastophic forgetting experiment results on 10 sequences for tPC. The table shows the mean normalized IoU and standard deviation of previous learned sequences after training new sequences. The Backward Transfer metric is the average of all the shown numbers. Results are averaged over 10 trials.
Train
Seqarnoe ID
0.326 0:357
0.393
0.256
0,062 0:00
0.103
0.197
0.322 =
0.170 0:12
03016 0.019
0.023 0.28
0.00%
0.052 8.089
0.133 0:009
0.288 = 0,302 0.237
Table 5:
Catastophic forgetting experiment results on 10 sequences for AHN with and W = 0.5.N. The table shows the mean normalized IoU and standard deviation of previous learned sequences after training on new sequences. The Backward Transfer metric is the average of all the shown numbers. Results are averaged over 10 trials.
Sequence
Sepuioe ID
00:34
0.338
00:34
0.780
0.236
01334
0,780 = 0.226
0.595 0.334
0,780 :: 0.226
= 0.257
0.689 192
0.595 0.334
0.765 0.338
0.780 = 0:226
0.667 0.226
0.558 0.257
E.3 Multiple Possibilities Generation
In this task, we evaluate the models' ability to generate meaningful sequences and recall the full dataset despite being presented with multiple valid possibilities. Ideally, the models are expected to sample single possibility if trained on sequences with ambiguous future continuations (equally valid possibilities). This is a challenging task for most biologically plausible (e.g.. tPC, AHN, etc.) and implausible (e.g., transformers, RNNs, etc) models. Most approaches assume the existence of full set of possibilities and transform the task from regression to classification (e.g., LLM). For vision tasks, some methods (VQ-VAR and its variants) cluster the dense representations create this set of possibilities and perform classification. We do not assume the existence of a full set of possibilities but instead perform a true generative evaluation as a regression task.
Figure 4 C In this experiment, we compute the average normalized IoU of the generated words. The model's 's ability to generate full sequence with high loU means can produce sharp single predictions despite being trained multiple equally valid future predictions. As the number of words
25

---

increases, the performance of other models decreases as they struggle to model ambiguous future predictions; however, PAM outperforms the other approaches by sampling from these possibilities.
This experiment evaluates the ability of the models to recall the dataset words. We compute the recall as the number of valid unique words generated divided by the total number of words in the dataset. Since PAM is generative stochastic model, the recall increases with every generation. The other methods are deterministic and, therefore, do not report an increase in dataset recall with more generations. The other methods completely fail to generate any meaningful words. We use an average IoU threshold of 0.9 classify a generated word correct, similar sequence capacity experiments.
Figure D
We provide qualitative results by showing the unique generated words by different models after 5 dataset generations. PAM Nº = 4 generates some of the dataset words but also generates many wrong words. By increasing the context memory neurons N₂ 8, the model generates many more correct words and reduces the false positives. The other methods cannot generate meaningful words.
Figure 12
part call live side both play from move open - city- - over name form wore port more make came hard most take line some life - self like made same near than many late land does this down will farm hand very-
Full Dataset
tell year- good
-when sald grow last know them only
read want with - what back just such need - case keep well tree look head kind give page even food home four- they long here time that much come have seem next mean
walk went help word find
real high
meam next Barry EVbG
page
keesi
word
Game Doien near draw mean
want left hiSR
same
Reem
City Jum
will - bada move Brow wore here
koee
free from
real
such
dORS-
moee-
hand - four vary good
PAM-4
much came - back head lapa
Ina show with most
land
port
Ward
food
whee
oven Base
your Bnly play
what moc EVDl Sliee
home
call mons must
harg
high
mic
Doon
hame MOsS when
JuEn
from
and
nex
move
BIde
STOW
Walk
Werrd
Whot left
herre
give Rear 1OOI
wani
home
Frrm
Iree
word
came
real
PAM-8
call
Spen
form
good
help does
very
high
your
play
came
back
bath
Soom
head
WC
boon
IPC
Nbea
0888
h888
8868
0688
baRa
AHN
Dana
desa
haan
Figure 12: Qualitative results showing the generated words from PAM, tPC [62] and AHN [12] Words highlighted in green available in the dataset (i.e., True positives). Words highlighted in red are not available in the dataset (i.e., False positives).
E.4 Noise Robustness
Figure C To evaluate for noise robustness, we plot each model's performance (Normalized IoU) with varying levels of noise added in an online generation setting. The noisy inputs are created by changing a percentage of the active neurons to different uniformly chosen neurons in the SDR. The noise is computed as a percentage for a fair comparison across different SDR sparsities (c.g., IPC vs. AHN). We show the results for sequences of lengths T 200 and no correlation. PAM has the ability to compare the noisy input representation to the learned attractors to recover the correct clean input. Therefore, even when the SDR is completely changed, PAM relies its predictions and completely ignores the noisy input. During generation, PAM always generates (or corrects a noisy input) from within the predicted set of possibilities. The other approaches use the noisy inputs during recall, which affects their performance.
Figure 4 E We provide qualitative results on the CLEVRER dataset. The memories sequence is learned by all the models, then a noisy sequence is used during generation. We only add noise starting from the second pattern in the input sequence. The results show that tPC models are performing relatively well, yet they are still outperformed by PAM N₂ = 8. We set N. = 200 in the SDR autoencoder to learn the SDRs used in this experiment. We use 40% noise in this experiment.
We perform additional experiments on varying the sequence lengths and the correlation in the sequence, all the other settings remain the same as in the experiment of Figure 3 C. The results
Figure 13
26

---

show that with shorter sequences ( 200), no noise, and no correlation, all the models recall the learned sequence well. When higher correlation used, 2-layered tPC performs relatively well with short sequences (i.e., = 10) but fails with longer sequences (i.e., 7 100). The Hopfield model fails more with correlation than sequence length. The added noise affects all reported methods except for PAM due to its ability to rely on its predictions and attractors to clean the noisy signal.
Figure 14 We provide an additional qualitative example with similar trend to Figure 4 E. We also provide quantitative results of CLEVRER averaged over 10 experiments. The mean squared error of the generated sequence for multiple models at different noise levels is reported. It is clear that PAM outperforms all methods, and 2-layered (PC is the second best.
PAM Mo.
--- LPC --- LPCL
AHN d=2, W=0.5M:
Sequence Length=10 Correlation-O.0
Sequence Langth=10 Carrelation-O.3
Saquence Length=11
Correlation-0.5
Nomalized lou
Normalized
0.0
-0.2
-02
-0.2
-0.4
-0.4
00.4
40
20
40
60
100
active changed
active changed
% active bits changed
Sequence Length=100 Correlation=0.0
Sequence Length=100 Correlation=0.3
Sequence Length=100 Correlation
Normaizze loU
04
Normalized
0.2
0.2
0.2
0.0
0.0
0.0
02
02
0.
-044
-0.4
-044
changed
changed
%
changed
Figure 13: correlations.
The effect of noise on online generation with varying sequence lengths and sequence
Memories
CLEVRER Image Reconstruction MSE
0.0035
W-0.5ML
EPC
holp Foxt
Nean Squered Error
0.0022
0.0015
0.0005
0.2 % active bits changed
0.4
Figure 14: Additional gualitative example of online generation with noise CLEVRER dataset, and quantitative results of reconstruction enror over 10 CLEVRER examples at different noise levels.
27

---

E,5 Efficiency
Figure 3D D We compare the efficiency of the models and show that PAM is at least two orders of magnitude efficient than rPC with 2 layers. A single layer tPC is almost equivalent PAM with high context memory of N₂ 24. AHN is highly efficient as the model is not usually trained, but the recall equation is used instead. Therefore, we exclude AHN from the comparison.
E.6 Additional Results
A Performance with Increasing Sequence Length 4
Performance for Varying Connection Density
Backward Transfer Vs. Protein Sequences
100
Density100 10
PAM Onine
Deruity 9
Normalized Averege leU
Densitywl
0.8
0.8
Backward Transter
07
0.6
PAM
0.6
S
0.4
TRANSFORMER
0.
0.2
02
Sequence Length
Seqquence
Number Procoin Sequences
Figure 15: (A) Performance of online and offline PAM (N₂ 50, N₂- = 4) generation with sequence
length. (B) Performance of PAM (N₂ 50, N₂ 4) generation for different connection densities. (C) Backward transfer performance for PAM and 2-layer transformer on the protein sequence task.
Figure 15 A We show the performance of PAM in an online and offline generation task as the sequence length increases. In offline generation, incorrect predictions accumulate quickly, leading to sharp drop in performance. However, online generation attempts to correct inaccurate predictions; therefore, performance smoothly degrades with sequence length. Note that both experiments (i.e., online and offline) use identical seeds/sequences for a fair comparison.
We provide additional results on testing the effect of connection density. While neurons in PAM are currently densely connected (similar to Hopfield Network), we show that PAM can still perform relatively well in a sparser setup.
Figure 15 B
Figure 15 C We provide additional results comparing PAM to 2-layer transformer on continually learning protein sequences. Backward transfer results show low performance of transformers in remembering older sequences after training on new sequences. Table 6 supplements these results by showing detailed srquence-by-sqqueee performance values. Transformers excel at learning the current task (diagonal values) but suffer in remembering previously learned tasks (lower triangle values).
Table Catastophic forgetting experiment results on protein sequences for 2-layer Transformer. The table shows the mean normalized loU of previous learned sequences after training on new sequences. The Backward Transfer metric the average of all the numbers in the lower triangle.
Test
Sequenee ID
0.749
0.000
0.903
Sequence ID
0.054
0.051
0.066
0.D458
0.002
0.931
0.066
0.056
0.054
0.035
0.043
0.076
0.945
0.061
0.042
0.104
0.099
0.041
0.071
0.727
0.039
0.039
0.078
0.019
0.052
0.026
0.077
0.078
0.723
0.062
0.077
0.135
0.042
0.029
0.008
0.032
0.065
0.073
0.955
28

---

F
Discussions
E.1 Biological Plausibility
PAM inherits assumptions from Hierarchical Temporal Memory (HTM) [25] that the sequence modeling takes place in Layer IV of the cortical column, where it receives both thalamocortical sensory driving input and contextual information. Sensory input arrives on the proximal dendrites of pyramidal cells, while contextual information reaches the basal dendrites, priming specific contexts by depolarizing neurons. Depolarized neurons then fire first and inhibit other neurons within the same minicolumn via lateral inhibition, process assumed be mediated by interneurons (possibly Double Bouquet Cells (DBCs)) [49]. HTM assumes interneurons [40] perform lateral inhibition during prediction but does not learn the synapses of the interneurons. Interneurons are also assumed to perform the inhibitory tasks of the attractor model [14].
Biological plausibility in PAM mainly refers to excluding the obvious biologically implausible learning rules, such as backpropagation or copying of weights. This allows models like PC and HN to be referred biologically plausible despite clear contradictions with anatomy [54, 14]. Moreover, PAM also constrains its learning and representations to sparse and binary cell assemblies, which improves the plausibility of the model. Therefore, a biologically plausible model here refers the ability to continually learn using only local learning rules (e.g., Hebbian, anti-Hebbian, STDP) and sparse binary representations. However, we do not intend accurately map every detail in PAM to the anatomy of the neocortex.
F.2 Future Directions
In future work, we plan to implement hierarchy of PAM blocks that supports compositional, part- whole predictions. Unlike simply stacking layers as in Transformer or CNN architectures, our aim establish hierarchy where higher-level blocks make predictions based on broader, contextual patterns. For instance, in vision, higher levels could predict entire objects based on multiple visual cues, while in NLP, they could predict phrases sentences rather than individual tokens. To achieve this, each PAM level would send summary representation of the current sequence (its latent state) to the next level for recursive, higher-order processing.
Additionally, we plan to explore top-down feedback mechanisms between PAM blocks inspired by the hierarchical feedback observed in the neocortex. In the visual system, for example, top-down signals travel from higher regions (e.g., V2) to lower ones (e.g.. V1), reaching the apical dendrites of pyramidal neurons. These connections originate in layer VI(a) of the higher region and form en-passant synapses in layer VI(a) of the lower region in the hierarchy (e.g., V1). These connections reach Layer and provide feedback the apical dendrites of other layers [61]. In PAM, similar feedback pathway could enable higher-level blocks to modulate lower-level processing based on partial observations, refining predictions by ruling out certain lower-level possibilities that conflict with higher-level context. For example, the attractor in the higher-level block can reach a decision on the object being seen (e.g., human) based on partial observation (VI: layer III V2: layer IV), and therefore, the feedback inhibitory connections on the apical dendrites can rule out some lower-level possibilities (e.g., paws).
G Sparse Distributed Representations
The neocortex stores and represents information using sparse activity patterns, as demonstrated by empirical evidence [4]. Inspired by HTM [25] and neuroscience- based theories of cortical function, we use Sparse Distributed Representations (SDRs) as the main representation format of PAM. An SDR a sparse binary representation of a cell assembly where only a small fraction of the neurons in the SDR are active at any time. The location of these active neurons encodes the information that represented by this SDR. In this section, we describe some useful properties of SDRs and discuss their robustness to noise opposed to dense representations.
29

---

G.1 SDR Properties
SDRs are used to represent rich sensory information in the neocortex as sparse activity pattern. Therefore, from mathematical viewpoint, an SDR must have the ability to represent many patterns and easily distinguish between them. The capacity of an SDR can be calculated as the possible combinations of locations where neurons can be active. Consider an SDR with size N and number of active neurons W. The total capacity of this SDR is computed as shown in Equation 12.
NV W
N! W!(N - W)!
(12)
Based the above Binomial coefficient equation, it may seem that sparsity is not optimal for capacity as the capacity will be the highest when W is exactly half of N. While capacity is important, we aim to represent multiple possibilities as union of SDRs and, therefore, minimize the overlap between them. From an information-theonr between SDRs ensure that each SDR carries unique information and the union represents more comprehensive and diverse set of features. We can minimize the expected loU by using lower sparsities as shown Theorem In our experiments, we use N 100 and W 5, which results in capacity of R 75 X 10t and an expected IoU of 0.02. However, when scaled up more typical values of SDR sizes and spansities the neocortex [25, 4] (i.c., N = 2048, W 40), we get capacity of x 2.37 x 10 (more than the estimated number of atoms in the observable universe 10th) and expected IoU of Rs 0.01. A sparsity of 0.5 maximizes the mutual information and results in expected IoU of 0.33, which cannot be used to represent multiple possibilities SDRs in spite of the optimal capacity. In practice, the size N of the SDR is increased to increase the capacity, and the sparsity W/N is decreased minimize the expected overlap.
viewpoint, the goal is to minimize mutual information
union of
G.2 The robustness of SDRs
Sparse representations naturally minimize the overlap between random SDRs; therefore, they are very tolerant to noise. To visualize this robustness property of SDRs, we design an experiment (Figure 16) where we train an SDR autoencoder with different sparsities and then decode SDRs at various levels of noise added. When the sparsity is increased to 50%, there is high chance of overlap between SDRs. Therefore, a small amount of noise can cause collisions between SDRs. However, 5% sparsity tolerate much more noise without overlapping with other SDRs.
30

---

SDR Sparsity
SDR Sparsity
SDR Sparsity
5%
25%
50%
5%
25%
50%
5%
25%
50%
0.0
8
0.1
Noise Level added to SDR
0.2
55g
58
0.4
0.5
x7g
0.6
0.7
0.9
1.0
Figure 16: Three examples of decoding SDR with different noise levels. The results are shown for SDRs with different sparsities trained in an SDR autoencoder on the CLEVRER dataset.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are supported by experimental results and theoretical proofs where applicable. Derivations are provided in the supplemental material.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the limitations section has been moved from the supplemental material to the main paper for the camera ready version.
Guidelines:
31

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We provide proofs and derivations in the appendix, where applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide general implementation details for all the methods used. Addition- ally, we describe the experimental setup for every plot/table in the main paper and in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our codebase (now released) includes a script for every single experiment to accurately reproduce the plots in the paper. We fix the seeds for exact reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /twpsscccppbllcc//eecccp ceolliic/ggiidsseeee /CodeSubmis ssi sisoiiiy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
33

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide implementation details for all methods and the datasets' details in the appendix. The code also contains separate configuration file with all the needed parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are repeated 10 times different seeds. The means and standard deviations are provided in all plots and tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the compute resources required for PAM (i.c., CPU) and plot a comparison of the time required by each method different parameters in Figure D.
Guidelines:
34

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httptp/s//eerrp....c//lt////e/e www.ietehiccliieeeeeeeeennee
cc
Answer: [Yes]
Justification:
The paper conforms with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: PAM powerful sequence modeling algorithm. We do not see direct positive or negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
35

---

Answer:
[NA]
Justification:
No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Authors of comparison methods and datasets are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
36

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "tzqy6wfzoG", "reviewer": "Reviewer_YbwU", "summary": "This paper introduces Predictive Attractor Models (PAM), a novel biologically-inspired approach for sequential memory and prediction. PAM combines a predictive model based on sparse distributed representations (SDRs) with an attractor model for generating future predictions. Key contributions include an online learning algorithm that encodes temporal context without catastrophic forgetting, the ability to represent and sample from multiple possible future predictions, noise robustness through learned attractors, and formulation as a variational inference problem within a state space model framework. The authors evaluate PAM on various sequence learning tasks, demonstrating superior performance compared to temporal Predictive Coding (tPC) and Asymmetric Hopfield Networks (AHN) in terms of sequence capacity, avoiding catastrophic forgetting, generating multiple possibilities, and noise robustness.", "questions": "1. How does PAM's performance change with increasing sequence length?\n2. Can PAM be extended to handle variable-length or incomplete sequences?\n3. How does PAM perform with more structured types of noise?\n4. Can you provide more insight into the attractor model's learning of multiple possibilities?\n5. Have you explored more sophisticated SDR encoding schemes for complex inputs?\n6. How might PAM incorporate top-down feedback or attention mechanisms?\n7. What specific neural circuit predictions arise from your model?", "limitations": "- The requirement for sparse binary representations limits direct application to dense inputs. \n- The use of a separate autoencoder for vision experiments adds complexity. \n- The paper lacks thorough discussion of computational complexity and memory requirements for larger-scale problems.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novelty in combining predictive coding with attractor dynamics for sequential memory.\n- Strong theoretical grounding within a variational inference framework.\n- Comprehensive evaluation on various tasks and datasets.\n- Biological plausibility through local learning rules and sparse binary representations.\n- Addressing key challenges in sequential memory (catastrophic forgetting, multiple possibility generation, noise robustness).", "weaknesses": "- Limited comparison to modern sequence learning approaches (e.g., transformers).\n- Scalability concerns for very long sequences or high-dimensional inputs.\n- Lack of hierarchical processing discussion or implementation.\n- Insufficient analysis of hyperparameter sensitivity.\n- Limited discussion of failure modes and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and positive remarks on novelty, theoretical grounding, and comprehensive evaluation of key challenges in sequential memory. We have provided additional figures to address your concerns.\n\n**Weaknesses**\n\n* Similar to other papers on biologically-plausible sequential memory [1], we have mainly focused on comparing to biologically plausible approaches that do not employ backpropagation for learning (i.e., only local learning rules). Transformers and deep learning based approaches (e.g., LSTMs, MAMBA, etc.) use backpropagation through time. While these modern sequence learning approaches use backpropagation to successfully learn meaningful context representations, they still struggle against key challenges such as catastrophic forgetting, generating multiple possibilities, and adversarial robustness. In the rebuttal attachment (Figure 1D), we compare PAM against a 2-layer transformer on the task of continually learning multiple protein sequences. We show that transformers completely forget older sequences after training on new sequences, whereas PAM successfully learns the sequences without catastrophic forgetting. We also show detailed results of the transformer in Table 1. Transformers excel at learning the current sequence (diagonal values) at the expense of forgetting previously learned sequences (lower triangle values).\n\n* PAM scales well with the size of the SDR $N_c$ and the number of neurons per minicolumn $N_k$. In the rebuttal attachment (Figure 1A), we show that PAM scales up exponentially with the size of the SDRs. In our experiments, we only use $N_c$ to be 100 neurons for fair comparison with previous approaches; however, a cortical column typically contains close to 1500 pyramidal cells in layer IV only [2]. Please note that Figure 1A in the main paper shows the capacity on a log scale and that while some other approaches seem to be competitive to PAM (i.e., AHN d=2), they lose all their capacity when provided with correlation in the sequence as shown in Figure 1B in the main paper.\n\n* In the future, we plan on implementing a hierarchy of PAM blocks. However, a hierarchy in PAM is different from simply stacking layers of transformer blocks (or CNN filters). We believe in an explicit part-whole hierarchy where the higher-level performs higher-order predictions in a true compositional manner. Higher-level predictions in vision can be predicting objects after seeing multiple objects (i.e., context), whereas in NLP, it would be predicting sentences or phrases instead of tokens. In PAM, each level would need to send a summary representation of the current sequence (i.e., latent state) [3,4] to the higher level for recursive processing. We will add a detailed discussion in the supplementary for hierarchical processing and higher-order predictions in PAM. \n\n* The main hyperparameters in PAM are $N_c$ and $N_k$, which we test extensively throughout the paper. We provide additional results on testing the effect of connection density (Rebuttal attachment figure 1C). While neurons in PAM are currently densely connected (similar to Hopfield Network), we show that PAM can still perform relatively well in a sparser setup. It is important to note that we do not tune any hyperparameters for specific tasks. We use the same model and parameters for testing sequence capacity, noise robustness, catastrophic forgetting, and sequence generation with different modalities (i.e., text, vision, protein sequences, synthetic).\n\n* PAM’s performance degrades when not using enough neurons to represent the context or when SDR representations become dense such that a union of two SDRs doesn’t provide unique information about either SDR. In figures 2A, 2B, 2C, and 2D (main paper), we show the effect of low values of $N_k$. At one neuron per minicolumn, the model fails to capture context and performs worse than other models. Also, Figure 12 in the supplementary material shows how the performance of the generated words degrades with lower $N_k$ (4). \n\n**Questions**\n\n1) We provide additional results (Rebuttal attachment Figure 1B) on the IoU performance of a small PAM model ($N_c=50$, $N_k=4$) with increasing sequence length. In offline generation, incorrect predictions accumulate quickly, leading to a sharp drop in performance. However, online generation attempts to correct inaccurate predictions. Therefore, performance smoothly degrades with sequence length. Note that both experiments (i.e., online and offline) use identical seeds/sequences for a fair comparison.\n\n2) PAM can handle variable-length and incomplete sequences. The task of the protein sequence is one example of dealing with variable lengths. Each protein sequence has a different length; PAM can learn each of them and generate them in an offline, auto-regressive manner.\n\n3) We only introduce noise into the SDRs to simulate neuronal noise caused by firing rate (i.e., misfiring, random spikes, or bursts). Different types of neurons and interneurons can generate uncorrelated spikes [2]. The collective behavior of neurons in a sequence model should be capable of filtering out and inhibiting noisy action potentials. We show that the introduced attractors can effectively remove noise. We have not tested other types of noise in this work, but in future work, we plan on testing the model’s performance on sequences after the order of observations has been completely or partially randomized.\n\n*Replies to the rest of the questions are available in a follow-up comment*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MhwKRQ0YuI", "reviewer": "Reviewer_qn9J", "summary": "The paper presents Predictive Attractor Models (PAM), designed to improve sequence memory by resolving issues like catastrophic forgetting. PAM processes inputs in real-time, remembering each just once, using lateral inhibition in cortical minicolumns to preserve old memories. By generating future predictions from previously trained possibilities, PAM leverages synaptic adjustments and competitive learning, inspired by Hierarchical Temporal Memory systems. This combination enhances memory robustness and versatility across different data types, marking a significant advance in creating biologically plausible, computationally efficient memory systems for cognitive science and AI research.", "questions": "1. The implementation of PAM involves complex mechanisms, such as lateral inhibition in cortical minicolumns and synaptic adjustments, which can be more intricate compared to traditional memory models. Moreover, the dimensionality of matrices involved in PAM (e.g., matrix A and B's dimension being $(N_c \\times N _k)^2+N_c^2$ contributes further to this complexity. Such high complexity demands more computational resources and sophisticated programming techniques, which may limit PAM’s accessibility and scalability in practical applications.\n\n2. In the introduction, it is emphasized that different contexts in a sequence should lead to different perceptions of the same stimulus. However, there is a concern that PAM might merely memorize the entire sequence without effectively considering the variability in sequence content. This approach could potentially overlook the importance of recognizing repetitive structures within sequences, which are crucial for efficient sequence memory. The ability to identify and utilize such repetitions can greatly enhance memory efficiency and predictive accuracy, suggesting a potential area for improvement in PAM's design.", "limitations": "The article compares related work in the introduction and mentions some limitations.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. By processing each input only once and using mechanisms such as lateral inhibition, PAM effectively retains old memories without them being overwritten by new ones.\n2. PAM's ability to handle inputs in real-time allows for dynamic updating of memory without the need for batch processing or retraining.\n3. It can generate multiple future predictions based on a set of learned possibilities, providing flexibility and adaptiveness in response generation.\n4. PAM has been tested and shown to be effective across various types of data, including text, visual inputs, and protein sequences, demonstrating its applicability in diverse fields.", "weaknesses": "1. Need for Detailed Comparison with HTM: There is a necessity for a more thorough comparison between PAM and Hierarchical Temporal Memory (HTM). By detailing the differences and improvements PAM offers over HTM, the unique contributions of PAM can be better understood and appreciated. This analysis would help delineate PAM's novel features and demonstrate its advancements over HTM in handling sequence memory.\n\n2. Risk of Overfitting: PAM's robust capacity to utilize past information and generate predictions from learned attractors might lead to overfitting. This risk is particularly high in environments with highly variable or noisy data. Additionally, if the sequence itself contains repetitive elements meant to convey information, PAM may struggle to capture this, potentially losing the ability to extract meaningful patterns from repetitions in the data.\n\n3. Parameter Sensitivity: The performance of PAM heavily relies on the precise configuration of its parameters, including the strength of synaptic connections and the degree of lateral inhibition. Incorrectly calibrated parameters can significantly impair the model's functionality, making it less effective and reducing its overall reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the strengths of PAM in retaining old memories, continual learning, and generating multiple possibilities.\n\n**Weaknesses**\n\n1) PAM extends HTM by providing a mechanism for filtering noisy inputs and generating sequences in an autoregressive manner.\n    * The current implementation of HTM predicts multiple possibilities as a union of SDRs. This union entangles the possibilities and therefore can only be used for anomaly detection (comparing new observation to the union prediction). PAM adds generative ability to HTM, enabling it to perform autoregressive sequence generation (i.e., LLMs, video completion, etc.)\n    * HTM does not use its predictions to clean the input observations from noise. The noisy observations are used as groundtruth. PAM applies its learned beliefs and predictions to filter the noisy observations. Note that observations are filtered based on the attractors from within the predicted unions, not the entire attractor space (see the intersection in line 8 of algorithm 2).\n    * PAM provides a mathematical formulation that unites both HTM and HN in a single Bayesian framework.\n    * PAM also extends Hopfield Networks by enabling temporal sequence processing without the need for asymmetric weights. Therefore, it inherits all the convergence and stability guarantees of vanilla HN [1,2].\n2) We believe that the tendency of PAM to push noisy observations towards familiar observations (attractors) matches behavioral experiments of “filling in” information [3]. In an online learning system, we can determine how close the prediction is to the observation; if the discrepancy is high, we learn new temporal dependencies. Otherwise, we can remove the noise from the observation, which allows us to make predictions based on our version of the observation (i.e., attractor center). Being able to make predictions after seeing noisy observations can be considered generalization and contributes to adversarial robustness. We agree that PAM’s ability to capture long context may not allow it to extract sub-patterns in repetitive data. This problem can be handled in a hierarchical structure of PAM blocks, where low-level repetitions are handled by a lower-level PAM block that sends a summary of the repeated pattern to the higher-level PAM block for higher-order predictions. In this work, we focused on implementing a modular PAM block, and we will continue to work on a hierarchical architecture to capture the compositional structure in data.\n3) It is important to note that we do not tune hyperparameters for specific tasks. Instead, we use the same set of parameters for all modalities (i.e., vision, text, protein sequences) and all tasks (i.e., sequence capacity, noise robustness, catastrophic forgetting, and autoregressive sequence generation). Therefore, we believe that the model is not sensitive to parameters. In addition to extensively evaluating the performance of the model at different SDR sizes (i.e., $N_c$) and number of neurons per minicolumn (i.e., $N_k$) in the main paper, we provide additional results on varying the connection density (rebuttal attachment Figure 1C) and show that PAM performs relatively well after randomly removing some of the connections.\n\n**Questions**\n\n1) We agree that the implementation involves additional mechanisms than a simple associative memory model; however, the structure of cortical columns is complex. Minicolumns do exist in every cortical column and are usually referred to as a vertical bundle of myelinated axons, where each minicolumn has a single inhibitory double bouquet cell [4], which can directly inhibit the pyramidal neurons in the minicolumn. This results in an intricate mechanism which allows for accurate temporal predictions that cannot be achieved with a simple memory model (HN). In figure 1D of the main paper, we show that PAM is significantly more efficient than temporal predictive coding (tPC) and is performed on a CPU.\n2) Please refer to the answer to point 2 in the weaknesses section.\n\n*We hope to have addressed all of your concerns and look forward to any additional discussions*\n\n**References**\n\n[1] Hopfield, John J. \"Neural networks and physical systems with emergent collective computational abilities.\" Proceedings of the national academy of sciences 79.8 (1982): 2554-2558.\n\n[2] Ramsauer, Hubert, et al. \"Hopfield networks is all you need.\" arXiv preprint arXiv:2008.02217 (2020).\n\n[3] Papenmeier, Frank, Alisa Brockhoff, and Markus Huff. \"Filling the gap despite full attention: The role of fast backward inferences for event completion.\" Cognitive Research: Principles and Implications 4 (2019): 1-17.\n\n[4] Peters, A., and C. Sethares. \"The organization of double bouquet cells in monkey striate cortex.\" Journal of neurocytology 26.12 (1997): 779-797.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UyeJCKSrZX", "reviewer": "Reviewer_E7YU", "summary": "This paper examines a biologically plausible model for sequential memory which overcomes various issues (capacity, forgetting, size of context window) in previous models. The proposed model is essentially an extension of Hierarchical Temporal Memory which allows the model to generate predictions.", "questions": "* What does $\\mathcal N(x; \\mu_c, \\Sigma_c)$ symbolize in Eq. (4)?\n* It would be nice to examine a crucial step of the generative algorithm -- lines 7-8 in algorithm 2, which combine a Hopfield step with influence from the context -- in more detail, theoretically and/or in experiment. If something like this has been explored in the literature before, it would also be good to point to it.\n* Please define Jaccard index/IoU in the main text.", "limitations": "Since the paper claims biological plausibility, I would appreciate a more thorough and insightful discussion of what features are plausible and which are less so, with references to the relevant neurophysiology literature.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Memorizing sequences is a fundamental ingredient of cognition and remains a challenge for neuroscience, while no theoretical model has been completely satisfactory.\n* Hierachical Temporal Memory remains a compelling model of sequence memorization and attempts to expand its capabilities are worthwhile.\n* The model presented here is simple, well-explained, and composed of familiar ingredients.\n* The experiments are clear and seem appropriate.", "weaknesses": "* This paper's contribution seems to be a bit weak. What it builds upon Hierarchical Temporal Memory [1] is the ability for the model to generate predictions, which can be used to sample autoregressively or robustly handle noisy input. While the sampling capability in particular could be interesting, key properties of such a function -- perhaps most obviously, which distribution does it produce? Can it learn the input statistics? -- are not explored.\n* In relation to the above, it inherits a (slight) weakness from HTM: reliance on the ability of dendritic computation to put neurons into a \"predictive\" state, which is necessary (but not sufficient) for firing. We should be leery of overstating the biological plausibility of what remains a hypothetical mechanism.\n* The model is biologically plausible in that it relies on local updates, but various other aspects of it (especially the configuration of the minicolumns) seems to need to be set up precisely in advance, and dense connectivity is assumed.\n* The theoretical results amount to showing that sampling (an observation given context) can be written as a Hopfield recall problem. A deeper theoretical analysis of this model -- touching on capacity, number of rounds for convergence, and characterizing the distribution which is sampled from -- would significantly strengthen this paper's contribution.\n\n[1] Hawkins, Jeff, and Subutai Ahmad. \"Why neurons have thousands of synapses, a theory of sequence memory in neocortex.\" Frontiers in neural circuits 10 (2016): 174222.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive remarks on the simplicity of the model and the importance of the challenge being solved by PAM.\n\n**Weaknesses**\n\n* PAM extends both HTM and Hopfield Network (HN). It allows HTM to autoregressively generate sequences (enabling HTM to do more than just anomaly detection) and also provides a framework for HN to perform temporal predictions without the need for asymmetric weights (unstable and lack capacity as shown in the experiments; Figure 1B in the main paper). The learning of attractors is performed alongside the sequence modeling in a novel contrastive manner (different from how HN weights are learned; summing up patterns). Therefore, the attractors “push away” only the patterns that appear as a possibility in the same context during prediction (not all the other patterns), improving the capacity of HN. In our mathematical framework, we have used a Gaussian Mixture Model (GMM) to represent the learned possibilities and proved that maximizing the likelihood of an observation under the mixture model is equivalent to performing recall in a HN framework (Theorem 1). We also showcase the model’s ability to generate full words despite multiple valid possibilities. Being able to generate a recall of close to 80% (Figure 2D in the main paper) of the full dataset means that the model learned the input statistics because it can cover most of the data when randomly sampling from the data distribution.\n* Biological plausibility in PAM mainly refers to excluding the obvious biologically implausible learning rules, such as backpropagation or copying of weights. This allows models like PC and HN to be referred to as biologically plausible despite clear contradictions with anatomy [1, 2]. Moreover, PAM also constrains its learning and representations to sparse and binary cell assemblies, which improves the plausibility of the model. Therefore, a biologically plausible model here refers to the ability to continually learn using only local learning rules (e.g., Hebbian, anti-Hebbian, STDP) and sparse binary representations. However, we do not intend to accurately map every detail in PAM to the anatomy of the neocortex. \n* HTM relies on lateral inhibition such that when a depolarized cell fires, it inhibits the firing of other cells in the same minicolumn. The configuration of the minicolumns can be seen in every cortical column as vertical bundles of myelinated axons; each bundle has a single inhibitory double bouquet cell (i.e., interneuron) [3]. Therefore, HTM assumes interneurons perform lateral inhibition during prediction but does not learn the synapses of the interneurons. It can be very hard to accurately map the details of any learning algorithm to the anatomy since “no computer simulation can fully replicate the intricate workings of the brain in every respect” [1]. We provide additional results (rebuttal attachment figure 1C) on varying the connection density and show that PAM can still perform relatively well after removing some of the connections. We will extend this discussion and add it to the supplementary material.\n* Thank you for these great suggestions for additional theoretical analyses. We show in Theorem 1 that minimizing the prediction error by forming attractors around the observations (Hopfield Network) can be characterized as a mixture of Gaussians distribution where each mode represents a center of an attractor (single observation).\n\n**Questions**\n\n* This is the Gaussian distribution for the component $c$ (of the GMM) at mean $\\mu_c$ and covariance $\\Sigma_c$. When substituting the point $\\boldsymbol{x}$ in this function, it returns a high number if it is close to the mean of this component (attractor center). Therefore, normalizing by the sum of distances to means of all components gives a similarity measure. This similarity measure becomes a normalized score for each component, denoting how close the observation is to each of them.\n* This step ensures that the Hopfield iterations stay bounded by the model’s beliefs (predicted union of SDRs). It effectively minimizes the number of attractors during generation. If removed, the noisy observation will be pushed towards the closest observation regardless of what is predicted by the model. Thank you for the suggestion of future experiment. To our knowledge, we are the first to implement such a rule for an associative memory model.\n* We will define it in the main text. Thank you.\n\n*We hope to have addressed all of your concerns and look forward to any additional discussions*\n\n**References**\n\n[1] Salvatori, Tommaso, et al. \"Brain-inspired computational intelligence via predictive coding.\" arXiv preprint arXiv:2308.07870 (2023).\n\n[2] Chrysanthidis, Nikolaos, Florian Fiebig, and Anders Lansner. \"Introducing double bouquet cells into a modular cortical associative memory model.\" Journal of computational neuroscience 47.2 (2019): 223-230.\n\n[3] Peters, A., and C. Sethares. \"The organization of double bouquet cells in monkey striate cortex.\" Journal of neurocytology 26.12 (1997): 779-797.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J1BEeqDnGG", "reviewer": "Reviewer_T6nZ", "summary": "The paper presents a novel sequence memory architecture focused on multistability, ie: generating multiple possible futures from every present context. They propose to train this architecture in a biologically plausible way via predictive coding under the Free Energy Principle, in the online learning setting where data items are seen once instead of repeated in mini-batches. Section 2 reviews background, and Section 3 describes the Predictive Attractor Model formally.  In Section 4 the paper evaluates PAMs for their storage capacity, robustness against catastrophic forgetting, multistability, and noise robustness, comparing to symmetric and asymmetric Hopfield Networks and temporal predictive coding (tPC).", "questions": "Can the authors provide a unified presentation of the different biologically-informed learning rules/computations they propose, and how those unite back with the variational inference perspective introduced at the start of the paper?\n\nI would suggest that the authors provide some hyperlink to Table 1 (in the Supplementary material, giving notations) in the main text, or even move Table 1 into the main text.\n\nRegarding the neuroscience, why does the paper refer to a cortical minicolumn (rather than a laminar column or cortical microcircuit?) rather than a hippocampal circuit?  Sequential memory architectures, particularly those akin to Hopfield networks, are more typically considered as models of hippocampal circuitry.", "limitations": "The authors freely admit that their model relies on having a pretrained model, such as an autoencoder, to convert sensory-level inputs into sparse distributed representations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The paper attempts and achieves a high degree of biological plausibility by focusing on specific learning mechanisms, observed in the brain, for each computation that must be carried out.  To my knowledge it is the first sequence memory architecture to explicitly consider a mixture likelihood in order to consider multiple possible continuations given every context. I have a hard time evaluating the quality of the experimental results, since I have not kept up with the state of the art in sequential memory models this past year or two.", "weaknesses": "Clarity is the main aspect in which the paper lacks.  The paper uses a lot of concepts without either initially laying them out together or introducing them explicitly the first time they are needed and then referencing back to them when they are used.  A table of notations somewhere would be quite helpful.  For instance, presenting terms as \\log 1/p(x) sows a bit of confusion over whether a likelihood ratio/Radon-Nikodym derivative is the intended concept being conveyed, or just \\log 1/p(x) = -\\log p(x) giving an entropy or cross-entropy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive remarks. Yes, to our knowledge, we are the first to model multiple possibilities as a GMM in a state space model and prove that maximizing the likelihood of a query observation under the mixture model is equivalent to a Hopfield recall function (Theorem 1).\n\n**Weaknesses**\n\n* We will provide a link to the notation table in the supplementary for clarity. Due to space constraints in the main paper, we moved the derivation of the variational free energy (VFE) equation to the supplementary material (Derivation 2; appendix B1). The log term of a conditional probability in VFE refers to the log-likelihood, whereas the expected log of a marginal probability refers to the entropy of the distribution.\n\n**Questions**\n\n* In PAM, we only use Hebbian plasticity as the biologically plausible learning rule. The variational free energy equation minimizes the KL divergence between the intractable true posterior and the approximate posterior in the Bayesian inference framework by simply minimizing the negative log-likelihood (NLL) terms (i.e., latent and observation terms in equation 3); see derivation 2 in appendix B1. Since we assume Gaussian form for the latent term and a mixture of Gaussians for the observation term, minimizing the prediction errors will maximize the likelihood and, therefore, push the approximate posterior towards the true posterior. We use Equation 7 to minimize the prediction error for the latent state term, which uses local Hebbian rules to correct its predictions by modifying the synaptic weights. For the observation state, we correct the model's predictions by forming attractors around the observation (in a contrastive manner), effectively learning the parameters of the GMM. During inference, applying the Hopfield recall function iteratively maximizes the likelihood of the query observation under the learned mixture model (Theorem 1). Therefore, using Hebbian rule in equations 7 and 8, minimize the variational free energy, which minimizes the KL divergence between $q(z_t)$ and $p(z_{t} | z_{t-1}, x_{t})$ (i.e., the goal of variational inference).\n* Thank you for the suggestion. We will link to the table in the supplementary or move it to the main paper.\n* While associative memory models are typically considered models of hippocampal circuitry, spatiotemporal sequence modeling and integration can be considered a function of the neocortex [1,2]. The architecture of PAM follows the same anatomy mapping of HTM [3], in which sequence modeling is performed in layer IV of the cortical column. Many connections between layers in a single cortical column (IV ↔︎ VI) and across columns (III ↔︎ III) are considered bidirectional associative connections; therefore, it is biologically plausible for associative connections to exist in the neocortex. We will provide more discussions on mapping of PAM to the neocortex anatomy in the camera-ready submission.\n\n*We hope to have addressed all of your concerns and look forward to any additional discussions*\n\n**References**\n\n[1] Oberländer, Jette, Younes Bouhadjar, and Abigail Morrison. \"Learning and replaying spatiotemporal sequences: A replication study.\" Frontiers in integrative neuroscience 16 (2022): 974177.\n\n[2] Synaptic inhibition in the neocortex: Orchestration and computation through canonical circuits and variations on the theme", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oU4WNdB6cN", "reviewer": "Reviewer_sj6N", "summary": "The paper deals with training generative sequence models under a few constraints to make it biologically plausible. \nSpecifically, states consist of a sparse binary representation (i.e. neurons that are on/off and sequence training and generation are performed using operations that are described as \"local and synaptic\". The proposed model seems to be a modification / generalization of Hopfield networks (a form of recurrent network) such that weights are asymmetric and there is an orthogonal \"hidden\" dimension to the representation (hidden in that is does not affect the output directly) that captures prior history and, in effect, turns the model in to an HMM.\nState transitions involve a linear transition matrix, lateral inhibition (within the hidden state dimension using a max operation) and output-dependent filtering. Output is generated through sampling from a Gaussian mixture model of attractors. The method is backed with some theoretical guaranty of optimality of the observation generation.\n\nThe model is compared against others of it's kind in online and offline (teacher-forced) settings and assessed with respect to memory capacity (number of sequences and their lengths), robustness to noise, correlation between states, training speed and parameterized hidden state dimension. The proposed model outperforms in every way.", "questions": "The state transition and the output mechanisms in PAM are different. One uses attractor dynamics and the other uses a deterministic transition and a max operation. Wouldn't it be more biologically plausible to have operations use similar mechanics?\n\nIt is unclear whether the models presented are useful from a machine learning point of view. Taking the word generation as an example, the level of complexity of that task is orders of magnitude below what language generation models are capable of. Assuming this assertion is correct, what does that imply about the state of what we consider biologically plausible compared with what real brains are capable of and what unconstrained ML methods can achieve. Is there an unaddressed gap?", "limitations": "It is unclear how the methods compare agains general ML methods. It is unclear whether the assumed biolgical plausibility helps understand biological neural mechanisms.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed model seems like a natural extension of Hopfield networks to the sequence and hidden-state domains. \nThe state, training and inference all seem to adhere to the constraints that make the method \"biologically plausible\".\nThe various (synthetic and \"real\") are used well to show the model streangths. Performance is very clearly better than the models against which it is compared.\nThe presentation is very clear. The accompanying video-based presentation is superb. The supplementary material is very extensive and provides additional background, proofs and implementation details at the same level of attention to clarity and detail.", "weaknesses": "* It is not clear to me what constitutes a biologically plausible mathematical operation. Are the various models equally biologically plausible? Is there a scale of plausibility? With that in mind, I wonder how the models that appear in this study compare against state of the art sequence memorization / generative models that are unconstrained (e.g. transformers?), whether those models are better and are they provably non-plausible. It would have been interesting to read a discussion of whether biological plausibility can help us produce better models or whether (/ how) it is helpful in explaining observations from studies of real biological networks.\n\n* Clearly, both memory capacity and training efficiency are highly dependent on the level of sparcity and the number of parameters (be they weights or neurons). The comparison across models is somewhat limited in its treatment of model complexity (albeit the proposed PAM models are compared across several  hidden-dimension sizes). I'd expect that some models would match PAM performance in some respects as they are scaled up. It may be useful for the analysis to be extended to cover this aspect of the models better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the great remarks on PAM’s strengths and the clarity of the presentation, supplementary material, and video. We appreciate the positive rating!\n\n**Weaknesses**\n\n* Biological plausibility is typically referred to as imposing constraints such as local learning rules [1] (e.g., Hebbian, anti-Hebbian, etc.) and sparsity of representations. For example, backpropagation approaches and methods that copy weights/gradients are known to be highly implausible. Therefore, approaches like Predictive Coding and Associative Memory Models are considered biologically plausible despite some contradictions to biology [1,2] (e.g., synapses cannot magically turn from excitatory to inhibitory; inhibition occurs through interneurons). Methods that map their approaches to the anatomy of the brain can be considered higher on the scale of plausibility. We do not intend to accurately map every detail in PAM to the cortical anatomy; however, some aspects of PAM (e.g., lateral inhibition in minicolumns, afferent thalamocortical input to layer IV, contextual information on basal dendrites, associative connections) can be mapped to the cortical column and subcortical structures. We will provide a discussion in the supplementary material on mapping to anatomy and hierarchical processing with PAM. Unconstrained models (e.g., transformers, LSTMs, CNNs) that use backpropagation to train their weights can learn unique contexts to solve prediction tasks. However, they completely fail in tasks such as continual learning since they overwrite previous knowledge. We provide additional results (rebuttal attachment; figure 1D) comparing PAM to a 2-layer transformer on continually learning protein sequences. Backward transfer results show low performance of transformers in remembering older sequences after training on new sequences. Table 1 (in the rebuttal attachment) supplements these results by showing detailed sequence-by-sequence performance values. Transformers excel at learning the current task (diagonal values) but suffer in remembering previous tasks (lower triangle values). Approaches that use sparsity and local learning rules (i.e., PAM) can easily learn new knowledge without overwriting previous knowledge.\n* Yes, some models perform well in sequence capacity. For example, AHN with a polynomial degree of 2 and sparsity of 50% seems to outperform PAM with $N_k=4$ (Figure 1A; main paper); however, it quickly loses capacity as soon as correlation is introduced (Figure 1B; main paper). Additionally, PAM with $N_k=1$ seems to consistently perform worse than other methods (Figure 2), as expected. Moreover, PAM requires sparse representations to represent multiple possibilities, and unions of SDRs will not be useful if the representations are not unique (low overlap in bits). This failure mode is specific to PAM; however, we argue that sparsity is the biological solution to continual learning and multiple possibilities representation and that models should be using sparsity to enhance their performance.\n\n**Questions**\n\n* While both mechanisms may seem quite different, they both rely on Hebbian learning and synaptic excitation/inhibition to learn associations between patterns. State transition learns to associate patterns at time $t$ to patterns at time $t+1$, while the emission model builds attractors by associating the firing neurons of each pattern and inhibiting predicted neurons of other patterns. It is important to note that the cortical column has a distinct structure of minicolumns where axons are bundled together [3]. This may force competitive learning rules (through lateral inhibition) across the pyramidal neurons of each minicolumn. This structure is not assumed for associative memory models. In other words, the morphology of the minicolumn may play a role in defining the learning mechanics.\n* It is important to note that the word generation task presented in PAM is very different from LLMs. PAM performs regression-based prediction; we do not assume knowledge of a dictionary of tokens/characters. PAM regresses the full future representation. However, current LLMs are deterministic in nature. Therefore, they assume knowledge of a set of possible tokens and predict a probability distribution over the token IDs, from which they sample a single ID that retrieves the token from the stored dictionary. Current LLMs are trained in a contrastive manner as classification models. While this approach can work for LLMs (to a certain extent), they are not suitable for tasks that do not assume knowledge of the entire set of possible predictions (e.g., video frame predictions). As a side note, this approach to LLMs does not allow building compositional models with higher-order predictions (e.g., sentence predicting sentence) because we simply cannot tokenize all possible sentences. However, in PAM, we can predict frames, regress multiple possibilities, and build true compositional models.\n\n*We hope to have addressed all of your concerns and look forward to any additional discussions*\n\n**References**\n\n[1] Salvatori, Tommaso, et al. \"Brain-inspired computational intelligence via predictive coding.\" arXiv preprint arXiv:2308.07870 (2023).\n\n[2] Chrysanthidis, Nikolaos, Florian Fiebig, and Anders Lansner. \"Introducing double bouquet cells into a modular cortical associative memory model.\" Journal of computational neuroscience 47.2 (2019): 223-230.\n\n[3] Peters, A., and C. Sethares. \"The organization of double bouquet cells in monkey striate cortex.\" Journal of neurocytology 26.12 (1997): 779-797.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their comments and useful suggestions. We think that these comments have helped us better refine the paper.

We provide additional supplementary results to help address some of the concerns on scaling, connection sparsity and comparing to transformer architecture. We refer to these experiments in the individual rebuttals.

Meta-review: This is a tough one. 

The paper proposes a biologically plausible (somewhat) model for learning and generating sequences. It essentially expends the HTM model of Subutai Ahmad and Jeff Hawkins, by introducing some Hopfield-like mechanisms, to allow for generation.

Two reviewers leaned towards "borderline rejection", highlighting some perceived potential limitations of the approach. Three reviewers recommended acceptance.

I personally found the paper difficult to follow. some reviewers share my concern (Reviewer T6nZ) but others did not (Reviewer sj6N found it very clear), so this may be due to lack of familiarity with Bayesian terminology.

Overall I would lean towards acceptance due to the interest of the contribution and the novelty of the approach, but given the middling reviews and the difficulty of fully understanding the method I am rather uncertain.
Decision: Accept (poster)