Title: Predictive Attractor Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Sequential Memory

Predictive Models

Fixed-point Attractors

Associative Memory Models

State Space Models

Biologically plausible

Hebbian Plasticity Rules

Local Computations

Hierarchical Temporal Memory

Continual Learning

Multiple Possibilities Generation

Noise Tolerance
Abstract: Sequential memory, the ability to form and accurately recall a sequence of events or stimuli in the correct order, is a fundamental prerequisite for biological and artificial intelligence as it underpins numerous cognitive functions (e.g., language comprehension, planning, episodic memory formation, etc.) However, existing methods of sequential memory suffer from catastrophic forgetting, limited capacity, slow iterative learning procedures, low-order Markov memory, and, most importantly, the inability to represent and generate multiple valid future possibilities stemming from the same context. Inspired by biologically plausible neuroscience theories of cognition, we propose Predictive Attractor Models (PAM), a novel sequence memory architecture with desirable generative properties. PAM is a streaming model that learns a sequence in an online, continuous manner by observing each input only once. Additionally, we find that PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns, which prevents new memories from overwriting previously learned knowledge. PAM generates future predictions by sampling from a union set of predicted possibilities; this generative ability is realized through an attractor model trained alongside the predictor. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework. Other desirable traits (e.g., noise tolerance, CPU-based learning, capacity scaling) are discussed throughout the paper. Our findings suggest that PAM represents a significant step forward in the pursuit of biologically plausible and computationally efficient sequential memory models, with broad implications for cognitive science and artificial intelligence research. Illustration videos and code are available on our project page: https://ramymounir.com/publications/pam.

Predictive Attractor Models

Abstract

Sequential memory, the ability to form and accurately recall a sequence of events or stimuli in the correct order, is a fundamental prerequisite for biological and artificial intelligence as it underpins numerous cognitive functions (e.g., language comprehension, planning, episodic memory formation, etc.) However, existing methods of sequential memory suffer from catastrophic forgetting, limited capacity, slow iterative learning procedures, low-order Markov memory, and, most importantly, the inability to represent and generate multiple valid future possibilities stemming from the same context. Inspired by biologically plausible neuroscience theories of cognition, we propose Predictive Attractor Models (PAM) , a novel sequence memory architecture with desirable generative properties. PAM is a streaming model that learns a sequence in an online, continuous manner by observing each input only once . Additionally, we find that PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns, which prevents new memories from overwriting previously learned knowledge. PAM generates future predictions by sampling from a union set of predicted possibilities; this generative ability is realized through an attractor model trained alongside the predictor. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework. Other desirable traits (e.g., noise tolerance, CPU-based learning, capacity scaling) are discussed throughout the paper. Our findings suggest that PAM represents a significant step forward in the pursuit of biologically plausible and computationally efficient sequential memory models, with broad implications for cognitive science and artificial intelligence research. Illustration videos and code are available on our project page: https://ramymounir.com/publications/pam .

1 Introduction

Modeling the temporal associations between consecutive inputs in a sequence (i.e., sequential memory) enables biological agents to perform various cognitive functions, such as episodic memory formation [53, 64, 15], complex action planning [22] and translating between languages [3]. For example, playing a musical instrument requires remembering the sequence of notes in a piece of music; similarly, playing a game of chess requires simulating, planning, and executing a sequence of moves in a specific order. While the ability to form such memories of static, unrelated events has been extensively studied [56, 55, 63, 69, 66], the ability of biologically-plausible artificial networks to learn and recall temporally-dependent patterns has not been sufficiently explored in literature [62]. The task of sequential memory is considered challenging for models operating under biological constraints (i.e., local synaptic computations) for many reasons, including catastrophic forgetting, ambiguous context representations, multiple future possibilities, etc.

In addition to the biological constraint, we impose the following set of desirable characteristics as learning constraints on sequence memory models.

The learning of one sequence does not overwrite the previously learned sequences. This property is defined under the continual learning framework as avoiding catastrophic forgetting and evaluated with the Backward Transfer (BWT) metric [38].

The model should uniquely encode inputs based on their context in a sequence. Consider the sequence of letters 'EVER'; the representation of 'E' at position 1 should be different from 'E' at position 3, thus resulting in different predictions: 'V' and 'R'. Moreover, the representation of 'E' at position 3 in 'EVER' should be different from 'E' at position 3 in 'CLEVER'. Therefore, positional encoding is not a valid solution.

When presented with multiple valid, future possibilities, the model should learn to represent each possibility separately yet stochastically sample a single valid possibility. Consider the two sequences 'THAT' and 'THEY'; after seeing 'TH', the model should learn to generate either 'A' or 'E', but not an average [37] or a union of both [25].

The model should be capable of incrementally learning each transition without seeing the whole sequence or revisiting older sequence transitions that are previously learned. This property falls under online learning constraints, also called stream learning [45, 25].

The learning algorithm should be tolerant and robust to significant input noise. A model should continuously clean the noisy inputs using learned priors and beliefs, thus performing future predictions based on the noise-free observations [25].

We propose Predictive Attractor Models (PAM) , which consists of a state prediction model and a generative attractor model. The predictor in PAM is inspired by the Hierarchical Temporal Memory (HTM) [25] learning algorithm, where a group of neurons in the same cortical minicolumn share the same receptive feedforward connection from the sensory input on their proximal dendrites. The depolarization of the voltage of any neuron in a single minicolumn (i.e., on distal dendrites) primes this depolarized neuron to fire first while inhibiting all the other neurons in the same minicolumn from firing (i.e., competitive learning). The choice of which neurons fire within the same minicolumn is based on the previously active neurons and their trainable synaptic weights to the depolarized neurons, which gives rise to a unique context representation for every input [20]. The sparsity of representations (discussed later in Section 3.2) allows for multiple possible predictions to be represented as a union of individual cell assemblies. The Attractor Model learns to disentangle possibilities by strengthening the synaptic weights between active neurons of input representations and inhibiting the other predicted possibilities from firing, effectively forming fixed point attractors during online learning. During recall, the model uses these learned conditional attractors to sample one of the valid predicted possibilities or uses the attractors as prior beliefs for removing noise from sensory observations.

PAM satisfies the above-listed constraints for a sequential memory model, whereas the current state-of-the-art models fail in all or many of the constraints, as shown in the experiments. Our contributions can be summarized as follows: (1) Present the novel PAM learning algorithm that can explicitly represent context in memory without backpropagation, avoid catastrophic forgetting, and perform stochastic generation of multiple future possibilities. (2) Perform extensive evaluation of PAM on multiple tasks (e.g., sequence capacity, sequence generation, catastrophic forgetting, noise robustness, etc.) and different data types (e.g., protein sequences, text, vision). (3) Formulate PAM and its learning rules as a State Space Model grounded in variational inference and the Free Energy Principle [17].

2 Background and Related Works

2.1 Predictive Coding

Predictive coding proposes a framework for the hierarchical processing of information. It was initially formulated as a time series compression algorithm to create a more efficient coding system [16, 46]. A few decades later, PC was used to model visual processing in the Retina [59, 27] as an inference model. In the seminal work of Rao and Ballard [52], PC was reformulated as a general computational model of the cortex. The main intuition is that the brain continuously predicts all perceptual inputs, resulting in a quantity of prediction error, which can be minimized by adjusting its neural activities and synaptic strengths. In-depth variational free energy derivation is provided in Appendix A.1.

PC defines two subgroups of neurons: value z and error. Each neuron contains a value node sending its prediction to the lower level ˆ z l = f l +1 ( z l +1 ) through learnable function f , and error node propagating its computed error to the higher level. The total prediction error is computed as ϵ = ∑ l || ( z l -ˆ z l ) || 2 2 , which is minimized by first running the network value nodes to equilibrium through optimizing the value nodes { z l } L l =0 . At equilibrium, the value nodes are fixed, and inferential optimization is performed by optimizing the functions { f l } L l =1 . Both optimizations aim to minimize the same prediction error over all neurons. This propagation of error to equilibrium is shown to be equivalent to backpropagation but using only local computations [66]. The PC formulation has shown success in training on static and i.i.d data [66, 69, 57, 23]. More recently [62], Temporal Predictive Coding (tPC) has also shown some success in sequential memory tasks by modifying error formulation to account for a one-step synaptic delay through interneurons, thus modeling temporal associations between sequence inputs. In the experiments, we compare our model to tPC and its 2-layer variant [62]. A few PC-based models [47, 48] have shown promise for class-incremental continual learning tasks, other PC-inspired models, such as [39, 23, 24], have diverged from the biologically plausible constraints by training through backpropagation through time.

2.2 Fixed-Point Attractor Dynamics

Attractor dynamics refer to mathematical models that describe the behavior of dynamical systems. In our review, we focus on fixed point attractors, specifically Hopfield Networks [26] (HN), which is an instance of associative memory models [32, 29, 30]. Consider, an ordered sequence of T +1 consecutive patterns x = [ x t ] T +1 t =1 , where x t ∈ {-1 , 1 } N . We refer to the Universal Hopfield Networks (UHN) framework [41] to describe all variants of HN architecture using a similarity (sim) function and a separation (sep) function, as shown in equation 1. This family of models can be viewed as a memory recall function, where a query ξ (i.e., noisy or incomplete pattern) is compared to the existing patterns to compute similarity scores using the 'sim' function. These scores are then used to weight the projection patterns after applying a 'sep' function to increase the separation between similarity scores. The classical HN uses symmetric weights to store the patterns; therefore, it cannot be used to model temporal associations in a sequence. The asymmetric Hopfield Network (AHN) [58] uses asymmetric weights to recall the next pattern in the sequence for a given query ξ .

When a dot product 'sim' function and identity 'sep' function are used, we get the classical HN [26] and AHN [58]. A few variants have been proposed to increase the capacity of the model. Recently, [12] has extended AHN by using a polynomial (with degree d ) or a softmax function (with temperature β ) as the 'sep' function. HN can also be applied to continuous dense patterns [34, 51, 12], and extended to sparse modern Hopfield models [28, 67].

2.3 Predictive Learning

Predictive learning takes a more general form of minimizing the prediction error between two views of the same input to improve representations. Many backpropagation-based approaches to predictive learning have been proposed; most recently, JEPA [37] and its variants [8, 10, 9, 70], learn useful dense representations from images and videos using the predictive objective. Other models, such as [11, 21, 13] - to list a few, use a similar methodology of predicting distorted versions of the same input to learn good feature representations. Prediction-based approaches have also been used to segment videos into events temporally [2, 42] and spatially [44, 43, 1]. More recently, STREAMER [45] used a predictive learning approach to achieve hierarchical segmentation and representation learning from streaming egocentric videos, where a high prediction error is used as an event boundary. While these biologically implausible approaches show impressive results on their respective tasks, they still suffer from deep learning known limitations, such as catastrophic forgetting and the inability to generate multiple possibilities in regression-based predictive tasks. Hierarchical Temporal Memory (HTM) [25] is a predictive approach that is heavily inspired by neurobiology. HTM relies on lateral inhibition between neurons of the same minicolumn and sparsity of input representations (i.e., SDR) to learn temporal context and associations using only local Hebbian rules.

HTM can be applied to online tasks, such as anomaly detection [6] and object recognition [36], but it is currently incapable of generating future predictions in auto-regressive prediction tasks.

3 Predictive Attractor Models

3.1 State Space Model (SSM) formulation

Content

Context

Multiple Possibilities Prior

Content

Context

Single Possibility Posterior

Single Possibility Observation

Multiple

Possibilities

Update Prior

Belief

Attractor 1

Posterior

Prior

PAM can be represented as a dynamical system with its structure depicted by a Bayesian probabilistic graphical [35, 31] model, more specifically, a State Space Model, where we can perform Bayesian inference on the latent variables and derive learning rules using Variational Inference. Formally, we define a state space model as a tuple ( Z , X , f, g ) , where Z is the latent state space, X is the observation space, and f and g are the transition and emission functions respectively (similar to HMM[50]). We consider a Gaussian form with white Gaussian noise covariance Σ z for the latent states. However, we assume a latent state z can generate multiple valid possibilities. Therefore, we model the conditional probability p ( x t | z t ) as a Multivariate Gaussian Mixture Model (GMM), where each mode is considered a possibility or a fixed-point attractor in an associative memory model. The GMMhas C components with means g c ( z t ) , covariances Σ c and component weights w c . The SSM dynamics can be formally represented with the following equations:

where z t ∈ Z and x t ∈ X . From the Bayesian inference viewpoint, we are interested in the posterior p ( z t | x t , z t -1 ) . Since the functions f and g are non-linear, the computation of this posterior is intractable (unlike an LG-SSM, such as Kalman Filter [65]). Therefore, we utilize variational inference to approximate the posterior by assuming the surrogate posterior q ( z ) has a Gaussian form, and minimize the Variational Free Energy (VFE) [18]. The minimization of VFE (in equation 3) minimizes the KL-divergence between the approximate posterior q ( z ) and the true posterior p ( z t | x t , z t -1 ) . Derivation 2 of Variational Free Energy is provided in appendix A.1

where E q ≡ E z t ∼ q ( z t ) and H q is the Entropy of the approximate posterior q ( z ) . The assumption of Gaussian forms for the latent and observable states can further simplify the negative log-likelihood

terms (i.e., Latent State Accuracy and Observation Accuracy) to prediction errors. This learning objective encourages the approximate posterior q ( z ) to assign a high probability to states that explain the observations well and follow the latent dynamics of the system. We minimize the prediction errors (i.e., learn the transition and emission functions) through Hebbian rules as shown in equations 7 and 8.

Theorem 1 Assume the likelihood p ( x t | z t ) in eqn 3 represents multiple possibilities using a Gaussian Mixture Model (GMM) conditioned on the latent state z t , as shown in eqn 2. The maximization of such log-likelihood function (i.e., ∂ ∂ x log p ( x t | z t ) ) w.r.t a query observation state x is equivalent to the Hopfield recall function (i.e., eqn 1) with the means of the GMM representing the attractors of a Hopfield model. Formally, the weighted average of the GMM means (i.e., { µ c } C c =1 ), with the weights being a similarity measure, maximizes the log-likelihood of x under the mixture model.

Proof: See derivation 3 in appendix A.2 for the full proof.

3.2 Preliminaries and Notations

Sparse Distributed Representation (SDR) Inspired by the sparse coding principles observed in the brain, SDRs encode information using a small set of active neurons in high dimensional binary representation. We adopt SDRs as a more biologically plausible cell assembly representation [30]. SDRs have desirable characteristics, such as a low chance of false positives and collisions between multiple SDRs and high representational capacity [5] (More on SDRs in Appendix G). An SDR is parameterized by the total number of neurons N and the number of active neurons W . The ratio S = W/N denotes the SDR sparsity. A 1-dimensional SDR x can be indexed as x i ∈ { 0 , 1 } , whereas a 2-dimensional SDR z can be indexed as z ij ∈ { 0 , 1 } . To identify the active neurons, we define the function I : { 0 , 1 } N ↦→ N W to represent the indices of the active neurons in an SDR x as I ( x ) = { i | x i = 1 } .

Context as Orthogonal Dimension We transform the high-order Markov dependencies between observation states into a first-order Markov chain of latent states by storing context information in those latent states. The latent states SDRs, z ∈ { 0 , 1 } N c × N k , are represented with two orthogonal dimensions, where content information about the input is stored in one dimension with size N c , while context related information is stored in an orthogonal dimension with size N k . Therefore, the projection of the latent state z on the first dimension (i.e., ↓ z ) removes all context information from the state. In contrast, adding context information to an observation state x expands the dimensionality of the state (i.e., ↑ x ) such that context can be encoded without affecting its content. Competitive learning is enforced on the context dimension through lateral inhibition, effectively minimizing the collisions between contexts of multiple SDRs. We define a projection operator ↓ : { 0 , 1 } N c × N k ↦→{ 0 , 1 } N c . Additionally, we define a projection operator ↑ : { 0 , 1 } N c ↦→{ 0 , 1 } N c × N k for 1-dimensional SDRs (i.e., x ) as shown in equation 5. A detailed list of notations is provided in the supplementary material (Table 1).

3.3 Sequence Learning

Given a sequence of T +1 SDR patterns [ x t ] T +1 t =1 , where x t ∈ { 0 , 1 } N c , the sequence can be learned by modeling the context-dependent transitions between consecutive inputs within the sequence. We define learnable weight parameters for transition and emission functions, A ∈ R N c N k × N c N k , B ∈ R N c × N c respectively. A single latent state transition is defined as ˆ z t = δ ( A · z t -1 ) = δ ( a t ) , where δ is a threshold function transforming the logits a t to the predicted SDR state ˆ z t . The full sequence learning algorithm is provided in algorithm 1.

Algorithm 1 : Sequence Learning . Given a sequence x of T+1 patterns, this algorithm learns the Transition and Emission synaptic weights ( A and B ). Fixed start context m ( a 0 ) is initialized for all learned sequences.

Algorithm 2 : Sequence Generation . Given a noisy sequence (i.e., online), or the first input in a sequence (i.e., offline). The model uses the learned functions A and B to generate the full sequence. ∼ denotes sampled from (eqn 9).

Context Encoding through Competitive Learning Every observation x t contains only content information about the input; we embed the observation with context by expanding the state with an orthogonal dimension (i.e., ↑ x t ), which activates all neurons in the minicolumns at the indices I ( x t ) . Then, for each active minicolumn, the neuron in a predictive state (i.e., higher than the prediction threshold) fires and inhibits all the other neurons in the same minicolumn from firing (i.e., lateral inhibition), as shown in Equation 6. If none - or more than one - of the neurons are in a predictive state, random Gaussian noise ( ϵ ) acts as a tiebreaker to select a context neuron. We do not allow multiple neurons to fire in the same minicolumn, which is different from HTM [25], where multiple cells can fire in any minicolumn (e.g., bursting).

State Transition Learning The transition between latent states is learned through local computations with Hebbian-based rules. We modify the synaptic weights A to model the transition between pre-synaptic neurons z t -1 and post-synaptic neurons z t . Only the synapses with active pre-synaptic neurons are modified. The weights operate on context-dependant latent states (i.e., z t -1 → z t ); thus, the learning of one transition does not overwrite previously learned transition of different contexts, regardless of the input contents (i.e., x t -1 ). We use the learning constant coefficients η + A and η -A to independently control learning and forgetting behavior, as shown in equation 7. A lower η -A encourages learning multiple possibilities by slowing down the forgetting behavior.

Contrastive Attractors Formation The attractors are formed in an online manner by contrasting the input observation x t to the predicted union set of possibilities ↓ ˆ z t . The goal is to learn excitatory synapses between active neurons of x t , and bidirectional inhibitory synapses between x t and the union set of predicted possibilities excluding the x t possibility (i.e., ( ↓ ˆ z t ) \ x t ) ), as shown in equation 8.

3.4 Sequence Generation

After learning one or multiple sequences using algorithm 1, we use algorithm 2 to generate sequences. First, we define two generative tasks: online and offline. In online sequence generation, a noisy

Sample

Union of

Possibilities

Attractor

Basins

Lateral Inhibition

Attractor

Basins

Lateral Inhibition

Offline Generation

Online Generation

Noisy Observation

version of the sequence is provided as input, and the model is expected to generate the original learned sequence. In offline sequence generation, the model is only provided with the first input, and it is expected to generate the entire sequence auto-regressively. For cases with equally valid future predictions (e.g., 'a' and 'e' after 'TH' in 'THAT' and 'THEY'), the model is expected to stochastically generate either one of the possibilities (i.e., 'THAT' or 'THEY'). The online generation task is a more challenging extension of the online recall task in [62], where the noise-free inputs are provided, and the model only makes a 1-step prediction into the future. During offline sequence generation, the model randomly samples from the union set of predictions ↓ ˆ z a single SDR with W active neurons (equation 9) to initialize the iterative attractor procedure. π denotes a random permutation function. This random permutation function allows the model to randomly generate a different sequence with every generation.

4 Experiments

4.1 Evaluation and Metrics

Metrics To evaluate the similarity of two SDRs, we use the Jaccard Index (i.e., IoU) 1 , which focuses on the active bits in sparse binary representations. Since the sparsity S of the representations can change across experiments and methods, we normalize the IoU by the expected IoU (Derived in Theorem 2 in Appendix A.3) of two random SDRs at their specified sparsities. The normalized IoU is computed as IoU -E [ IoU ] 1 -E [ IoU ] . We use the Backward Transfer [38] metric in evaluating catastrophic forgetting. Mean Squared Error (MSE) is used to compare images for vision datasets.

Datasets We perform evaluations on synthetic and real datasets. The synthetic datasets allow us to manually control variables (e.g., sequence length, correlation, noise, input size) to better understand the models' behavior across various settings. Additionally, we evaluate on real datasets of various types (e.g., protein sequences, text, vision) to benchmark PAM's performance relative to other models on more challenging and real sequences. For all vision experiments, we use an SDR autoencoder to learn a mapping between images and SDRs (Details on the autoencoder are provided in Appendix C). We run all experiments for 10 different seeds and report the mean and standard deviation in all the figures. More experimental details and results are provided in Appendices C and E.

A

B

C

D

PAM     = 8

tPC L=1

tPC L=2

AHN d=1

AHN d=2

Offline

Online

Memories

Input

Hidden

E

F

4.2 Results

We align our evaluation tasks with the desired characteristics of a biologically plausible sequence memory model, as listed in the introduction. We show that PAM outperforms current predictive coding and associative memory SoTA approaches on all tasks. Most importantly, PAM is capable of long context encoding, multiple possibilities generation, and learning continually and efficiently while avoiding catastrophic forgetting. These tasks pose numerous significant challenges to other methods.

Offline Sequence Capacity We evaluate the models' capacity to learn long sequences by varying the input size N c , model parameters (e.g., N k ), and sequence correlation. The correlation is increased by reducing the number of unique patterns (i.e., vocab) used to create a sequence of length T . Correlation is computed as 1 . 0 -vocab T . In Figure 3 A , we vary the input size N c and ablate the models to find the maximum sequence length to be encoded and retrieved, in an offline manner, with a Normalized IoU higher than 0.9. We set the number of active bits W to be 5 unless otherwise specified. Results show that Hopfield Networks (HN) fail to learn with sparse representations; therefore, we use W of 0 . 5 N c only for HN and normalize the IoU metric accordingly. PAM's capacity significantly increases with context neurons N k , as expected. HN's capacity also increases with the polynomial degree d of its separation function; however, as shown in Figure 3 B , the capacity sharply drops as correlation increases. PAM retains its capacity with increasing correlation, reflecting its ability to encode context in long sequences (i.e., high-order Markov memory). This context encoding property is also demonstrated in the qualitative CIFAR [33] results in Figure 3 E and F , where a short sequence of images with high correlation is used. The model must uniquely encode the context to correctly predict at every step in the sequence. While PAM correctly predicts the full context, single-layer tPC learns to indefinitely alternate between the patterns, while two-layered tPC attempts to average its predictions. AHN shows similar low performance and failure mode as in [62].

Catastrophic Forgetting To assess the model's performance in a continual learning setup, we sequentially train each model on multiple sequences and compute the Backward Transfer (BWT) [38]

tPC L=1

tPC L=2

AHN d=1

AHN d=2

PAM     = 8

time

time

Memories

Then

Memories

E

B

A

C

D

F

女

h

2222

2

2

hh

metric by reporting the average normalized IoU on previously learned sequences after learning a new one. In Figure 4 A , we report BWT for 50 synthetically generated sequences with varying correlations. AHNcan avoid catastrophic forgetting when the patterns are not correlated, whereas tPC fails to retain previous knowledge regardless of the correlation value. PAM, with high enough context N k , does not overwrite or forget previously learned sequences after learning new ones but performs poorly when N k = 1 , as expected. We repeat the experiment on more challenging sequences from ProteinNet [7], which contains highly correlated ( > 0 . 9 ) and long sequences (details in appendix). The results in Figure 4 B show a similar trend with PAM requiring more context neurons N k to handle the more challenging data. Qualitative results on moving-MNIST [60] in Figure 4 F further demonstrate the catastrophic forgetting challenge where the learning of the second sequence overwrites the learned sequence. PAM successfully retrieves the previously learned sequence while the other models fail.

Multiple Possibilities Generation In addition to accurately encoding input contexts, PAM is designed to represent multiple valid possibilities and sample a single possibility. We perform evaluation on a dataset of four-letter English words (details in appendix), which includes many possible future completions (e.g., 'th[is]', 'th[at]', 'th[em]', etc.) We train PAM on the list of letters sequentially (i.e., one word at a time); the other methods are trained in a simpler batched setup as in [62] because they suffer from catastrophic forgetting. This puts PAM at a disadvantage, but as shown in Figure 4 C , PAM still significantly outperforms the other methods in accurately generating valid words (high average IoU) in an offline manner. Both tPC and AHN fail to generate meaningful words when trained on sequences with multiple future possibilities. Figure 4 D further demonstrates the stochastic generative property of PAM. We show PAM's ability to recall more of the dataset as it repeats the generation process, whereas PC and AHN fail in the dataset recall task.

Online Noise Robustness The online generation ability of PAM shown in Figure 2 allows the model to use the learned attractors to clean the noisy observations before using them as inputs to the predictor. This step allows the model to use its prior belief about future observations to modify the noisy inputs. In Figure 3 C , we evaluate the models' performances by changing a percentage of the active bits during online generation. PAM is able to completely replace the noisy input with its prior belief if it does not exist in the predicted set of possibilities ˆ z . In contrast, the other methods use the noisy inputs, thus hindering their performances. We provide qualitative results on CLEVRER [68]

in Figure 4 E ; PAM retrieves the original sequence despite getting noisy inputs (40% noise), and outperforms the other models. Interestingly, tPC performs reasonably well on this task despite the added noise.

Efficiency We report, in Figure 3 D , the time each model requires to learn and recall a sequence. For this study, we use input size N c = 100 and vary the sequence length. PAM operates entirely on CPU . The results show that a single-layer tPC model requires more time than all PAM variants ( N k ≤ 24 ). Additionally, a two-layered tPC requires two to three orders of magnitude more time than PAM or single-layered tPC, significantly limiting its scalability and practicality when applied to real data with long sequences.

5 Limitations

PAM requires that its representations be sparse and binary (i.e., SDRs) in order to represent multiple possibilities as a union of SDRs with minimal overlap. Therefore, PAM cannot be directly applied to images in the input space like Dense Hopfield Networks. We argue that the neocortex encodes sensory input into SDRs for processing and does not operate directly on the input in its dense representation. Methods that operate directly on dense representations (e.g., images) are arguably less biologically plausible as the neocortex uses sparse binary representations (i.e., cell assemblies with a small fraction of firing neurons) to store and manipulate information. This paper focuses on learning multiple sequences without catastrophic forgetting and stochastically generating multiple possibilities efficiently, and we assume the sequence is provided in SDR format. Additionally, methods that operate on the input directly face challenges when the input is naturally sparse (see Figure 3 A ). Therefore, it is useful to encode the input into a representation with fixed sparsity before applying sequential memory learning algorithms. In future work, we plan to investigate how to encode high dimensional complex inputs (e.g., images) in a compositional part-whole structure of SDRs where we can apply PAM at different levels of abstraction.

6 Conclusion

We proposed PAM , a biologically plausible generative model inspired by neuroscience findings and theories of cognition. We demonstrate that PAM is capable of encoding unique contexts with tremendous scalable capacity that is not affected by sequence correlation or noise. PAM is a generative model; it can represent multiple possibilities as a union of SDRs (a property of sparsity) and sample single possibilities, thus predicting multiple steps in the future despite multiple possible continuations. We also show that PAM does not suffer from catastrophic forgetting as it learns multiple sequences. PAM is trained using local computations through Hebbian rules and runs efficiently on CPUs. Future directions include hierarchical sensory processing and higher-order sparse predictive models.

References

Sathyanarayanan Aakur and Sudeep Sarkar. Action localization through continual predictive learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 , pages 300-317. Springer, 2020.

Sathyanarayanan N Aakur and Sudeep Sarkar. A perceptual prediction framework for self supervised event segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1197-1206, 2019.

Valentin Afraimovich, Xue Gong, and Mikhail Rabinovich. Sequential memory: Binding dynamics. Chaos: An Interdisciplinary Journal of Nonlinear Science , 25(10), 2015.

Subutai Ahmad and Jeff Hawkins. Properties of sparse distributed representations and their application to hierarchical temporal memory. arXiv preprint arXiv:1503.07469 , 2015.

Subutai Ahmad and Jeff Hawkins. How do neurons operate on sparse distributed representations? a mathematical theory of sparsity, neurons and active dendrites. arXiv preprint arXiv:1601.00720 , 10, 2016.

Subutai Ahmad, Alexander Lavin, Scott Purdy, and Zuha Agha. Unsupervised real-time anomaly detection for streaming data. Neurocomputing , 262:134-147, 2017.

Mohammed AlQuraishi. Proteinnet: a standardized data set for machine learning of protein structure. BMC bioinformatics , 20:1-10, 2019.

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15619-15629, 2023.

Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471 , 2024.

Adrien Bardes, Jean Ponce, and Yann LeCun. Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features. arXiv preprint arXiv:2307.12698 , 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Hamza Chaudhry, Jacob Zavatone-Veth, Dmitry Krotov, and Cengiz Pehlevan. Long sequence hopfield memory. Advances in Neural Information Processing Systems , 36, 2024.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

Nikolaos Chrysanthidis, Florian Fiebig, and Anders Lansner. Introducing double bouquet cells into a modular cortical associative memory model. Journal of computational neuroscience , 47(2):223-230, 2019.

Martin A Conway. Episodic memories. Neuropsychologia , 47(11):2305-2313, 2009.

Peter Elias. Predictive coding-i. IRE transactions on information theory , 1(1):16-24, 1955.

Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience , 11(2):127138, 2010.

Karl Friston, Jérémie Mattout, Nelson Trujillo-Barreto, John Ashburner, and Will Penny. Variational free energy and the laplace approximation. Neuroimage , 34(1):220-234, 2007.

Karl J Friston, N Trujillo-Barreto, and Jean Daunizeau. Dem: a variational treatment of dynamic systems. Neuroimage , 41(3):849-885, 2008.

Dileep George, Rajeev V Rikhye, Nishad Gothoskar, J Swaroop Guntupalli, Antoine Dedieu, and Miguel Lázaro-Gredilla. Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps. Nature communications , 12(1):2392, 2021.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Patrick Haggard. Planning of action sequences. Acta Psychologica , 99(2):201-215, 1998.

Kuan Han, Haiguang Wen, Yizhen Zhang, Di Fu, Eugenio Culurciello, and Zhongming Liu. Deep predictive coding network with local recurrent processing for object recognition. Advances in neural information processing systems , 31, 2018.

Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0-0, 2019.

Jeff Hawkins and Subutai Ahmad. Why neurons have thousands of synapses, a theory of sequence memory in neocortex. Frontiers in neural circuits , 10:174222, 2016.

John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences , 79(8):2554-2558, 1982.

Toshihiko Hosoya, Stephen A Baccus, and Markus Meister. Dynamic predictive coding by the retina. Nature , 436(7047):71-77, 2005.

Jerry Yao-Chieh Hu, Donglin Yang, Dennis Wu, Chenwei Xu, Bo-Yu Chen, and Han Liu. On sparse modern hopfield model. Advances in Neural Information Processing Systems , 36, 2024.

Hiroshi Kage. Implementing associative memories by echo state network for the applications of natural language processing. Machine Learning with Applications , 11:100449, 2023.

Pentti Kanerva. Sparse distributed memory . MIT press, 1988.

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

Bart Kosko. Bidirectional associative memories. IEEE Transactions on Systems, man, and Cybernetics , 18(1):49-60, 1988.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems , 29, 2016.

Steffen L Lauritzen. Graphical models , volume 17. Clarendon Press, 1996.

Niels Leadholm, Marcus Lewis, and Subutai Ahmad. Grid cell path integration for movement-based visual object recognition. arXiv preprint arXiv:2102.09076 , 2021.

Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. 2022.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08104 , 2016.

Henry Markram, Maria Toledo-Rodriguez, Yun Wang, Anirudh Gupta, Gilad Silberberg, and Caizhi Wu. Interneurons of the neocortical inhibitory system. Nature reviews neuroscience , 5(10):793-807, 2004.

Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning , pages 15561-15583. PMLR, 2022.

Ramy Mounir, Sathyanarayanan Aakur, and Sudeep Sarkar. Self-supervised temporal event segmentation inspired by cognitive theories. In Advanced Methods and Deep Learning in Computer Vision , pages 405-448. Elsevier, 2022.

Ramy Mounir, Roman Gula, Jörn Theuerkauf, and Sudeep Sarkar. Spatio-temporal event segmentation for wildlife extended videos. In International Conference on Computer Vision and Image Processing , pages 48-59. Springer, 2021.

Ramy Mounir, Ahmed Shahabaz, Roman Gula, Jörn Theuerkauf, and Sudeep Sarkar. Towards automated ethogramming: Cognitively-inspired event segmentation for streaming wildlife video monitoring. International journal of computer vision , 131(9):2267-2297, 2023.

Ramy Mounir, Sujal Vijayaraghavan, and Sudeep Sarkar. Streamer: Streaming representation learning and event segmentation in a hierarchical manner. Advances in Neural Information Processing Systems , 36, 2024.

J O'Neal. Entropy coding in speech and television differential pcm systems (corresp.). IEEE Transactions on Information Theory , 17(6):758-761, 1971.

Alex Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Lifelong neural predictive coding: Learning cumulatively online without forgetting. Advances in Neural Information Processing Systems , 35:58675881, 2022.

Alexander Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Continual learning of recurrent neural networks by locally aligning distributed representations. IEEE transactions on neural networks and learning systems , 31(10):4267-4278, 2020.

A Peters and C Sethares. The organization of double bouquet cells in monkey striate cortex. Journal of neurocytology , 26(12):779-797, 1997.

Lawrence Rabiner and Biinghwang Juang. An introduction to hidden markov models. ieee assp magazine , 3(1):4-16, 1986.

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, et al. Hopfield networks is all you need. In International Conference on Learning Representations , 2020.

Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience , 2(1):79-87, 1999.

Edmund T Rolls. A computational theory of episodic memory formation in the hippocampus. Behavioural brain research , 215(2):180-196, 2010.

Tommaso Salvatori, Ankur Mali, Christopher L Buckley, Thomas Lukasiewicz, Rajesh PN Rao, Karl Friston, and Alexander Ororbia. Brain-inspired computational intelligence via predictive coding. arXiv preprint arXiv:2308.07870 , 2023.

Tommaso Salvatori, Luca Pinchetti, Beren Millidge, Yuhang Song, Tianyi Bao, Rafal Bogacz, and Thomas Lukasiewicz. Learning on arbitrary graph topologies via predictive coding. Advances in neural information processing systems , 35:38232-38244, 2022.

Tommaso Salvatori, Yuhang Song, Yujian Hong, Lei Sha, Simon Frieder, Zhenghua Xu, Rafal Bogacz, and Thomas Lukasiewicz. Associative memories via predictive coding. Advances in Neural Information Processing Systems , 34:3874-3886, 2021.

Tommaso Salvatori, Yuhang Song, Beren Millidge, Zhenghua Xu, Lei Sha, Cornelius Emde, Rafal Bogacz, and Thomas Lukasiewicz. Incremental predictive coding: A parallel and fully automatic learning algorithm. arXiv preprint arXiv:2212.00720 , 2022.

Haim Sompolinsky and Ido Kanter. Temporal association in asymmetric neural networks. Physical review letters , 57(22):2861, 1986.

Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: a fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences , 216(1205):427-459, 1982.

Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In International conference on machine learning , pages 843-852. PMLR, 2015.

Dan D Stettler, Aniruddha Das, Jean Bennett, and Charles D Gilbert. Lateral connectivity and contextual interactions in macaque primary visual cortex. Neuron , 36(4):739-750, 2002.

Mufeng Tang, Helen Barron, and Rafal Bogacz. Sequential memory with temporal predictive coding. Advances in Neural Information Processing Systems , 36, 2024.

Mufeng Tang, Tommaso Salvatori, Beren Millidge, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Recurrent predictive coding models for associative memory employing covariance learning. PLoS computational biology , 19(4):e1010719, 2023.

Endel Tulving et al. Episodic and semantic memory. Organization of memory , 1(381-403):1, 1972.

Greg Welch, Gary Bishop, et al. An introduction to the kalman filter. 1995.

James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation , 29(5):1229-1262, 2017.

Dennis Wu, Jerry Yao-Chieh Hu, Weijian Li, Bo-Yu Chen, and Han Liu. Stanhop: Sparse tandem hopfield model for memory-enhanced time series prediction. arXiv preprint arXiv:2312.17346 , 2023.

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations , 2019.

Jinsoo Yoo and Frank Wood. Bayespcn: A continually learnable predictive coding associative memory. Advances in Neural Information Processing Systems , 35:29903-29914, 2022.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning , pages 12310-12320. PMLR, 2021.

Supplementary Material

A Theorems and Derivations

A.1 Variational Free Energy

Predictive Coding Consider a hierarchical generative model with hidden states { z } L l =0 , where l ≤ L denotes the level in the hierarchy. The conditional probability p ( z l | z l +1 ) is assumed to be a multivariate Gaussian distribution with its mean calculated as a function f l +1 of the higher-level hidden representation z l +1 and covariance Σ l as shown in equation 10.

The goal is to calculate the posterior of hidden states given an observation x , (i.e., P ( { z } L l =0 | x ) ). Since the prediction function f contains a non-linear activation, we cannot analytically compute the posterior and we have to approximate it with a surrogate posterior (i.e., q ( { z } L l =0 ) by maximizing the Evidence Lower Bound (ELBO). We apply the mean field approximation to factorize this joint posterior probability into conditionally independent posteriors { q ( z l ) } L l =0 } , and apply the Laplace approximation to use Gaussian forms for the approximate distribution [19, 18, 54]. Through these approximations, we can maximize the ELBO, or equivalently minimize the Variational Free Energy, in equation 11.

The variational free energy can be reduced to minimizing the negative log-likelihood (Accuracy term), which is simply the prediction error when the likelihood is assumed to take a Gaussian Form. Therefore, minimizing the prediction error reduces to the sum of the squared prediction error of every neuron.

Derivation 1 Variational Free Energy derivation for the predictive coding objective function in equation 11. We approximate the true posterior p ( z | x ) with a surrogate posterior q ( z ) . The objective is to minimize the reverse KL divergence D KL ( q ( z ) || p ( z | x )) .

To minimize the KL divergence, we can minimize the Variational Free energy instead because the Evidence term ( log p ( x ) ) is a constant negative term. The Variational Free Energy can be further simplified as follows:

We arrive at equation 11. Minimizing the error term (i.e., negative log-likelihood) is equivalent to minimizing the Sum of Squared Error (SSE) when a Gaussian form is assumed for the likelihood p ( x | z ) .

Derivation 2 Variational Free Energy derivation for a State Space Model (SSM) in equation 3. Latent states are denoted with z , whereas observations are denoted with x . We assume non-linear transition and emission function (i.e., f and g ); therefore a variational approximation is needed to approximate the true posterior p ( z t | z t -1 , x t ) with a surrogate posterior q ( z t ) . As in derivation 1, the goal is to minimize the divergence between the true posterior and the approximate posterior (i.e., D KL ( q ( z t ) || p ( z t | z t -1 , x t )) ). Note that, for notation brevity, E q ≡ E z ∼ q ( z t ) .

We can minimize the Variational Free Energy term, which reduces to the log-likelihood of two prediction error terms and the negative entropy of the approximate posterior q ( z t ) as shown below.

Minimizing the Variational Free Energy above forces q ( z t ) to better approximate the true posterior.

A.2 Gaussian Mixture Model and Hopfield Recall

Derivation 3 We derive theorem 1 which states that the maximization of the log-likelihood of p ( x t | z t ) in the form of a Gaussian Mixture Model is equivalent to the recall function in Hopfield networks (i.e., eqn 1), where the means of the GMM (i.e., { µ c } C c =1 ) represents the attractors of a Hopfield model. To maximize the log-likelihood, we compute its partial derivative with respect to x .

By setting the partial derivative of the log-likelihood to 0 , we can estimate the value of x , which maximizes the function log p ( x | z ) .

Finally, we can rearrange the equation in terms of x and show that it is equivalent to the Hopfield recall function where the recall value x equals a weighted average of the attractors (i.e., means of GMM { µ c } C c =1 ), with the weights being a similarity score function.

A.3 Expected IoU of Random SDRs

Theorem 2 Consider two SDRs with sparsity defined as random variables p ∼ U (0 , 1) and q ∼ U (0 , 1) , the expected Jaccard Index (i.e., IoU) of the two random SDRs follows:

Proof: Given the sparsity random variables of both SDRs (i.e., p and q ) and the SDR size n , the number of active bits at the same location in both SDRs is equal to the joint probability of both SDRs being active multiplied by the SDR size (i.e., npq ). The union of both SDRs is the total number of active bits minus the active bits in both SDRs, which is equal to np + nq -npq . Therefore, the expected intersection over union is npq np + nq -npq = pq p + q -pq

Empirical Validation: We perform empirical validation of the above theorem as shown in figure 5. The sparsity of the first SDR is fixed at 0.1 and 0.5. We vary the sparsity of the second SDR between 0.0 and 1.0 in steps of 0.1 and calculate the average IoU over a population of 1000 pairs of SDR for every setting. Empirical results agree with the derived formulation in theorem 2.

B Notations

The notations used in our paper are summarized in Table 1.

C Datasets

Synthetic For synthetic experiments, we generate SDRs with the specified size N c and uniformly initialized active bits W to match the required sparsity S . In many of the experiments, N c is set to 100 with 5 active bits unless otherwise specified. For Hopfield experiments, we set the sparsity to 50% to improve its performance.

Protein Sequences We use the dataset ProteinNet 7 [7] to extract protein sequences. Each sequence consists of a chain of Amino Acids. In the dataset, there are only 20 different types of Amino Acids (i.e., vocabulary), creating long protein sequences with hundreds of Amino Acids. The dataset is reported in the fasta format, where each Amino Acid is represented with a single-letter code. We create a dictionary mapping from the Amino Acid types to random SDRs with N c = 100 and W = 5 to train the models. When choosing the sequences, we ensure that the starting Amino Acid is unique for all the dataset sequences to avoid ambiguous predictions in the continual learning evaluation. A sample of the protein sequence is provided below:

MGAAASIQTTVNTLSERISSKLEQEANASAQTKCDIEIGNFYIRQNHGCN LTVKNMCSADADAQLDAVLSAATETYSGLTPEQKAYVPAMFTAALNIQTS VNTVVRDFENYVKQTCNSSAVVDNKLKIQNVIIDECYGAPGSPTNLEFIN TGSSKGNCAIKALMQLTTKATTQIAPKQVAGTGVQFYMIVIGVIILAALF MYYAKRMLFTSTNDKIKLILANKENVHWTTYMDTFFRTSPMVIATTDMQN

Text To evaluate the generative ability of PAM, we use a dataset of the most frequently used English words. For preprocessing, we extract one hundred 4-letter words from the dataset and create a mapping dictionary from all the unique letters in the dataset to random SDRs with N c = 100 and

W = 5 (except for AHN; W = 0 . 5 N c ). The dataset contains many words with ambiguous future possibilities. The selected words are provided below:

Vision In our experiment, we evaluate on sequences extracted from Moving MNIST [60], CLEVRER [68] as well as synthetically generated sequences of CIFAR [33] images. In order to convert images to SDRs and SDRs back to images while encoding semantics into the SDRs, we design an SDR AutoEncoder. The goal is to force the bottleneck representation of the autoencoder to become a sparse binary representation, where two visually similar images would result in two SDRs with a high overlap of active neurons. We simply design a CNN autoencoder with a 3-layer CNN encoder and 3-layer CNN decoder and apply top K binarization operation on the bottleneck embedding during training. The full architecture of the SDR autoencoder is shown in Figure 6.

In practice, we use a weighted average of the SDR and Dense representation to allow gradients of the reconstruction loss to freely propagate into the encoder. The weight of the SDR (i.e., α ) is gradually and linearly increased (from 0.0 to 1.0) with the number of training epochs. This gradual increase is fundamental to the training of the SDR Autoencoder as it smooths the loss landscape and allows the model to distribute the active bits on the full SDR. The total mse loss becomes L enc + L recon . We use Adam optimizer with a learning rate of 1 × 10 -4 . For Moving MNIST, we use a bottleneck embedding

Top K binarization

+

CNN

Encoder

CNN

Decoder

Dense

Sparse

Representation

0.23

-0.05

0.79

0.11

.

.

.

0.54

SDR

1

0

1

0

.

.

.

1

48

48

(i.e., N c ) of size 100 with 5 active bits, whereas, for more complex datasets (i.e., CLEVRER, CIFAR), we use an SDR of size 200 with 10 active bits. We show examples of the autoencoder reconstruction with full binary SDR (i.e., α = 1 ) for all three datasets in Figure 7.

Moving MNIST

Image

Reconstruction

CLEVRER

Image

Reconstruction

CIFAR

Image

Reconstruction

3

3

3

5/

5/

5/

5

5

5

4

4

D Implementation Details

In this section, we describe the implementation details and hyperparameters of each method. For each model, we optimize a single set of hyperparameters for all the experiments.

PAM The neurons in both, transition and emission, functions are fully connected. We do not assume any of the weight matrices are symmetric. All synaptic weights are initialized by sampling from a normal distribution with a mean of 0.0 and a standard deviation of 0.1. All η + values in Equations 7 & 8 are set to 0 . 1 . η -B is set to -0 . 1 , while η -A is set to 0 . 0 to avoid forgetting previous possibilities when learning new transitions; PAM learns a union of possibilities. The threshold for the δ function is set as a function of the SDR sparsity. For the transition function, we use a threshold of 0 . 8 W , where W is the active number of bits in the latent state ( z ) SDR. For the emission function, we use a threshold of 0 . 1 W , where W is the active number of bits in the observation state ( x ) SDR. During offline generation we sample an initial ˜ x from ↓ ˆ z with W = 1 active neurons. During generation, we set the maximum number of attractor iterations to 100 but stop iterating when the energy of the state converges to a local minimum. During sequence learning, we update A and B iteratively until the transition is learned before learning the next transition. This iterative weight update makes the model insensitive to the hyperparameter values η . Both A and B are always clamped in the range [ -1 , 1] . The states z are flattened into a single dimension before applying the learning rule in Equation 7. Binary representations (i.e., {0, 1}) are used as inputs.

Temporal Predictive Coding For the tPC architecture, we use learning rate of 1e-4 for 800 learning iterations. When a 2-layer tPC model is used, the inference learning rate is set to 1e-2 for 400 inference iterations. Also, the hidden size is set to twice the input size. We found that these parameters work best for all of the experiments and allow the model to fully converge. Bipolar representations (i.e., {-1, 1}) are used as inputs.

Asymmetric Hopfield Network The Hopfield model does not require hyperparameters other than the ablated separation function. In many experiments, we use a polynomial separation function with degree d set to 1 or 2. Bipolar representations (i.e., {-1, 1}) are used as inputs.

E Experiments

In this section, we describe the setup of each figure in the main paper and provide additional quantitative and qualitative results for each task. All experiments are run for 10 different seeds/trials. We report the mean and standard deviation in all the figures and tables.

E.1 Sequence Capacity

Figure 3 A This experiment plots the maximum offline sequence length (i.e., sequence capacity, T max ) at different input sizes. The input size N c is varied from 10 to 100 while the number of active bits W is fixed to 5 . We compare variants of our model with N k set to 4 and 8 to temporal predictive coding (tPC) and Asymmetric Hopfield Network (AHN). We observe that AHN completely fails as the sparsity S of the pattern decreases. Therefore, we also compare to AHN with the sparsity set to 50% (i.e., W = 0 . 5 N c ). For AHN models, we experiment with a polynomial exponential function with degrees 1 and 2 , as recently proposed [12] and used for evaluation in recent papers [62]. All models in this experiment are set to recall/generate in an offline manner, where only the first input is provided. PAM outperforms all other methods and has the potential to improve further by expanding the context neurons N k . The patterns in this experiment are uncorrelated, such that each pattern has active bits that are uniformly initialized.

Figure 3 B This experiment plots the effect of sequence correlation on the maximum offline capacity. The higher the correlation value, the more exact repetitions of patterns are available in the sequence. We enforce correlation by limiting the number of unique patterns (i.e., vocab) used to create the sequence. All patterns in this experiment are set to a size of N c = 100 and W = 5 (except for AHN, which is set at W = 0 . 5 N c ). Results show that the capacity of all other methods sharply drops when correlation is introduced. PAM retains most of its original capacity.

Figure 3 E & F In this experiment, we provide a qualitative example of a short sequence ( T = 10 ) with high correlation ( 0 . 8 ). The sequence is learned by all the methods; then we perform offline ( E ) and online ( F ) recall on the sequence. We use the SDR autoencoder to create SDRs from these CIFAR images for training and recall. The SDRs have a size N c of 200 and W = 10 . In the offline recall, only the first input is provided and the model auto-regressively generates the full sequence using its own predictions at every time step. In online recall, the models perform a single-step prediction and always use the ground truth input at every time step to perform predictions. Results show that only PAM can retain a context of correlated sequence and accurately predict into the future based on this context.

Figure 8 A In this experiment, we show the effect of scaling the model context memory beyond a simple N k = 4 and N k = 8 . We show that when using N k = 16 and N k = 24 , PAM can model much longer sequences. We vary the input size N c from 10 to 50 and report the offline sequence capacity of the model as ablations.

A

B

Figure 8 B Similar to the experiment plotted in Figure 3 A , we report the sequence capacity with input size N c . However, this experiment evaluates the online generation capacity, where the model uses the correct pattern at every prediction time step instead of using its own prediction from the previous time step. Results show that PAM significantly increased in capacity (three times in some cases), whereas the other methods have not increased as much in modeling longer sequences.

Figure 9 We provide additional qualitative example on a different highly correlated sequence. The result shows a different failure mode for AHN, whereas PAM still performs well.

E.2 Catastrophic Forgetting

Figure 4 A We benchmark the performance of difference models in the challenging continual learning setup. The models are expected to avoid catastrophic forgetting by not overwriting previously learned sequences. In this experiment, we use 50 sequences, each with size N c = 100 and length T = 10 . We vary the correlation of the sequences from 0 . 0 to 0 . 5 and compute the backward transfer metric with the normalized IoU as the measure of similarity. Results show that AHN can avoid catastrophic forgetting when the sequence is uncorrelated but quickly drops in performance with correlation. tPC fails to retain learned sequences regardless of correlation. PAM performs well with more context neurons N k . When setting N k to 1 , the model fails to retain its knowledge due to the decreased context modeling capability with a single context neuron.

tPC L=1

tPC L=2

AHN d=1

AHN d=2

PAM     = 8

Memories

Offline

Online

Input

Hidden

Figure 4 B In this experiment, we report the performance of the models on protein sequences. This is a more challenging setup due to the long sequence (a few hundred on average) with high correlation (only 20 unique Amino Acids). We show a similar trend where the other methods fail due to high correlation or sequence lengths. PAM outperforms the other methods when using context memory N k of 16 or 24 . All Amino Acid types are converted to fixed and randomly initialized SDRs with N c = 100 . The sparsity is set similar to sequence capacity experiments (i.e., W = 5 and W = 0 . 5 N c ).

Figure 4 F We provide qualitative results on a simple experiment with 2 sequences from moving MNIST. The models learn the first sequence and then learn the second sequence. The models are not allowed to train on the first sequence after they have trained on the second sequence. We then perform online generation on the first sequence with all models. We use the SDR autoencoder to generate SDRs for all images in the sequences; the SDRs have N c = 100 with W = 5 (for all methods except AHN). The results show that PAM can recall the full sequence even after being trained on another sequence. Other methods fail in this simple task, even in online recall setup.

Figure 10 We provide continual learning results similar to Figures 4 A & B ; however, instead of offline generation, we perform the evaluation in online manner.

Figure 11 We provide additional qualitative results on Moving MNIST, as well as quantitative results averaged over 10 trials of MNIST sequence pairs. These quantitative results are reported as the Mean Squared Error of the reconstructed image. Results show that PAM reports the lowest error with a much smaller variance.

Tables 2, 3, 4, & 5 The Backward Transfer metric (BWT) to evaluate catastrophic forgetting is computed by taking the average of the performance on previously learned sequences after training on a new sequence. In addition to plotting this average in previous experiments, we provide the full tables for one of the experiments as an example. All tables show results on 10 sequences and N c = 100 . The BWT metric is calculated as the average of the similarity metric reported in these tables.

time

Memories

Then

tPC L=1

tPC L=2

AHN d=1

AHN d=2

PAM     = 8

E.3 Multiple Possibilities Generation

In this task, we evaluate the models' ability to generate meaningful sequences and recall the full dataset despite being presented with multiple valid possibilities. Ideally, the models are expected to sample a single possibility if trained on sequences with ambiguous future continuations (equally valid possibilities). This is a challenging task for most biologically plausible (e.g., tPC, AHN, etc.) and implausible (e.g., transformers, RNNs, etc) models. Most approaches assume the existence of a full set of possibilities and transform the task from regression to classification (e.g., LLM). For vision tasks, some methods (VQ-VAE and its variants) cluster the dense representations to create this set of possibilities and perform classification. We do not assume the existence of a full set of possibilities but instead perform a true generative evaluation as a regression task.

Figure 4 C In this experiment, we compute the average normalized IoU of the generated words. The model's ability to generate a full sequence with high IoU means it can produce sharp single predictions despite being trained on multiple equally valid future predictions. As the number of words

increases, the performance of other models decreases as they struggle to model ambiguous future predictions; however, PAM outperforms the other approaches by sampling from these possibilities.

Figure 4 D This experiment evaluates the ability of the models to recall the dataset words. We compute the recall as the number of valid unique words generated divided by the total number of words in the dataset. Since PAM is a generative stochastic model, the recall increases with every generation. The other methods are deterministic and, therefore, do not report an increase in dataset recall with more generations. The other methods completely fail to generate any meaningful words. We use an average IoU threshold of 0.9 to classify a generated word as correct, similar to sequence capacity experiments.

Figure 12 We provide qualitative results by showing the unique generated words by different models after 5 dataset generations. PAM N k = 4 generates some of the dataset words but also generates many wrong words. By increasing the context memory neurons to N k = 8 , the model generates many more correct words and reduces the false positives. The other methods cannot generate meaningful words.

part - call - live - side - both - play - from - move - open - city - left - over - name - form - were - port - more - make - came - hard

- most - take - line - some - life - self - like - made - same - near - than - many - late - land - does - this - down - will - farm - hand

- turn - work - very - then - been - tell - year - good - must - draw - show - when - said - also - grow - last - know - them - only -

read - want - with - what - back - just - such - need - ease - keep - well - tree - look - head - kind - give - page - even - food -

home - four - they - long - here - time - that - much - come - have - seem - next - mean - your - walk - went - help - word - find -

each - real - high

Full

Dataset

meem - next - bany - evbe - page - keep - word - came - open - read - kind - went - some - late - seem - tell - city - turn - more -

will - bada - move - grow - were - here - keee - near - draw - mean - tree - left - real - such - does - meee - hand - four - very -

much - come - back - head - looo - lina - show - with - most - land - from - higa - port - ward - food - whee - good - even - ease -

your - only - play - both - name - walk - what - meel - evbr - give - down - year - find - life - want - home - farm - just - form - alea

- call - meea - must - hard - alei - each - high - meei - been

name - most - when - turn - more - will - from - land - next - move - side - grow - port - walk - evaf - were - what - here - look -

give - near - year - food - find - life - mean - draw - want - page - line - home - farm - just - keep - tree - word - left - came - real -

have - call - open - form - good - such - time - help - does - kind - went - even - must - ease - hard - four - hand - also - some -

very - high - your - play - come - back - late - both - seem - tell - head - well - city - been - show - with

hbaa - oaaa - naaa - keaa - laaa - gaaa - thaa - eaaa - mbaa - caaa - raaa - saaa - paaa - veaa - faaa - jaaa - yaaa - waaa -

aaaa - daaa - baaa

oaaa - dcaa - naaa - keaa - laaa - maaa - eaaa - raaa - caaa - saaa - yeaa - paaa - veaa - faaa - haaa - taaa - juca - gcaa -

waaa - aaaa - baaa

PAM-4

PAM-8

tPC

AHN

E.4 Noise Robustness

Figure 3 C To evaluate for noise robustness, we plot each model's performance (Normalized IoU) with varying levels of noise added in an online generation setting. The noisy inputs are created by changing a percentage of the active neurons to different uniformly chosen neurons in the SDR. The noise is computed as a percentage for a fair comparison across different SDR sparsities (e.g., tPC vs. AHN). We show the results for sequences of lengths T = 200 and no correlation. PAM has the ability to compare the noisy input representation to the learned attractors to recover the correct clean input. Therefore, even when the SDR is completely changed, PAM relies on its predictions and completely ignores the noisy input. During generation, PAM always generates (or corrects a noisy input) from within the predicted set of possibilities. The other approaches use the noisy inputs during recall, which affects their performance.

Figure 4 E We provide qualitative results on the CLEVRER dataset. The memories sequence is learned by all the models, then a noisy sequence is used during generation. We only add noise starting from the second pattern in the input sequence. The results show that tPC models are performing relatively well, yet they are still outperformed by PAM N k = 8 . We set N c = 200 in the SDR autoencoder to learn the SDRs used in this experiment. We use 40% noise in this experiment.

Figure 13 We perform additional experiments on varying the sequence lengths and the correlation in the sequence, all the other settings remain the same as in the experiment of Figure 3 C . The results

show that with shorter sequences ( ≤ 200 ), no noise, and no correlation, all the models recall the learned sequence well. When higher correlation is used, 2-layered tPC performs relatively well with short sequences (i.e., T = 10 ) but fails with longer sequences (i.e., T = 100 ). The Hopfield model fails more with correlation than sequence length. The added noise affects all reported methods except for PAM due to its ability to rely on its predictions and attractors to clean the noisy signal.

tPC L=1

tPC L=2

AHN* d=1

AHN* d=2

PAM     = 8

time

Memories

E.5 Efficiency

Figure 3 D We compare the efficiency of the models and show that PAM is at least two orders of magnitude more efficient than tPC with 2 layers. A single layer tPC is almost equivalent to PAM with high context memory of N k = 24 . AHN is highly efficient as the model is not usually trained, but the recall equation is used instead. Therefore, we exclude AHN from the comparison.

E.6 Additional Results

A

B

C

Figure 15 A We show the performance of PAM in an online and offline generation task as the sequence length increases. In offline generation, incorrect predictions accumulate quickly, leading to a sharp drop in performance. However, online generation attempts to correct inaccurate predictions; therefore, performance smoothly degrades with sequence length. Note that both experiments (i.e., online and offline) use identical seeds/sequences for a fair comparison.

Figure 15 B We provide additional results on testing the effect of connection density. While neurons in PAM are currently densely connected (similar to Hopfield Network), we show that PAM can still perform relatively well in a sparser setup.

Figure 15 C We provide additional results comparing PAM to a 2-layer transformer on continually learning protein sequences. Backward transfer results show low performance of transformers in remembering older sequences after training on new sequences. Table 6 supplements these results by showing detailed sequence-by-sequence performance values. Transformers excel at learning the current task (diagonal values) but suffer in remembering previously learned tasks (lower triangle values).

F Discussions

F.1 Biological Plausibility

PAM inherits assumptions from Hierarchical Temporal Memory (HTM) [25] that the sequence modeling takes place in Layer IV of the cortical column, where it receives both thalamocortical sensory driving input and contextual information. Sensory input arrives on the proximal dendrites of pyramidal cells, while contextual information reaches the basal dendrites, priming specific contexts by depolarizing neurons. Depolarized neurons then fire first and inhibit other neurons within the same minicolumn via lateral inhibition, a process assumed to be mediated by interneurons (possibly Double Bouquet Cells (DBCs)) [49]. HTM assumes interneurons [40] perform lateral inhibition during prediction but does not learn the synapses of the interneurons. Interneurons are also assumed to perform the inhibitory tasks of the attractor model [14].

Biological plausibility in PAM mainly refers to excluding the obvious biologically implausible learning rules, such as backpropagation or copying of weights. This allows models like PC and HN to be referred to as biologically plausible despite clear contradictions with anatomy [54, 14]. Moreover, PAM also constrains its learning and representations to sparse and binary cell assemblies, which improves the plausibility of the model. Therefore, a biologically plausible model here refers to the ability to continually learn using only local learning rules (e.g., Hebbian, anti-Hebbian, STDP) and sparse binary representations. However, we do not intend to accurately map every detail in PAM to the anatomy of the neocortex.

F.2 Future Directions

In future work, we plan to implement a hierarchy of PAM blocks that supports compositional, partwhole predictions. Unlike simply stacking layers as in Transformer or CNN architectures, our aim is to establish a hierarchy where higher-level blocks make predictions based on broader, contextual patterns. For instance, in vision, higher levels could predict entire objects based on multiple visual cues, while in NLP, they could predict phrases or sentences rather than individual tokens. To achieve this, each PAM level would send a summary representation of the current sequence (its latent state) to the next level for recursive, higher-order processing.

Additionally, we plan to explore top-down feedback mechanisms between PAM blocks - inspired by the hierarchical feedback observed in the neocortex. In the visual system, for example, top-down signals travel from higher regions (e.g., V2) to lower ones (e.g., V1), reaching the apical dendrites of pyramidal neurons. These connections originate in layer VI(a) of the higher region and form en-passant synapses in layer VI(a) of the lower region in the hierarchy (e.g., V1). These connections reach Layer I and provide feedback on the apical dendrites of other layers [61]. In PAM, a similar feedback pathway could enable higher-level blocks to modulate lower-level processing based on partial observations, refining predictions by ruling out certain lower-level possibilities that conflict with higher-level context. For example, the attractor in the higher-level block can reach a decision on the object being seen (e.g., human) based on partial observation (V1: layer III → V2: layer IV), and therefore, the feedback inhibitory connections on the apical dendrites can rule out some lower-level possibilities (e.g., paws).

G Sparse Distributed Representations

The neocortex stores and represents information using sparse activity patterns, as demonstrated by empirical evidence [4]. Inspired by HTM [25] and neuroscience-based theories of cortical function, we use Sparse Distributed Representations (SDRs) as the main representation format of PAM. An SDR is a sparse binary representation of a cell assembly where only a small fraction of the neurons in the SDR are active at any time. The location of these active neurons encodes the information that is represented by this SDR. In this section, we describe some useful properties of SDRs and discuss their robustness to noise as opposed to dense representations.

G.1 SDR Properties

SDRs are used to represent rich sensory information in the neocortex as a sparse activity pattern. Therefore, from a mathematical viewpoint, an SDR must have the ability to represent many patterns and easily distinguish between them. The capacity of an SDR can be calculated as the possible combinations of locations where neurons can be active. Consider an SDR with size N and number of active neurons W . The total capacity of this SDR is computed as shown in Equation 12.

Based on the above Binomial coefficient equation, it may seem that sparsity is not optimal for capacity as the capacity will be the highest when W is exactly half of N . While capacity is important, we aim to represent multiple possibilities as a union of SDRs and, therefore, minimize the overlap between them. From an information-theoretic viewpoint, the goal is to minimize mutual information between SDRs to ensure that each SDR carries unique information and the union represents a more comprehensive and diverse set of features. We can minimize the expected IoU by using lower sparsities as shown in Theorem 2. In our experiments, we use N = 100 and W = 5 , which results in a capacity of ≈ 75 × 10 6 and an expected IoU of ≈ 0 . 02 . However, when scaled up to more typical values of SDR sizes and sparsities in the neocortex [25, 4] (i.e., N = 2048 , W = 40 ), we get capacity of ≈ 2 . 37 × 10 84 (more than the estimated number of atoms in the observable universe ≈ 10 80 ) and expected IoU of ≈ 0 . 01 . A sparsity of 0 . 5 maximizes the mutual information and results in an expected IoU of 0 . 33 , which cannot be used to represent multiple possibilities as a union of SDRs in spite of the optimal capacity. In practice, the size N of the SDR is increased to increase the capacity, and the sparsity W/N is decreased to minimize the expected overlap.

G.2 The robustness of SDRs

Sparse representations naturally minimize the overlap between random SDRs; therefore, they are very tolerant to noise. To visualize this robustness property of SDRs, we design an experiment (Figure 16) where we train an SDR autoencoder with different sparsities and then decode SDRs at various levels of noise added. When the sparsity is increased to 50% , there is a high chance of overlap between SDRs. Therefore, a small amount of noise can cause collisions between SDRs. However, a 5% sparsity can tolerate much more noise without overlapping with other SDRs.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Noise Level added to SDR

5%

25%

50%

5%

25%

50%

5%

25%

50%

SDR Sparsity

SDR Sparsity

SDR Sparsity

国

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims are supported by experimental results and theoretical proofs where applicable. Derivations are provided in the supplemental material.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the limitations section has been moved from the supplemental material to the main paper for the camera ready version.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide proofs and derivations in the appendix, where applicable.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide general implementation details for all the methods used. Additionally, we describe the experimental setup for every plot/table in the main paper and in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our codebase (now released) includes a script for every single experiment to accurately reproduce the plots in the paper. We fix the seeds for exact reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide implementation details for all methods and the datasets' details in the appendix. The code also contains a separate configuration file with all the needed parameters.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are repeated 10 times at different seeds. The means and standard deviations are provided in all plots and tables.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify the compute resources required for PAM (i.e., CPU) and plot a comparison of the time required by each method at different parameters in Figure 3 D .

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: PAM is a powerful sequence modeling algorithm. We do not see direct positive or negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Authors of comparison methods and datasets are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are released.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with training generative sequence models under a few constraints to make it biologically plausible. \nSpecifically, states consist of a sparse binary representation (i.e. neurons that are on/off and sequence training and generation are performed using operations that are described as \"local and synaptic\". The proposed model seems to be a modification / generalization of Hopfield networks (a form of recurrent network) such that weights are asymmetric and there is an orthogonal \"hidden\" dimension to the representation (hidden in that is does not affect the output directly) that captures prior history and, in effect, turns the model in to an HMM.\nState transitions involve a linear transition matrix, lateral inhibition (within the hidden state dimension using a max operation) and output-dependent filtering. Output is generated through sampling from a Gaussian mixture model of attractors. The method is backed with some theoretical guaranty of optimality of the observation generation.\n\nThe model is compared against others of it's kind in online and offline (teacher-forced) settings and assessed with respect to memory capacity (number of sequences and their lengths), robustness to noise, correlation between states, training speed and parameterized hidden state dimension. The proposed model outperforms in every way.\n\nSTRENGTHS:\nThe proposed model seems like a natural extension of Hopfield networks to the sequence and hidden-state domains. \nThe state, training and inference all seem to adhere to the constraints that make the method \"biologically plausible\".\nThe various (synthetic and \"real\") are used well to show the model streangths. Performance is very clearly better than the models against which it is compared.\nThe presentation is very clear. The accompanying video-based presentation is superb. The supplementary material is very extensive and provides additional background, proofs and implementation details at the same level of attention to clarity and detail.\n\nWEAKNESSES:\n* It is not clear to me what constitutes a biologically plausible mathematical operation. Are the various models equally biologically plausible? Is there a scale of plausibility? With that in mind, I wonder how the models that appear in this study compare against state of the art sequence memorization / generative models that are unconstrained (e.g. transformers?), whether those models are better and are they provably non-plausible. It would have been interesting to read a discussion of whether biological plausibility can help us produce better models or whether (/ how) it is helpful in explaining observations from studies of real biological networks.\n\n* Clearly, both memory capacity and training efficiency are highly dependent on the level of sparcity and the number of parameters (be they weights or neurons). The comparison across models is somewhat limited in its treatment of model complexity (albeit the proposed PAM models are compared across several  hidden-dimension sizes). I'd expect that some models would match PAM performance in some respects as they are scaled up. It may be useful for the analysis to be extended to cover this aspect of the models better.\n\nQUESTIONS:\nThe state transition and the output mechanisms in PAM are different. One uses attractor dynamics and the other uses a deterministic transition and a max operation. Wouldn't it be more biologically plausible to have operations use similar mechanics?\n\nIt is unclear whether the models presented are useful from a machine learning point of view. Taking the word generation as an example, the level of complexity of that task is orders of magnitude below what language generation models are capable of. Assuming this assertion is correct, what does that imply about the state of what we consider biologically plausible compared with what real brains are capable of and what unconstrained ML methods can achieve. Is there an unaddressed gap?\n\nLIMITATIONS:\nIt is unclear how the methods compare agains general ML methods. It is unclear whether the assumed biolgical plausibility helps understand biological neural mechanisms.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel sequence memory architecture focused on multistability, ie: generating multiple possible futures from every present context. They propose to train this architecture in a biologically plausible way via predictive coding under the Free Energy Principle, in the online learning setting where data items are seen once instead of repeated in mini-batches. Section 2 reviews background, and Section 3 describes the Predictive Attractor Model formally.  In Section 4 the paper evaluates PAMs for their storage capacity, robustness against catastrophic forgetting, multistability, and noise robustness, comparing to symmetric and asymmetric Hopfield Networks and temporal predictive coding (tPC).\n\nSTRENGTHS:\nThe paper attempts and achieves a high degree of biological plausibility by focusing on specific learning mechanisms, observed in the brain, for each computation that must be carried out.  To my knowledge it is the first sequence memory architecture to explicitly consider a mixture likelihood in order to consider multiple possible continuations given every context. I have a hard time evaluating the quality of the experimental results, since I have not kept up with the state of the art in sequential memory models this past year or two.\n\nWEAKNESSES:\nClarity is the main aspect in which the paper lacks.  The paper uses a lot of concepts without either initially laying them out together or introducing them explicitly the first time they are needed and then referencing back to them when they are used.  A table of notations somewhere would be quite helpful.  For instance, presenting terms as \\log 1/p(x) sows a bit of confusion over whether a likelihood ratio/Radon-Nikodym derivative is the intended concept being conveyed, or just \\log 1/p(x) = -\\log p(x) giving an entropy or cross-entropy.\n\nQUESTIONS:\nCan the authors provide a unified presentation of the different biologically-informed learning rules/computations they propose, and how those unite back with the variational inference perspective introduced at the start of the paper?\n\nI would suggest that the authors provide some hyperlink to Table 1 (in the Supplementary material, giving notations) in the main text, or even move Table 1 into the main text.\n\nRegarding the neuroscience, why does the paper refer to a cortical minicolumn (rather than a laminar column or cortical microcircuit?) rather than a hippocampal circuit?  Sequential memory architectures, particularly those akin to Hopfield networks, are more typically considered as models of hippocampal circuitry.\n\nLIMITATIONS:\nThe authors freely admit that their model relies on having a pretrained model, such as an autoencoder, to convert sensory-level inputs into sparse distributed representations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines a biologically plausible model for sequential memory which overcomes various issues (capacity, forgetting, size of context window) in previous models. The proposed model is essentially an extension of Hierarchical Temporal Memory which allows the model to generate predictions.\n\nSTRENGTHS:\n* Memorizing sequences is a fundamental ingredient of cognition and remains a challenge for neuroscience, while no theoretical model has been completely satisfactory.\n* Hierachical Temporal Memory remains a compelling model of sequence memorization and attempts to expand its capabilities are worthwhile.\n* The model presented here is simple, well-explained, and composed of familiar ingredients.\n* The experiments are clear and seem appropriate.\n\nWEAKNESSES:\n* This paper's contribution seems to be a bit weak. What it builds upon Hierarchical Temporal Memory [1] is the ability for the model to generate predictions, which can be used to sample autoregressively or robustly handle noisy input. While the sampling capability in particular could be interesting, key properties of such a function -- perhaps most obviously, which distribution does it produce? Can it learn the input statistics? -- are not explored.\n* In relation to the above, it inherits a (slight) weakness from HTM: reliance on the ability of dendritic computation to put neurons into a \"predictive\" state, which is necessary (but not sufficient) for firing. We should be leery of overstating the biological plausibility of what remains a hypothetical mechanism.\n* The model is biologically plausible in that it relies on local updates, but various other aspects of it (especially the configuration of the minicolumns) seems to need to be set up precisely in advance, and dense connectivity is assumed.\n* The theoretical results amount to showing that sampling (an observation given context) can be written as a Hopfield recall problem. A deeper theoretical analysis of this model -- touching on capacity, number of rounds for convergence, and characterizing the distribution which is sampled from -- would significantly strengthen this paper's contribution.\n\n[1] Hawkins, Jeff, and Subutai Ahmad. \"Why neurons have thousands of synapses, a theory of sequence memory in neocortex.\" Frontiers in neural circuits 10 (2016): 174222.\n\nQUESTIONS:\n* What does $\\mathcal N(x; \\mu_c, \\Sigma_c)$ symbolize in Eq. (4)?\n* It would be nice to examine a crucial step of the generative algorithm -- lines 7-8 in algorithm 2, which combine a Hopfield step with influence from the context -- in more detail, theoretically and/or in experiment. If something like this has been explored in the literature before, it would also be good to point to it.\n* Please define Jaccard index/IoU in the main text.\n\nLIMITATIONS:\nSince the paper claims biological plausibility, I would appreciate a more thorough and insightful discussion of what features are plausible and which are less so, with references to the relevant neurophysiology literature.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Predictive Attractor Models (PAM), designed to improve sequence memory by resolving issues like catastrophic forgetting. PAM processes inputs in real-time, remembering each just once, using lateral inhibition in cortical minicolumns to preserve old memories. By generating future predictions from previously trained possibilities, PAM leverages synaptic adjustments and competitive learning, inspired by Hierarchical Temporal Memory systems. This combination enhances memory robustness and versatility across different data types, marking a significant advance in creating biologically plausible, computationally efficient memory systems for cognitive science and AI research.\n\nSTRENGTHS:\n1. By processing each input only once and using mechanisms such as lateral inhibition, PAM effectively retains old memories without them being overwritten by new ones.\n2. PAM's ability to handle inputs in real-time allows for dynamic updating of memory without the need for batch processing or retraining.\n3. It can generate multiple future predictions based on a set of learned possibilities, providing flexibility and adaptiveness in response generation.\n4. PAM has been tested and shown to be effective across various types of data, including text, visual inputs, and protein sequences, demonstrating its applicability in diverse fields.\n\nWEAKNESSES:\n1. Need for Detailed Comparison with HTM: There is a necessity for a more thorough comparison between PAM and Hierarchical Temporal Memory (HTM). By detailing the differences and improvements PAM offers over HTM, the unique contributions of PAM can be better understood and appreciated. This analysis would help delineate PAM's novel features and demonstrate its advancements over HTM in handling sequence memory.\n\n2. Risk of Overfitting: PAM's robust capacity to utilize past information and generate predictions from learned attractors might lead to overfitting. This risk is particularly high in environments with highly variable or noisy data. Additionally, if the sequence itself contains repetitive elements meant to convey information, PAM may struggle to capture this, potentially losing the ability to extract meaningful patterns from repetitions in the data.\n\n3. Parameter Sensitivity: The performance of PAM heavily relies on the precise configuration of its parameters, including the strength of synaptic connections and the degree of lateral inhibition. Incorrectly calibrated parameters can significantly impair the model's functionality, making it less effective and reducing its overall reliability.\n\nQUESTIONS:\n1. The implementation of PAM involves complex mechanisms, such as lateral inhibition in cortical minicolumns and synaptic adjustments, which can be more intricate compared to traditional memory models. Moreover, the dimensionality of matrices involved in PAM (e.g., matrix A and B's dimension being $(N_c \\times N _k)^2+N_c^2$ contributes further to this complexity. Such high complexity demands more computational resources and sophisticated programming techniques, which may limit PAM’s accessibility and scalability in practical applications.\n\n2. In the introduction, it is emphasized that different contexts in a sequence should lead to different perceptions of the same stimulus. However, there is a concern that PAM might merely memorize the entire sequence without effectively considering the variability in sequence content. This approach could potentially overlook the importance of recognizing repetitive structures within sequences, which are crucial for efficient sequence memory. The ability to identify and utilize such repetitions can greatly enhance memory efficiency and predictive accuracy, suggesting a potential area for improvement in PAM's design.\n\nLIMITATIONS:\nThe article compares related work in the introduction and mentions some limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Predictive Attractor Models (PAM), a novel biologically-inspired approach for sequential memory and prediction. PAM combines a predictive model based on sparse distributed representations (SDRs) with an attractor model for generating future predictions. Key contributions include an online learning algorithm that encodes temporal context without catastrophic forgetting, the ability to represent and sample from multiple possible future predictions, noise robustness through learned attractors, and formulation as a variational inference problem within a state space model framework. The authors evaluate PAM on various sequence learning tasks, demonstrating superior performance compared to temporal Predictive Coding (tPC) and Asymmetric Hopfield Networks (AHN) in terms of sequence capacity, avoiding catastrophic forgetting, generating multiple possibilities, and noise robustness.\n\nSTRENGTHS:\n- Novelty in combining predictive coding with attractor dynamics for sequential memory.\n- Strong theoretical grounding within a variational inference framework.\n- Comprehensive evaluation on various tasks and datasets.\n- Biological plausibility through local learning rules and sparse binary representations.\n- Addressing key challenges in sequential memory (catastrophic forgetting, multiple possibility generation, noise robustness).\n\nWEAKNESSES:\n- Limited comparison to modern sequence learning approaches (e.g., transformers).\n- Scalability concerns for very long sequences or high-dimensional inputs.\n- Lack of hierarchical processing discussion or implementation.\n- Insufficient analysis of hyperparameter sensitivity.\n- Limited discussion of failure modes and limitations.\n\nQUESTIONS:\n1. How does PAM's performance change with increasing sequence length?\n2. Can PAM be extended to handle variable-length or incomplete sequences?\n3. How does PAM perform with more structured types of noise?\n4. Can you provide more insight into the attractor model's learning of multiple possibilities?\n5. Have you explored more sophisticated SDR encoding schemes for complex inputs?\n6. How might PAM incorporate top-down feedback or attention mechanisms?\n7. What specific neural circuit predictions arise from your model?\n\nLIMITATIONS:\n- The requirement for sparse binary representations limits direct application to dense inputs. \n- The use of a separate autoencoder for vision experiments adds complexity. \n- The paper lacks thorough discussion of computational complexity and memory requirements for larger-scale problems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}