SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mufeng Tang, Helen Barron, Rafal Bogacz
MRC Brain Network Dynamics Unit, University of Oxford, UK {mufeng.tang, helen.barron, rafal.bogacz}@bndu.ox.ac.uk
Forming accurate memory of sequential stimuli is a fundamental function of biological agents. However, the computational mechanism underlying sequential memory in the brain remains unclear. Inspired by neuroscience theories and recent successes in applying predictive coding (PC) to static memory tasks, in this work we propose a novel PC-based model for sequential memory, called temporal predictive coding (tPC). We show that our tPC models can memorize and retrieve sequential inputs accurately with a biologically plausible neural implementation. Importantly, our analytical study reveals that tPC can be viewed as a classical Asymmetric Hopfield Network (AHN) with an implicit statistical whitening process, which leads to more stable performance in sequential memory tasks of structured inputs. Moreover, we find that tPC exhibits properties consistent with behavioral observations and theories in neuroscience, thereby strengthening its biological relevance. Our work establishes a possible computational mechanism underlying sequential memory in the brain that can also be theoretically interpreted using existing memory model frameworks.
The ability to memorize and recall sequences of events with temporal dependencies is crucial for biological memory systems that also underpins many other neural processes [1-3]. For example, forming correct memories of words requires humans to memorize not only individual letters but also the sequential order following which the letters appear (e.g., "cat" and "act"). However, despite extensive research into models of static, temporally unrelated memories from both neuroscience and machine learning [4-10], computational modeling of sequential memory is not as developed. Existing models for sequential memory are either analytically intractable or have not yet been systematically evaluated in challenging sequential memory tasks [11-15], which hinders a comprehensive understanding of the computational mechanism underlying sequential memory, arguably a more general form of memory in the natural world than static memories.
In this work, we propose a novel approach to modeling sequential memory based on predictive coding (PC) [16-18], a biologically plausible neural network model able to reproduce many phenomena observed in the brain [19], which has also shown to have a close relationship to backpropagation in artificial neural networks [20, 21]. Using PC to model sequential memory is motivated by two key factors: Firstly, neuroscience experiments and theories have suggested that (temporal) predictive processing and memory are two highly related computations in the hippocampus, the brain region crucial for memory [22, 23]. Secondly, the modeling of static memories (e.g., a single image) using PC has recently demonstrated significant success [8, 10, 24, 25], raising the question of whether PC can also be employed to model sequential memory. To take into account the temporal dimension in sequential memory, in this work we adopt a temporal extension of the original PC models, which has been employed in filtering problems and in modeling the visual system [26-29]. Here we investigate its performance in sequential memory tasks. Our contributions can be summarized as follows:
- We propose temporal predictive coding (tPC), a family of PC models capable of sequential memory tasks that inherit the biologically plausible neural implementation of classical PC models [18];
- We present an analytical result showing that the single-layer tPC can be viewed as the classical Asymmetric Hopfield Network (AHN) performing an implicit statistical whitening step during memory recall, providing a possible mechanism of statistical whitening in the brain [30-32];
- Experimentally, we show that the whitening step in single-layer tPC models results in more stable performance than the AHN and its modern variants [15] in sequential memory, due to the highly variable and correlated structure of natural sequential inputs;
- We show that tPC can successfully reproduce several behavioral observations in humans, including the impact of sequence length in word memories and the primacy/recency effect;
- Beyond memory, we show that our tPC model can also develop context-dependent representations [2, 11] and generalize learned dynamics to unseen sequences, suggesting a potential connection to cognitive maps in the brain [33, 34].
Predictive Coding Models for Static Memory The original PC model for memory follows a hierarchical and generative structure, where higher layers generate top-down predictions of lower layers' activities and the network's learning and inference are driven by the minimization of all the prediction errors [8]. Since the memorized patterns minimize the prediction errors in PC, they can be considered attractors of the energy function defined as the (summed) prediction errors, which is similar to the energy perspective of Hopfield Networks (HNs) [4]. Subsequent research has also shown that PC models for memory can be formulated as recurrent networks to account for the recurrently connected hippocampal network [10, 25], and that continual memorization and recall of a stream of patterns can be achieved by combining the hierarchical PC model with conjugate Bayesian updates [24]. Despite these abundant investigations into the memory capability of PC models, they all focused on static memories. Although the BayesPCN model [24] is capable of recalling patterns memorized in an online manner, the memories stored in this model are still static, order-invariant patterns, rather than sequences of patterns with underlying temporal dependencies.
Predictive Coding with Time PC was extended to include temporal predictions by earlier works (see [35] for a review) noticing its relationship to Kalman filters [36]. However, the Kalman filtering approach to temporal predictions sacrifices the plausible neural implementation of PC models due to non-local computations, and a recent work proposed an alternative way of formulating PC with temporal predictions that inherits the plausible implementation of static PC, while approximating Kalman filters [37]. However, none of these models were examined in memory tasks. Broadly, models based on temporal predictions were proposed in both neuroscience and machine learning [29, 38-40]. However, these models are either trained by the implausible backpropagation or rely on complex architectures to achieve temporal predictions. In this work, our tPC model for sequential memory is based on the model in [37], which inherits the simple and biologically plausible neural implementation of classical PC models that requires only local computations and Hebbian plasticity.
Hopfield Networks for Sequential Memory Although there exist other models of sequential memory [11-13], these models are mostly on the conceptual level, used to provide theoretical accounts for physiological observations from the brain, and are thus hard to analyze mathematically. Therefore, here we focus our discussion on the AHN [14] and its modern variants [15], which extended the HN [4] to account for sequential memory, and have a more explicit mathematical formulation. Denoting a sequence of  $P + 1$  patterns  $\mathbf{x}^{\mu} (\mu = 1, \dots, P + 1)$ ,  $\mathbf{x}^{\mu} \in \{-1, 1\}^{N}$ , the  $N \times N$  weight matrix of an AHN is set as follows:
$$
\mathbf {W} ^ {A H N} = \sum_ {\mu = 1} ^ {P} \mathbf {x} ^ {\mu + 1} \left(\mathbf {x} ^ {\mu}\right) ^ {\top} \tag {1}
$$
Notice that it differs from the static HN only by encoding the asymmetric autocovariance rather than the symmetric covariance  $\sum_{\mu = 1}^{P}\mathbf{x}^{\mu}(\mathbf{x}^{\mu})^{\top}$ , thus the name. The single-shot retrieval, which we define as  $R$ , is then triggered by a query,  $\mathbf{q}\in \{-1,1\} ^N$ :
$$
R ^ {A H N} (\mathbf {q}) = \operatorname {s g n} \left(\mathbf {W} ^ {A H N} \mathbf {q}\right) = \operatorname {s g n} \left(\sum_ {\mu = 1} ^ {P} \mathbf {x} ^ {\mu + 1} \left(\mathbf {x} ^ {\mu}\right) ^ {\top} \mathbf {q}\right) \tag {2}
$$
The retrieval process in Eq. 2 can be viewed as follows: the query  $\mathbf{q}$  is first compared with each  $\mathbf{x}^{\mu}$  using a dot product function  $(\mathbf{x}^{\mu})^{\top}\mathbf{q}$  that outputs a similarity score, then the retrieval is a weighted sum of all  $\mathbf{x}^{\mu +1}$ $(\mu = 1,\dots,P)$  using these scores as weights. Thus, if  $\mathbf{q}$  is identical to a certain  $\mathbf{x}^{\mu}$ , the next pattern  $\mathbf{x}^{\mu +1}$  will be given the largest weight in the output retrieval. Following the Universal Hopfield Network (UHN) framework [7], we can generalize this process to define a retrieval function of a general sequential memory model with real-valued patterns  $\mathbf{x}^{\mu}\in \mathbb{R}^{N}$ :
$$
R ^ {U H N} (\mathbf {q}) = \sum_ {\mu = 1} ^ {P} \mathbf {x} ^ {\mu + 1} \operatorname {s e p} \left(\operatorname {s i m} \left(\mathbf {x} ^ {\mu}, \mathbf {q}\right)\right) \tag {3}
$$
where sim is a similarity function such as dot product or cosine similarity, and sep is a separation function that separates the similarity scores i.e., emphasize large scores and de-emphasize smaller ones [7]. When sim is dot product and sep is an identity function, we get the retrieval function of the original AHN (for binary patterns an ad-hoc sgn function can be applied to the retrievals). Chaudhry et al. [15] have shown that it is possible to extend AHN to a model with polynomial sep function with a degree  $d$ , and a model with softmax sep function, which we call "Modern Continuous AHN" (MCAHN):
$$
R ^ {A H N} (\mathbf {q}, d) = \sum_ {\mu = 1} ^ {P} \mathbf {x} ^ {\mu + 1} \left(\left(\mathbf {x} ^ {\mu}\right) ^ {\top} \mathbf {q}\right) ^ {d} \tag {4}
$$
$$
R ^ {M C A H N} (\mathbf {q}, \beta) = \sum_ {\mu = 1} ^ {P} \mathbf {x} ^ {\mu + 1} \operatorname {s o f t m a x} \left(\beta \left(\mathbf {x} ^ {\mu}\right) ^ {\top} \mathbf {q}\right) \tag {5}
$$
where  $\beta$  is the temperature parameter that controls the separation strength of the MCAHN. Note that these two models can be respectively viewed as sequential versions of the Modern Hopfield Network [5] and the Modern Continuous Hopfield Network [6], which is closely related to the self-attention mechanism in Transformers [41]. However, the family of AHNs has not yet been investigated in sequential memory tasks with structured and complex inputs such as natural movies.
Other Models for Sequential Memory Beyond Hopfield Networks, many other computational models have been proposed to study the mechanism underlying sequential memory. Theoretical properties of self-organizing networks in sequential memory were discussed as early as in [42]. In theoretical neuroscience, models by Jensen et al. [43] and Mehta et al. [44] suggested that the hippocampus performs sequential memory via neuron firing chains. Other models have suggested the role of contextual representation in sequential memory [11, 45], with contextual representations successfully reproducing the recency and contiguity effects in free recall [46]. Furthermore, Howard et al. [47] proposed that sequential memory is represented in the brain via approximating the inverse Laplacian transform of the current sensory input. However, these models were still at the conceptual level, lacking neural implementations of the computations. Recurrent networks with backpropagation and large spiking neural networks also demonstrate sequential memory [48, 49]. We compare our model with [48] to validate tPC's alignment with behavior.
Our model is also closely related to the concept of cognitive map in the hippocampal formation [50-52], which is often discussed within the context of sequence learning to explain knowledge abstraction and generalization. In this work, we present two preliminary results related to cognitive maps, showing that our tPC model can 1) disambiguate aliased observation via latent representations and 2) generalize with simple sequential dynamics as a result of performing sequential memory [52]. However, as this work centers on memory, we leave cognitive maps for future explorations of tPC.
In this section, we introduce the tPC models by describing their computations during memorization and recall respectively, as well as the neural implementations of these computations. We describe the single-layer tPC first, and then move to the 2-layer tPC.
3.1 Single-layer tPC
**Memorization** The most straightforward intuition behind "good" sequential memory models is that they should learn the transition between every pair of consecutive patterns in the sequence, so that accurate recall of the full sequence can be achieved recursively by recalling the next pattern based on the current one. Given a sequence of real-valued patterns  $\mathbf{x}^{\mu}$ ,  $\mu = 1, \dots, P + 1$ , this intuition can be formalized as the model minimizing the following loss at each time-step:
$$
\mathcal {F} _ {\mu} (\mathbf {W}) = \left\| \mathbf {x} ^ {\mu} - \mathbf {W} f \left(\mathbf {x} ^ {\mu - 1}\right) \right\| _ {2} ^ {2} \tag {6}
$$
which is simply the squared temporal prediction error.  $\mathbf{W}$  is the weight parameter of the model, and  $f(\cdot)$  is a nonlinear function. Similar to static PC models [8, 10], we assume that the model has two populations of neurons: value neurons that are loaded with the inputs  $\mathbf{x}^{\mu}$  at step  $\mu$ , and error neurons representing the temporal prediction error  $\varepsilon^{\mu} := \mathbf{x}^{\mu} - \mathbf{W} f(\mathbf{x}^{\mu - 1})$ . To memorize the sequence, the weight parameter  $\mathbf{W}$  is updated at each step following gradient descent:
$$
\Delta \mathbf {W} \propto - \partial \mathcal {F} _ {\mu} (\mathbf {W}) / \partial \mathbf {W} = \varepsilon^ {\mu} f \left(\mathbf {x} ^ {\mu - 1}\right) ^ {\top} \tag {7}
$$
and the model can be presented with the sequence for multiple epochs until  $\mathbf{W}$  converges. Note that the model only has one weight parameter for the whole sequence, rather than  $P$  weight parameters for a sequence of length  $P + 1$ .
Recall During recall, the weight matrix  $\mathbf{W}$  is fixed to the learned values, and the value neurons no longer receive the correct patterns  $\mathbf{x}^{\mu}$ . Instead, while trying to recall the pattern  $\mathbf{x}^{\mu}$  based on the query  $\mathbf{q}$ , the value neurons are updated to minimize the squared temporal prediction error based on the query  $\mathbf{q}$  and the learned  $\mathbf{W}$ :
$$
\mathcal {F} _ {\mu} \left(\hat {\mathbf {x}} ^ {\mu}\right) = \left\| \hat {\mathbf {x}} ^ {\mu} - \mathbf {W} f (\mathbf {q}) \right\| _ {2} ^ {2} \tag {8}
$$
where we denote the value neurons' activities during recall as  $\hat{\mathbf{x}}^{\mu}$  to differentiate it from the memorized patterns  $\mathbf{x}^{\mu}$ . The value neurons then perform the following inferential dynamics to minimize the loss  $\mathcal{F}_{\mu}(\hat{\mathbf{x}}^{\mu})$ :
$$
\dot {\hat {\mathbf {x}}} ^ {\mu} \propto - \partial \mathcal {F} _ {\mu} (\hat {\mathbf {x}} ^ {\mu}) / \partial \hat {\mathbf {x}} ^ {\mu} = - \varepsilon^ {\mu} \tag {9}
$$
and the converged  $\hat{\mathbf{x}}^{\mu}$  is the final retrieval. Note that the error neurons' activities during recall are defined as  $\varepsilon^{\mu} \coloneqq \hat{\mathbf{x}}^{\mu} - \mathbf{W}f(\mathbf{q})$ , which is also different from their activities during memorization.
In the case of sequential memory, there are two types of recall. We define the first type as "online" recall, where the query  $\mathbf{q}$  at each step  $\mu$  is the ground-truth pattern at the previous step  $\mathbf{x}^{\mu -1}$ . It is called online as these ground-truth queries can be viewed as real-time online feedback during the sequence recall. In this case, the original  $\mathbf{x}^{\mu}$  will define a memory attractor as it defines an optimum of the loss, or energy in Eqs. 6 and 8. The second type is referred to as "offline" recall, where  $\mathbf{q}$  is the recall from the previous step i.e.,  $\hat{\mathbf{x}}^{\mu -1}$ , except at the first step, where a ground-truth  $\mathbf{x}^{1}$  is supplied to elicit recall of the whole sequence. This is called offline as there is no real-time feedback. In this case, errors from earlier steps may accumulate through time and  $\mathbf{x}^{\mu}$  is no longer an ascertained attractor unless  $\hat{\mathbf{x}}^{\mu -1} = \mathbf{x}^{\mu -1}$ , which makes it more challenging and analogous to the replay of memories during sleep [53].
Neural implementation A possible neural network implementation of these computations is shown in Fig. 1A, which is similar to that of static PC models [18] characterized by separate populations of value and error neurons. The difference from static models is that the predictions are now from the previous time-step  $\mu - 1$ . To achieve this, we assume that the value neurons are connected to the error neurons via two pathways: the direct pathway (the straight arrows between value and error neurons) and the indirect pathway through an additional population of inhibitory interneurons, which provides inhibitory inputs to the error neurons via  $\mathbf{W}$ . These interneurons naturally introduce a synaptic delay of one time-step, such that when the inputs from step  $\mu - 1$  reach the error neurons through the indirect pathway, the error neurons are already receiving inputs from step  $\mu$  via the direct pathway, resulting in the temporal error. Moreover, we assume that memory recall is a much faster process than the time-steps  $\mu$  so that the interneurons can hold a short working memory of  $\mathbf{q}$  during the (iterative) inferential dynamics in Eq. 9 at step  $\mu$ , which can be achieved by the mechanisms described in [54]. Notice that in this implementation, the learning rule (Eq. 7) is Hebbian and the inference rule (Eq. 9) is also local, inheriting the plausibility of static PC implementations [18].
3.2 2-layer tPC
Similar to multi-layer static PC models for memory, we can have multiple layers in tPC to model the hierarchical processing of raw sensory inputs by the neocortex, before they enter the memory system [8, 10]. We focus on a 2-layer tPC model in this work. In this model, we assume a set of hidden value neurons  $\mathbf{z}^{\mu}$  to model the brain's internal neural responses to the sequential sensory inputs  $\mathbf{x}^{\mu}$ . The hidden neurons make not only hierarchical, top-down predictions of the current activities in the sensory layer like in static PC models [8], but also temporal predictions like in the single-layer tPC. We also assume that the sensory layer  $\mathbf{x}^{\mu}$  does not make any temporal predictions in this case. Thus, this 2-layer tPC can be viewed as an instantiation of the hidden Markov model [55].
Figure 1: Neural implementations of the tPC models. A: single-layer tPC. B: 2-layer tPC.
Memorization During memorization, the 2-layer tPC tries to minimize the sum of squared errors at step  $\mu$ , with respect to the model parameters and the hidden activities:
$$
\mathcal {F} _ {\mu} \left(\mathbf {z} ^ {\mu}, \mathbf {W} _ {H}, \mathbf {W} _ {F}\right) = \left\| \mathbf {z} ^ {\mu} - \mathbf {W} _ {H} f \left(\hat {\mathbf {z}} ^ {\mu - 1}\right) \right\| _ {2} ^ {2} + \left\| \mathbf {x} ^ {\mu} - \mathbf {W} _ {F} f \left(\mathbf {z} ^ {\mu}\right) \right\| _ {2} ^ {2} \tag {10}
$$
where  $\mathbf{W}_H$  governs the temporal prediction in the hidden state,  $\mathbf{W}_F$  is the forward weight for the top-down predictions, and  $\hat{\mathbf{z}}^{\mu -1}$  is the hidden state inferred at the previous time-step. During memorization, the 2-layer tPC follows a similar optimization processing to that of static hierarchical PC [8]. It first infers the hidden representation of the current sensory input  $\mathbf{x}^{\mu}$  by:
$$
\dot {\mathbf {z}} ^ {\mu} \propto - \partial \mathcal {F} _ {\mu} \left(\mathbf {z} ^ {\mu}, \mathbf {W} _ {H}, \mathbf {W} _ {F}\right) / \partial \mathbf {z} ^ {\mu} = - \boldsymbol {\varepsilon} ^ {\mathbf {z}, \mu} + f ^ {\prime} \left(\mathbf {z} ^ {\mu}\right) \odot \mathbf {W} _ {F} ^ {\top} \boldsymbol {\varepsilon} ^ {\mathbf {x}, \mu} \tag {11}
$$
where  $\odot$  denotes the element-wise product between two vectors, and  $\varepsilon^{\mathbf{z},\mu}$  and  $\varepsilon^{\mathbf{x},\mu}$  are defined as the hidden temporal prediction error  $\mathbf{z}^{\mu} - \mathbf{W}_Hf(\hat{\mathbf{z}}^{\mu -1})$  and the top-down error  $\mathbf{x}^{\mu} - \mathbf{W}_{F}f(\mathbf{z}^{\mu})$  respectively. After  $\mathbf{z}^{\mu}$  converges,  $\mathbf{W}_H$  and  $\mathbf{W}_F$  are updated following gradient descent on  $\mathcal{F}_{\mu}$ :
$$
\Delta \mathbf {W} _ {H} \propto - \partial \mathcal {F} _ {\mu} (\mathbf {z} ^ {\mu}, \mathbf {W} _ {H}, \mathbf {W} _ {F}) / \partial \mathbf {W} _ {H} = \pmb {\varepsilon} ^ {\mathbf {z}, \mu} f (\hat {\mathbf {z}} ^ {\mu - 1}) ^ {\top};
$$
$$
\Delta \mathbf {W} _ {F} \propto - \partial \mathcal {F} _ {\mu} \left(\mathbf {z} ^ {\mu}, \mathbf {W} _ {H}, \mathbf {W} _ {F}\right) / \partial \mathbf {W} _ {F} = \boldsymbol {\varepsilon} ^ {\mathbf {x}, \mu} f \left(\mathbf {z} ^ {\mu}\right) ^ {\top} \tag {12}
$$
which are performed once for every presentation of the full sequence. The converged  $\mathbf{z}^{\mu}$  is then used as  $\hat{\mathbf{z}}^{\mu}$  for the memorization at time-step  $\mu + 1$ .
Recall After learning/memorization,  $\mathbf{W}_H$  and  $\mathbf{W}_F$  are fixed. We also assume that the hidden activities  $\mathbf{z}^{\mu}$  are unable to store memories, by resetting their values to randomly initialized ones. Thus, the sequential memories can only be recalled through the weights  $\mathbf{W}_H$  and  $\mathbf{W}_F$ . Again, the sensory layer has no access to the correct patterns during recall and thus needs to dynamically change its value to retrieve the memories. The loss thus becomes:
$$
\mathcal {F} _ {\mu} \left(\mathbf {z} ^ {\mu}, \hat {\mathbf {x}} ^ {\mu}\right) = \left\| \mathbf {z} ^ {\mu} - \mathbf {W} _ {H} f \left(\hat {\mathbf {z}} ^ {\mu - 1}\right) \right\| _ {2} ^ {2} + \left\| \hat {\mathbf {x}} ^ {\mu} - \mathbf {W} _ {F} f \left(\mathbf {z} ^ {\mu}\right) \right\| _ {2} ^ {2} \tag {13}
$$
where  $\hat{\mathbf{x}}^{\mu}$  denotes the activities of value neurons in the sensory layer during recall. Both the hidden and sensory value neurons are updated to minimize the loss. The hidden neurons will follow similar dynamics specified in Eq. 11, with the top-down error  $\varepsilon^{\mathbf{x},\mu}$  defined as  $\hat{\mathbf{x}}^{\mu} - \mathbf{W}_{F}f(\mathbf{z}^{\mu})$ , whereas the sensory neurons are updated according to:
$$
\dot {\hat {\mathbf {x}}} ^ {\mu} \propto - \partial \mathcal {F} _ {\mu} \left(\mathbf {z} ^ {\mu}, \hat {\mathbf {x}} ^ {\mu}\right) / \partial \hat {\mathbf {x}} ^ {\mu} = - \boldsymbol {\varepsilon} ^ {\mathbf {x}, \mu} \tag {14}
$$
and the converged  $\hat{\mathbf{x}}^{\mu}$  is the final retrieval. Similar to the single-layer case, if the converged  $\hat{\mathbf{z}}^{\mu}$  is used for the recall at the next step directly, the recall is offline; on the other hand, if we query the model with  $\mathbf{q} = \mathbf{x}^{\mu}$  i.e., the ground-truth and use the query to infer  $\hat{\mathbf{z}}^{\mu}$ , and then use  $\hat{\mathbf{z}}^{\mu}$  for the recall at the next step, the recall is online.
Neural implementation A neural implementation of the computations above is shown in Fig. 1B. The hidden layer follows the same mechanism in the single-layer tPC, with interneurons introducing a synaptic delay for temporal errors and short-term memory to perform the inferential dynamics (Eq. 11). The connection between the hidden layer and the sensory layer  $\mathbf{W}_F$  is modeled in the same way as in static PC models, which requires only Hebbian learning and local computations [18]. The memorization and recall pseudocode for the tPC models is provided in SM.
A
C
Figure 2: Comparison between single-layer tPC and AHNs. A: Capacity of models with uncorrelated binary patterns. B: Capacity of models with binary patterns with increasing feature correlations. C: Recall performance with sequences of binary MNIST digits.