SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.1 Theoretical relationship to AHNs
We first develop a theoretical understanding of single-layer tPC by relating it to AHNs:
Property 1 Assume, without loss of generality, a sequence of memories  $\mathbf{x}^{\mu} \in \mathbb{R}^{N}$  ( $\mu = 1, \dots, P + 1$ ) with zero mean. With an identity nonlinear function  $f(x) = x$  in Eq. 6, the retrieval of the single-layer tPC with query  $\mathbf{q}$ , defined as  $R^{tPC}(\mathbf{q})$ , can be written as:
$$
R ^ {t P C} (\mathbf {q}) = \sum_ {\mu = 1} ^ {P} \mathbf {x} ^ {\mu + 1} \left(\mathbf {M} \mathbf {x} ^ {\mu}\right) ^ {\top} \mathbf {M} \mathbf {q} \tag {15}
$$
where  $\mathbf{M}$  is an empirical whitening matrix such that:
$$
\langle \mathbf {M} \mathbf {x} ^ {\mu} \left(\mathbf {M} \mathbf {x} ^ {\mu}\right) ^ {\top} \rangle_ {\mu} = \mathbf {I} _ {N} \tag {16}
$$
where  $\langle \cdot \rangle_{\mu}$  is the expectation operation over  $\mathbf{x}^{\mu}s$
Proof of this property is provided in the SM. Essentially, this property implies that our single-layer tPC, in its linear form, can be regarded as a special case of the UHN for sequential memories (Eq. 3), with a "whitened dot product" similarity function where the two vectors  $\mathbf{x}^{\mu}$  and  $\mathbf{q}$  are first normalized and decorrelated (to have identity covariance  $\mathbf{I}_N$ ) before the dot product. AHNs, on the other hand, calculate the dot product directly. Biologically, this property provides a possible mechanism of statistical whitening in the brain that, unlike earlier models of biological whitening with explicit objectives [30-32], performs this computation implicitly via the circuit shown in Fig. 1 that minimizes the temporal prediction errors.
4.2 Experimental comparison to AHNs
To understand how the whitening step affects the performance in sequential memory tasks, we compare our single-layer tPC with the family of AHNs experimentally. To ensure consistency to Property 1 above, we use an identity nonlinearity  $f(x) = x$  for all these experiments. Empirically, we found that using a tanh nonlinearity makes subtle differences irrelevant to our main discussion in this work, and we discuss it in SM.
Polynomial AHNs We first compare tPC with polynomial AHNs (Eq. 4) in sequences of uncorrelated binary patterns, where AHNs are known to work well [14, 15]. We plot their sequence capacity
Figure 3: A: Recall MSE of MNIST sequences with increasing length; B: Recall MSE of MovingMNIST sequences of a fixed length 10 but with an increasing number of sequences. Error bars obtained with 5 seeds
$P_{max}$  against the number of value neurons of the models i.e., the pattern dimension  $N$ . Here,  $P_{max}$  is defined as the maximum length of a memorized sequence, for which the probability of incorrectly recalled bits is less than or equal to 0.01. Fig. 2A shows that the capacity of our single-layer tPC is greater than that of the original AHN ( $d = 1$ ) but smaller than that of a quadratic AHN ( $d = 2$ ). Notice that the single-layer tPC has an identity sep function like the original AHN. Therefore the whitening operation has indeed improved the performance of the dot product sim function. Inspired by the decorrelation effect of statistical whitening, we then generated binary patterns with  $N = 100$  correlated features, with a parameter  $b$  controlling the level of correlation ( $b = |\sqrt{\mathrm{correlation}}|$ ). The approach that we followed to generate the correlated patterns is provided in SM. As shown in Fig. 2B, as the correlation increases, all AHNs up to  $d = 3$  suffer from a quick decrease of capacity  $P_{max}$ , whereas the capacity of single-layer tPC almost remains constant. This observation is consistent with the theoretical property that the whitening transformation essentially decorrelates features such that patterns with any level of correlation are regarded as uncorrelated in tPC recall (Eq. 15). This result also explains the comparison in Fig. 2A: although the patterns generated in this panel are theoretically uncorrelated, the small correlation introduced due to experimental randomness will result in the performance gap between AHN and single-layer tPC.
We then investigate the performance of these models with sequences of binarized MNIST images [56] in Fig. 2C. It can be seen that the AHNs with  $d = 1$  and  $d = 2$  quickly fail as the sequence length  $P$  reaches 3, whereas the single-layer tPC performs well. These results suggest that our tPC with a whitening step is superior to simple AHNs due to the inherently correlated structure of natural inputs such as handwritten digits. For all the experiments with binary patterns, we used the online recall mentioned above, and polynomial AHNs already fail in this simpler recall scenario.
MCAHN Due to the quick failure of AHNs with a polynomial sep function, we now compare our single-layer tPC with the MCAHN (Eq. 5). In static memory tasks, it is known that the softmax separation leads to exponentially high capacity, especially when  $\beta$  is high [7, 57]. In this work we use  $\beta = 5$  for all MCAHNs. We first compare the performances of our single-layer tPC model and an MCAHN on random sequences of MNIST digits with varying lengths. Here we trigger recalls with online queries. Fig. 3A shows that the performance of our single-layer tPC, measured as the mean squared error (MSE) between the recalled sequence and the ground truth, is better than that of the MCAHN, further demonstrating the usefulness of the implicit whitening in our model.
Despite the superior performance of our model in this task, we note that random sequences of MNIST images are not naturally sequential inputs i.e., there are no sequential dynamics underlying them. We thus examine the models on the MovingMNIST dataset [58]. Each video in this dataset consists of 20 frames of 2 MNIST digits moving inside a  $64 \times 64$  patch. Due to the fixed sequence length, for experiments with MovingMNIST we vary the total number of sequences to memorize and fix the sequence length to the first 10 frames of the videos. The performance of the models is shown in Fig. 3B. On average, the recall MSE of MCAHN has a slower increase than that of the single-layer tPC as the total number of sequences increases. However, the performance of MCAHN has very large variations across all sequence numbers. To probe into this observation, we visually examined 3 examples of the MovingMNIST movies recalled by MCAHN and our single-layer tPC in Fig. 4A, when the total number of sequences to memorize is 40. MCAHN produces very sharp recalls for the first 2 example sequences, but totally fails to recall the third one by converging to a different memory sequence after the red triangle in Fig. 4A. On the other hand, the recall by single-layer tPC is less
Figure 4: Visual results of offline memory recall with 3 datasets. A: MovingMNIST. B: CIFAR10. C: UCF101.
sharp but stably produces the correct sequence. This phenomenon can be understood using the UHN framework [7]: in Eq. 5, when  $\beta$  is large, the softmax separation function used by MCAHN will assign a weight close to 1 to the memory whose preceding frame is most similar to  $\mathbf{q}$  (measure by dot product), and weights close to 0 to all other memories, which results in the sharp recall. In contrast, our single-layer tPC model uses an identity separation function that fails to suppress the incorrect memories, resulting in blurry recalls. Importantly, however, the failure of MCAHN in sequence 3 in Fig. 4A suggests that there are "strong attractors" in the memories with an undesirable advantage in dot product similarity which resulted in the large variance in the numerical results, and is addressed by the whitening step in single-layer tPC as it effectively normalized the patterns before dot product.
The importance of whitening in tPC is further demonstrated in our experiments with random sequences of CIFAR10 [59] images and movies from the UCF101 [60] dataset, shown in Figs. 4B and C. When recalling these colored sequences, MCAHN can easily converge to strong attractors preceded by frames with many large pixel values that lead to large dot products e.g., the third image in the CIFAR10 example with bright backgrounds, and the penultimate frame in the UCF101 example with a large proportion of sand, which give their subsequent frames large similarity scores. This problem is consistent with earlier findings with static memories [8], and is circumvented in our single-layer tPC model with the whitening matrix  $\mathbf{M}$  normalizing the pixel values across the sequence, yielding the correct and more stable memory recalls. However, they are less sharp due to the identity separation function. For all the experiments in Fig. 3B and Fig. 4, we used offline recall to make the tasks more challenging and more consistent with reality. Results with online recalls are shown in SM.
4.3 Comparison to behavioral data in sequential memory experiments
We further demonstrated the biological relevance of tPC by comparing it to data from behavioral experiments in sequential memory. In Fig 5A, our 2-layer tPC is compared with Crannell and Parrish's [61] study on sequence length's impact on serial recall of English words. Using one-hot vectors to represent letters (i.e., each "word" in our experiment is an ordered combination of one-hot vectors of "letters", a minimal example of a word with 3 letters being:  $[0,1,0]$ ,  $[1,0,0]$ ,  $[0,0,1]$ ), we demonstrate accuracy as the proportion of perfectly recalled sequences across varying lengths. Our model aligns consistently with experimental data in [61] as well as the recurrent network model by Botvinick and Plaut [48], displaying a sigmoidal accuracy drop with increasing sequence length.
Fig 5B introduces a qualitative comparison to Henson's [62] experimental data, examining primacy/recency effects in serial recall of letters. These effects involve higher accuracy in recalling early (primacy) and late (recency) entries in a sequence, with the recency effect slightly weaker than the primacy effect. Using one-hot vectors and a fixed sequence length 7 (6 positions are shown as the first position is given as the cue to initiate recall in our experiment), we visualize recall frequency at different positions across simulated sequences (100 repetitions, multiple seeds for error bars).
Figure 5: Replicating behavioral data with tPC. A: Experimental data from [61] that studies the impact of sequence length on serial recall of English words and the replications by Botvinick and Plaut [48] and tPC. B: tPC replicates the primacy/recency effects in serial recall experiments [62].
Figure 6: Context representation and generalization of tPC. A: A simple aliased example with MNIST. B: Numerical investigation into the impact of aliased or repeating elements on model performance. C: Different latent representations of aliased inputs by the 2-layer tPC. D/E: Recall/generalization of sequences with rotational dynamics. "GT" stands for ground truth and 16 and 1024 are the numbers of training sequences. F: Recall and generalization MSE of seen and unseen rotating MNIST images. Error bars obtained with 5 seeds.
Each bar in Fig 5B indicates the frequency of an entry at a particular position being recalled at each position. Our 2-layer tPC reproduces primacy/recency effects, albeit weaker than the model in [62] and previous models [48]. Additionally, the model tends to recall neighboring entries upon errors, echoing Henson's data. We attribute the weaker effects to tPC's memory storage in weights, leading to overall improved performance across positions. Details for these experiments are shown in SM.
4.4 tPC develops context-dependent representations
In Fig. 3, we plotted the recall MSE of the 2-layer tPC model in MovingMNIST, which is similar to that of the single-layer tPC. The close performance of these models raises the question of whether and when the hidden layer and hierarchical processing of sequential inputs are necessary. Inspired by earlier neuroscience theories that the hippocampus develops neuron populations signaling the sensory inputs and the context of the inputs separately [3, 11], we hypothesize that the hidden neurons in our model represent when an element occurs in a memory sequence i.e., its context. We thus designed a sequential memory task with aliased, or repeating inputs at different time-steps [28, 63]. An example of such an aliased sequence can be seen in Fig. 6A, where the second and the fourth frames of a short MNIST sequence are exactly the same ("2"). Recalling such a sequence is inherently more challenging as the models have to determine, when queried with the "2" during recall (either online or offline), whether they are at the second or the fourth step to give the correct recall at the next step ("1" or "3"). As can be seen in Fig. 6A, both MCAHN (which can be regarded as a single-layer network [7]) and single-layer tPC fail in this task, recalling an average frame of "1" and "3" after the
aliased steps, whereas the 2-layer tPC can recall the correct sequence. We then conducted a numerical investigation into this problem. We first plotted the (online) recall MSEs of the single- and 2-layer tPC models in random MNIST sequences (sampled from the training set) of varying lengths, which are shown as the solid lines in Fig. 6B. We then randomly replaced  $20\%$  (rounded up to the closest integer) of the elements in these sequences with a single digit from the test set of MNIST so that each sequence now has  $20\%$  repeating i.e., aliased elements, and plotted the recall MSEs as the dotted lines in Fig. 6B. The result suggests that sequences with aliased inputs affect the single-layer model much more than it affects the 2-layer one, producing significantly larger MSEs than recalls without aliased inputs. It is worth noting that aliased inputs are ubiquitous in natural sequential memories. In SM, we provide a natural example from the UCF101 dataset where the 2-layer tPC successfully de-aliased repeating inputs whereas the MCAHN and single-layer tPC failed.
To further understand the mechanism underlying the 2-layer tPC with aliased memories, we used a simpler synthetic sequential memory shown in Fig. 6C bottom, where a white bar moves first down and then up in a  $5 \times 5$  frame so that the steps 2 and 4 are aliased [28]. We then trained a 2-layer tPC with a hidden size 5 to memorize this sequence and queried it offline. The smaller hidden size allows us to plot, when the recall dynamics (Eq. 11) have converged, the exact hidden activities in Fig. 6C top, where each vertical line represents the activity of a hidden value neuron  $\hat{\mathbf{z}}^{\mu}$ . As can be seen in the circled time-steps, the 2-layer model represents the aliased inputs differentially in its hidden states, which helps it recall the next frame correctly. This property is consistent with early observations in neuroscience that when memorizing sequences, the hippocampus develops a conjunction of neurons representing individual inputs, as well as neurons signaling the temporal context within which an individual appears [3, 11]. In our simple 2-layer model, the "sensory" layer represents individual inputs, whereas the hidden layer plays the role of indexing time and context.
4.5 tPC generalizes learned dynamics to unseen sequences
After memorizing a large number of training sequences sharing underlying dynamics, can tPC generalize the dynamics to unseen sequences? In this experiment, we train the 2-layer tPC with sequences of rotating MNIST digits of identical rotating dynamics (i.e., the digits are rotating towards a fixed direction with a fixed angle at each time step) and vary the number of training sequences ("training size"). An example of these rotating MNIST digits can be seen in Fig 6D, row "ground truth". The model's performance is assessed by its ability to rotate both seen MNIST digits and unseen EMNIST letters. For small training sizes (16), tPC can recall seen rotating digits but struggles with generalizing to unseen letters (Fig 6D and E, second row). Increasing the training size to 1024 improves generalization, evident in clearer rotating sequences (Fig 6D and E, bottom row). Panel F quantitatively confirms this trend: the generalization MSE on unseen EMNIST drops as MNIST training size increases, indicating the model learns the underlying dynamics. Interestingly, the recall MSEs for seen MNIST sequences also decrease due to the model extracting rotational dynamics from the larger training set, differing from the behavior observed in random MNIST sequences (Fig 3A). Generalization and the capability of developing contextual representations that disambiguate aliased inputs are two critical functions underlying the flexible behavior of animals, thus connecting our tPC to models of cognitive maps in the hippocampus and related brain regions [51].
Inspired by experimental and theoretical discoveries in neuroscience, in this work we have proposed a temporal predictive coding model for sequential memory tasks. We have shown that our tPC model can memorize and recall sequential inputs such as natural movies, and performs more stably than earlier models based on Asymmetric Hopfield Networks. We have also provided a theoretical understanding of the stable performance of the tPC model, showing that it is achieved by an additional statistical whitening operation that is missing in AHNs. Importantly, this whitening step is achieved implicitly by a plausible neural circuit performing local error minimization. Moreover, our 2-layer tPC has exhibited representational and behavioral properties consistent with biological observations, including contextual representations and generalization. Overall, our model has not only provided a possible neural mechanism underlying sequential memory in the brain but also suggested a close relationship between PC and HN, two influential computational models of biological memories. Future directions include systematic investigations into tPC with more than 2 layers and modelling cognitive maps with tPC.
This work has been supported by Medical Research Council UK grant MC_UU_00003/1 to RB, an E.P. Abraham Scholarship in the Chemical, Biological/Life and Medical Sciences to MT, and UKRI Future Leaders Fellowship to HB (MR/W008939/1). The authors would like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work. http://dx.doi.org/10.5281/zenodo.22558
[59] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. online: http://www.cs.toronto.edu/kriz/cifar.html, 55(5), 2014.
[60] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
[61] CW Crannell and JM Parrish. A comparison of immediate memory span for digits, letters, and words. The Journal of Psychology, 44(2):319-327, 1957.
[62] Richard NA Henson. Short-term memory for serial order: The start-end model. Cognitive psychology, 36(2):73-137, 1998.
[63] Steven D Whitehead and Dana H Ballard. Learning to perceive and act by trial and error. Machine Learning, 7:45-83, 1991.