SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Disentangled representation of ordinal structure facilitates transfer learning. (A-B) Illustrating a transfer learning task, in which new temporal sequences have the same ordinal structure as the template. (A) From left to right. Four new sequences are constructed by using three primitive items (phonemes), and they have the same tree-structured orders as the template stored in the RNN. By freezing the recurrent connections in the RNN, the model learns the same tree-structured representation trajectories as the old ones, and their discrepancy is negligible after training (for the measurement of the discrepancy, see Sec. C in SI). (B) The learning of the model is accelerated significantly compared to the case of no transfer learning. (C) A schematic diagram of a transfer learning task, in which the ordinal structure of new temporal sequences only covers part of the template. (D) The learning of the model in (C) is accelerated significantly compared to the case of no transfer learning. For the details of model parameters and performances, see Sec. C, F in SI.
RNN (note that freezing recurrent connections in the RNN does not ensure that the representation trajectories are also fixed); 2) the speed of transfer learning is accelerated significantly compared to that without freezing the RNN.
We further consider a transfer learning task, in which the ordinal structure of new temporal sequences is only part of the order template stored in the RNN (more details see Sec. C in SI). Firstly, we create a task of discriminating eight sequences, which are constructed by the synthetic primitive items given in Sec.2.3. They are:  $\mathbf{S}^1 = \{\mathbf{a},\mathbf{b},\mathbf{a}\}$ ,  $\mathbf{S}^2 = \{\mathbf{a},\mathbf{b},\mathbf{c}\}$ ,  $\mathbf{S}^3 = \{\mathbf{a},\mathbf{c},\mathbf{a}\}$ ,  $\mathbf{S}^4 = \{\mathbf{a},\mathbf{c},\mathbf{b}\}$ ,  $\mathbf{S}^5 = \{\mathbf{b},\mathbf{a},\mathbf{b}\}$ ,  $\mathbf{S}^6 = \{\mathbf{b},\mathbf{a},\mathbf{c}\}$ ,  $\mathbf{S}^7 = \{\mathbf{b},\mathbf{c},\mathbf{a}\}$ ,  $\mathbf{S}^8 = \{\mathbf{b},\mathbf{c},\mathbf{b}\}$ . We train the RNN to learn the three-layer tree structure of the temporal orders of these sequences. We then design a new task of discriminating six temporal sequences formed by the above three primitive phonemes, which are "pcl-tcl-pcl", "pcl-tcl-pau", "pcl-pau-pcl", "pcl-pau-tcl", "tcl-pau" and "tcl-pcl", and their ordinal structure only cover part of the stored template in the RNN (shown in Fig.3C, denoted by red line). Following the same transfer learning procedure as described above, we train the model to learn new sequences, and observe that 1) the model can reuse part of the stored template to represent new sequences; 2) the model speeds up the learning process significantly compared to the case of no transfer learning (Fig.3D).
In addition to the aforementioned, we also conduct an additional experiment to demonstrate that our model can also combine and reuse primitive tree attractor structures to rapidly solve longer sequence task, shown in Fig. S8. Overall, our study shows that disentangled representation of ordinal structure in recurrent neural circuits can be efficiently combined and reused to facilitate transfer learning, if new temporal sequences have the same or partial ordinal structure.
Figure 4: Robust key-word spotting with the ordinal structure. (A) The tree-structured orders of four words, 'water', 'wash', 'year', 'had'. (B) The visualization of the learned tree-structured neural representations of four temporal sequences in the reduced state space of the RNN. (C) Examples of the model discriminating the word 'water' with different warping effects ( $F = 0.6, 1, 2$ ). Notably, the time for the model recognizing the word is also stretched or compressed accordingly. (D) With respect to temporal, our model achieves robust performances; whereas, a control model which does not follow the training protocol proposed in this work (and hence does not learn the tree-structured attractor dynamics) fails to do so. For the details of model parameters and performances, see Sec. D, F in SI.
3.2 Robust temporal sequence discrimination using ordinal structure
A hallmark of temporal information processing in the brain is that the neural system can recognize temporally stretched or compressed signals, known as temporal warping, as the same pattern [20]. It remains unclear how neural circuits in the brain achieves this goal. Here, we argue that storing ordinal structure as tree-structured attractor dynamics in recurrent neural circuits provides a feasible solution.
To test this hypothesis, we apply our model to a real-world application called key-word spotting. Specifically, we design a four spoken words discrimination task, where the four words, "water", "wash", "year", and "had", are chosen from the TIMIT dataset. For each word, 10 original temporal sequences are randomly sampled from the dataset, and each sequence is chunked into two primitive items by syllables or phonemes, for example, the word "wash" is chunked into "w-ao" and "sh". Overall, the temporal orders of these four words form a two-layer tree (Fig.4A). Following the same training protocol, we first augment training data by inserting varied intervals between neighboring primitive items and construct their target functions accordingly (for details, see Sec. A in SI). We then train the model to extract the ordinal structure (see Fig.4B). In testing, we present temporal sequences of four words without inserting separation intervals, rather we stretch or compress the original temporal sequences to reflect the temporal warping effect in practice (for details, see Sec. D in SI). We use a variable  $F$  to denote the warping effect, with  $F = 1$  representing no warping,  $F > 1$  stretching, and  $F < 1$  compressing. The results are shown in Fig.4B-D. We observe that: 1) the RNN extracts the ordinal structure of four temporal sequences; 2) the model realizes word discrimination
in a coarse-to-fine manner; 3) the mode achieves robust performance over a wide range of temporal warping.
The brains is good at discovering and utilizing abstract representations, known as "schemas", from daily experiences. These schemas are suggested to be stored in the prefrontal cortex, and can be reinitialized and reused to afford efficient transfer learning to new tasks [21, 22]. However, there is still lack of deep understanding of how these schemas are represented and reused in neural circuits in the brain. In this work, we study the representation and reuse of a specific schema, the ordinal structure of temporal sequences, in neural circuits. We show that with an appropriate training protocol, a recurrent neural circuit can learn tree-structured attractor dynamics to represent the ordinal structure of temporal sequences. This abstract order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. We showcase this advantage by using a transfer learning and a key-word spotting tasks.
Our model may have some far-reaching implications to the representation of abstract schemas in the brain. Recently, researchers have conducted interesting studies on the format of neural schemas. For example, Guntupalli et al. proposed a graph model [23]. Similar to our work here, by learning new associations between new sensory features and schema representations, this graph model can utilize common abstract knowledge to facilitate transfer learning. However, their schema representations employ a high-order Markov dynamic model rather than the attractor dynamic model, which raises the question of how the brain biologically represents such schema representations. On the other hand, Goudar et al. trained a recurrent neural network model on various common sensor-motor mapping tasks and discovered a low-dimensional attractor dynamics which emerges and acts as an abstract structure to accelerate new decision-making tasks [24]. In contrast to Goudar's work, our research primarily focuses on abstract temporal order representation in the context of a real-world task involving key-word spotting. This task provides a more intricate and complicated setting for exploring abstract schema representations than what has been considered in previous studies. Overall, studying various attractor dynamics for different schema representations in the neural system would offer fascinating avenues for future research.
Although the focus of the present study is on unveiling the biological mechanism for representing the ordinal structure of temporal sequences, the computational principle we have gained can be applied to other brain-inspired computations. To test this idea, we replace the RNN in the model with two popular recurrent networks in machine learning, which are GRU and LSTM. Applying to the key-word spotting task, we observe that (Table.1): if the same training protocol proposed in this work is used, both GRU and LSTM achieve good performances; while if the conventional training protocol (i.e., no data augmentation and using the cross-entropy loss function) is used, they can still discriminate original temporal sequences but fail to be robust to temporal warping. These results suggest that our method (including the model structure and the training protocol) is general and has potential to be applied to brain-inspired computing.
Table 1: Model performances using different recurrent networks and different training protocols in a key-word spotting task. We replace the RNN in our model by GRU or LSTM. We train the model following either the same training protocol used in this work (marked by*) or the conventional protocol, i.e., no data augmentation and the cross-entropy loss function. Both models are trained on the dataset without warping  $(F = 1)$  and tested on a dataset with warping  $(F = 0.6, 1.0, 1.4, 1.8, 2.2, 2.6, 3.0)$ . The classification accuracy of each model is obtained by averaging over 8 trials with randomly split training and testing data, as well as network initialization. For each task, we have 10 samples per class in the train dataset and 300 samples per class in the test dataset. The word sequences are sampled from the TIMIT dataset. For more details, see Sec. D, F in SI.
Model	4 classes	8 classes	12 classes
no warping	warping	no warping	warping	no warping	warping
GRU	94.0% ± 2.5	88.9% ± 6.2	95.1% ± 2.5	87.9% ± 7.6	94.3% ± 1.2	84.4% ± 9.0
GRU*	97.4% ± 1.5	96.6% ± 1.8	97.2% ± 0.6	95.6% ± 1.8	96.4% ± 0.9	94.8% ± 1.8
LSTM	94.1% ± 2.2	82.8% ± 13.7	93.8% ± 1.6	80.5% ± 12.3	92.1% ± 0.7	75.9% ± 14.8
LSTM*	95.1% ± 1.8	92.5% ± 3.7	94.0% ± 0.8	90.1% ± 3.5	92.6% ± 2.1	89.5% ± 3.3
In order to learn a tree-like structure in our model, we require precise ordinal details about continuous sequences. Several mechanisms can be employed in our brain to access this kind of information. Firstly, a large volume of experimental studies has shown that in the brain, continuous sequences are often chunked into discrete items to support high-level cognition [25]. For examples, speech sequences can be hierarchically chunked into words and syllables [1]; neurons in the hippocampus have been shown to detect event boundaries when watching continuous movie videos [26]. This chunking/segmentation process naturally outputs the ordinal structure of a temporal sequence. Additionally, computational models have been proposed for the brain performing sequence chunking, such as self-supervised learning [27] and oscillation [28]. Secondly, the development path of our brain also indicates the brain can naturally acquire sequence chunking. For example, in language acquisition, young children learn primitive phoneme categories around 2 months [29]; around 7 months, they learn sequences of either an ABB or ABA form [30], which can be represented as tree-like attractor structures in our model; subsequently, children learn to recognize spoken words around 12 months. Thus, in language acquisition, phoneme learning serves as a building block for word learning, which defines the ordinal structure of language sequences.
Incorporating a wide range of diverse temporal intervals presents another challenge for our model. How does the neural system handle this in a biologically plausible manner? Firstly, in our brain, motor and speech sequences are often generated with a large variability in speed, and hence they exhibit a large variability in separations between motor motifs and speech chunks. This large variability in separations enables the brain to learn the tree-like attractor structure, as demonstrated in our model. Secondly, in our network training, we do not really need very large intervals. For the clean synthetical data, we can actually train the tree-structured attractors using fixed interval values, shown in Fig. S10A. For the noisy spoken words, we do need an amount of variations to achieve good performances, but the range is only about 2 times of the item length. Overall, the requirement for a wide range of temporal intervals can be largely relaxed in practice.
Finally, there are several interesting directions worth exploring further. First, our current model is currently limited to small sequences. Figuring out how to effectively expand and scale our approach for more complex scenarios, like sequences involving large classes and longer sequences, will be a crucial focus for our research. Second, in the context of transfer learning, we employ a stochastic gradient descent method to learn the binding between sensory features and abstract representations. However, this technique is not biologically plausible and efficient. Experimental studies have shown the existence of a fast time-scale Hebbian rule in the brain, such as in the hippocampus [31], which is suggested to mediate rapid binding of distinct neural populations. By incorporating this fast Hebbian rule, our model could learn new associations between contents and the ordinal structure in an one-shot manner. Third, beyond ordinal structure and contents, the timing structure in a temporal sequence is also crucial for some tasks. Recent experiments have shown that the human brain could integrate abstract order template, timing structure, and contents to generate diverse finger action sequences [6]. Thus, integrating the timing structure into our model could potentially enhance its power for processing more complicated temporal sequence processing tasks. Last, the ordinal structure also serves as the basis for some more high-level abstract structures, such as algebraic patterns in motor sequence representation and recursive syntactic structures in language representation [1, 32]. How these abstract representations are concretely represented in neural circuits is largely unknown. Our model may serve as a starting point for exploring these mysteries.
This work was supported by the National Natural Science Foundation of China (N0: T2122016, Y.Y.Mi), National Science and Technology Innovation 2030 Major Program (No. 2021ZD0203700 / 2021ZD0203705, Y.Y. Mi, No. 2021ZD0200204, Si Wu).
[21] Tse, D., Langston, R. F., Kakeyama, M., Bethus, I., Spooner, P. A., Wood, E. R., ... & Morris, R. G. (2007). Schemas and memory consolidation. Science, 316(5821), 76-82.
[22] Zhou, J., Jia, C., Montesinos-Cartagena, M., Gardner, M. P., Zong, W., & Schoenbaum, G. (2021). Evolving schema representations in orbitofrontal ensembles during learning. Nature, 590(7847), 606-611.
[23] Guntupalli, J. S., Raju, R. V., Kushagra, S., Wendelken, C., Sawyer, D., Deshpande, I., ... & George, D. (2023). Graph schemas as abstractions for transfer learning, inference, and planning. arXiv preprint arXiv:2302.07350.
[24] Goudar, V., Peysakhovich, B., Freedman, D. J., Buffalo, E. A., & Wang, X. J. (2023). Schema formation in a neural population subspace underlies learning-to-learn in flexible sensorimotor problem-solving. Nature Neuroscience, 1-12.
[25] Dehaene, S., Al Roumi, F., Lakretz, Y., Planton, S., & Sablé-Meyer, M. (2022). Symbols and mental programs: a hypothesis about human singularity. Trends in Cognitive Sciences.
[26] Hahamy, A., Dubossarsky, H., & Behrens, T. E. (2023). The human brain reactivates context-specific past information at event boundaries of naturalistic experiences. Nature neuroscience, 1-10.
[27] Asabuki, T., & Fukai, T. (2020). Somatodendritic consistency check for temporal feature segmentation. Nature communications, 11(1), 1554.
[28] Giraud, A. L., & Poeppel, D. (2012). Cortical oscillations and speech processing: emerging computational principles and operations. Nature neuroscience, 15(4), 511-517.
[29] Kuhl, P. K. (2004). Early language acquisition: cracking the speech code. Nature reviews neuroscience, 5(11), 831-843.
[30] Marcus, G. F., Vijayan, S., Bandi Rao, S., & Vishton, P. M. (1999). Rule learning by seven-month-old infants. Science, 283(5398), 77-80.
[31] Bittner, K. C., Milstein, A. D., Grienberger, C., Romani, S., & Magee, J. C. (2017). Behavioral time scale synaptic plasticity underlies CA1 place fields. Science, 357(6355), 1033-1036.
[32] Shima, K., Isoda, M., Mushiake, H., & Tanji, J. (2007). Categorization of behavioural sequences in the prefrontal cortex. Nature, 445(7125), 315-318.
A The Training Protocol
A
Figure S5: Illustration of the augmentation of temporal sequences and the setting of ramping target functions. (A) An example temporal sequence,  $a - b - a$ , in the synthetic task with  $K = 8$ , which is used to train the model shown in Fig.3C of maintext. The upper panel is augmented input stimuli by inserting duration and adding Gaussian noise, the lower panel is ramping target functions of readout neurons. (B) An example sequence, "water", in the key-word spotting task with  $K = 8$  (used in Tab.1 of maintext), which includes the following words: "water, wash, had, year, carry, oily, dark, suit". The upper panel is augmented MFCC feature vector of "water", the lower is the corresponding target functions of read-out neurons.
To train the model, we first augment training data and construct their target functions (see illustration in Fig.2A of maintext), after that we apply backpropagation-through-time (BPTT) algorithm to optimize the model parameters  $\{\mathbf{W}^{in},\mathbf{W}^{rec},\mathbf{W}^{out},\mathbf{I}^b\}$ . Denote  $\mathbf{y}(t)$  the actual model output sequence and  $\hat{\mathbf{y}}(t)$  the corresponding target function. The loss function is given by
$$
L = \frac {1}{T _ {b} B} \sum_ {b = 1} ^ {B} \sum_ {k = 1} ^ {K} \sum_ {t = 1} ^ {T _ {b}} \left[ y _ {k} ^ {b} (t) - \hat {y} _ {k} ^ {b} (t) \right] ^ {2}, \tag {3}
$$
where  $b = 1, \ldots, B$  denote the index of temporal sequences in the training data,  $T_{b}$  the length of the  $b$ th temporal sequence, and  $k = 1, \ldots, K$  the index of the output neuron. We adopt the ADAM optimizer, with the learning rate varying in the range of  $lr = [1e^{-4}, 1e^{-2}]$ , and the decay rates of the first and second moments are 0.9 and 0.999.
Inspired by the ramping activity of decision making neurons observed in the experiment [1-3], we set the target function of a read-out neuron increases with the unfold of the items in the temporal
sequence this neuron encodes. Consider the temporal sequence of the first class, denoted as  $\mathbf{S}^1$ , is presented to the network. The target functions of read-out neurons are constructed as below.
(1) The task of discriminating four synthetic temporal sequences  $(K = 4)$ , Fig.2 in the main text.
In this task,  $\mathbf{S}^1$  consists of two consecutive items  $\{S_1^1,S_2^1\}$  with the duration  $\Delta T$ , as shown in Fig.2A-B in the main text. Before the temporal sequence is presented, all read-out neurons are at the spontaneous state, whose target functions are set to be  $\hat{y}_k = O_{init}$ , for  $k = 1,\dots,K$ . When the first item  $S_1^1$  appears, both the target functions of neuron 1 and 2 increase to  $\hat{y}_{1,2} = O_1$ , and the target functions of other neurons decay to the silent state,  $\hat{y}_{3,4} = O_{sile}$ . This lasts for the duration  $\Delta T$ , until the second item appears. When  $S_2^1$  appears, the target function of neuron 1 further increases to  $\hat{y}_1 = O_2$ , while the target function of neuron 2 decreases to  $O_{sile}$ , and the target functions of other neurons keep the same, i.e.,  $\hat{y}_{2,3,4} = O_{sile}$ . In this task, the parameters used are  $O_{init} = 0.25$ ,  $O_1 = 0.5$ ,  $O_2 = 1$ ,  $O_{sile} = 0$ .
For other tasks of discriminating four temporal sequences, the target functions are designed similarly.
(2) The task of discriminating eight synthetic temporal sequences  $(K = 8)$ , Fig.3C in the main text
In this task,  $\mathbf{S}^1$  consists of three consecutive items  $\{S_1^1,S_2^1,S_3^1\}$  with the duration  $\Delta T$ , as shown in Fig.3C in the main text. The design of target functions is illustrated in Fig.S1A, similar to the task with  $K = 4$ . Before the sequence is presented, all target functions of all read-out neurons are set to be  $\hat{y}_k = O_{init}$  for  $k = 1,\dots,8$ . When the first item  $S_1^1$  appears, the target functions of neurons  $1 - 4$  increase to be  $\hat{y}_k = O_1$ , for  $k = 1,2,3,4$ , since the four neurons share the same first item; while the target functions of other neurons decay to the silent state, i.e.,  $\hat{y}_k = O_{sile}$  for  $k = 5,6,7,8$ . When  $S_2^1$  appears, the target functions of neurons  $1 - 2$  keep increasing to  $\hat{y}_k = O_2$  for  $k = 1,2$ ; while the target functions of neurons  $3 - 4$  decay to silent, and the target functions of other neurons keep silent, i.e.,  $\hat{y}_k = O_{sile}$ , for  $k = 3,\ldots ,8$ . When  $S_3^1$  appears, only the target function of neuron 1 keeps increasing to  $\hat{y}_1 = O_3$ , while all other neurons are in the silent state, i.e.,  $\hat{y}_k = O_{sile}$ , for  $k = 2,\ldots ,8$ . The parameters used are  $O_{init} = 0.25$ ,  $O_1 = 0.5$ ,  $O_2 = 0.75$ ,  $O_3 = 1$ , and  $O_{sile} = 0$ .
For other tasks of discriminating eight temporal sequences, the target functions are designed similarly.
(3) The key-word spotting task of discriminating eight or twelve temporal sequences  $(K = 8,12)$ , Table 1 in the main text.
In this task,  $\mathbf{S}^1$  denotes the temporal sequence of the key word 'water', which is chunked into two different items by syllables, i.e., 'w-ao' and 'dx-axr'. The training data is augmented according to the learning protocol described in the main text, as illustrated in Fig.S1B (upper panel). Before the sequence is presented, all read-out neurons are at the spontaneous state, i.e.,  $\hat{y}_k = O_{init}$ , for  $k = 1,\dots,K$ , with  $K = 8,12$ . When the first item ('w-ao') appears, the target functions of neurons  $1 - 2$  increase to  $O_1$ , while the target functions of other neurons decay to  $O_{sile}$ . When the second item appears, only the target function of neuron 1 keeps increasing to  $O_2$ , while the target functions of other neurons are,  $\hat{y}_k = O_{sile}$  for  $k = 2,\dots,K$ . The design of target functions for  $\mathbf{S}^1$  in 8-classes task is illustrated in Fig.S1B (lower panel). The parameters used are  $O_{init} = 0.25$ ,  $O_1 = 1$ ,  $O_2 = 2$ ,  $O_{sile} = 0$ .
B The Visualization Method
B.1 Principle Component Analysis (PCA) of neural activities in the RNN (Sec.2.4.1 in the Main text)