Title: Learning and processing the ordinal information of temporal sequences in recurrent neural circuits
Venue: NeurIPS 2023
Accepted: yes
Keywords: temporal sequence processing

temporal order structure

tree-structured attractor
Abstract: Temporal sequence processing is fundamental in brain cognitive functions. 
Experimental data has indicated that the representations of ordinal information and contents of temporal sequences are disentangled in the brain, but the neural mechanism underlying this disentanglement remains largely unclear. Here, we investigate how recurrent neural circuits learn to represent the abstract order structure of temporal sequences, and how this disentangled representation of order structure from that of contents facilitates the processing of temporal sequences. We show that with an appropriate learn protocol, a recurrent neural circuit can learn a set of tree-structured attractor states to encode the corresponding tree-structured orders of given temporal sequences. This abstract temporal order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. Using a transfer learning task, we demonstrate that the reuse of a temporal order template facilitates the acquisition of new temporal sequences of the same or similar ordinal structure. Using a key-word spotting task, we demonstrate that the attractor representation of order structure improves the robustness of temporal sequence discrimination, if the ordinal information is the key to differentiate different sequences. We hope this study gives us insights into the neural mechanism of representing the ordinal information of temporal sequences in the brain, and helps us to develop brain-inspired temporal sequence processing algorithms.

Learning and Processing the Ordinal Information of Temporal Sequences in Recurrent Neural Circuits

Abstract

Temporal sequence processing is fundamental in brain cognitive functions. Experimental data has indicated that the representations of ordinal information and contents of temporal sequences are disentangled in the brain, but the neural mechanism underlying this disentanglement remains largely unclear. Here, we investigate how recurrent neural circuits learn to represent the abstract order structure of temporal sequences, and how the disentangled representation of order structure facilitates the processing of temporal sequences. We show that with an appropriate training protocol, a recurrent neural circuit can learn tree-structured attractor dynamics to encode the corresponding tree-structured orders of temporal sequences. This abstract temporal order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. Using a transfer learning task, we demonstrate that the reuse of a temporal order template facilitates the acquisition of new temporal sequences, if these sequences share the same or partial ordinal structure. Using a key-word spotting task, we demonstrate that the tree-structured attractor dynamics improves the robustness of temporal sequence discrimination, if the ordinal information is the key to differentiate these sequences. We hope that this study gives us insights into the mechanism of representing the ordinal information of temporal sequences in the brain, and helps us to develop brain-inspired temporal sequence processing algorithms.

1 Introduction

A temporal sequence involves a set of items or events that unfold in a specific order over time. Temporal sequence processing is fundamental in many cognitive functions [1], such as speech recognition, language comprehension, motor control, and memory formation. Effective temporal sequence processing relies on extracting the temporal structure of a sequence, which includes particularly the ordinal information, i.e., the order of events' occurrence. A large volume of studies has suggested that the representations of ordinal information and contents of a temporal sequence are disentangled in the brain [1-6]. For instances, studies using single-unit recording in primates have found that a significant proportion of neurons in the dorsolateral prefrontal and intraparietal cortexes are sensitive to the temporal order of presented visual items, but not to their physical attributes [5], and similar temporal order representations were also observed in the auditory and linguistic sequence processing [2]. The imaging study has shown that the human brain can encode the phonetic order during speech processing, regardless of the phonetic contents [2]. Recent large-scale neuron recordings have revealed that the temporal order structure can be robustly encoded in a low-dimensional space formed by coordinated neuron population activities [4]. The advantage of disentangled representation of order structure is that it enables the brain to flexibly and rapidly process temporal sequences via conjunctive coding [1], i.e, the brain can easily generate various temporal sequences by combining the temporal order with different contents. This was evident in a finger sequence learning task [6], where the brain stored a temporal order template and reused it to combine actions to generate various finger action sequences. Understanding how ordinal information is acquired and stored in the brain is crucial for us to understand how temporal sequences are processed in cognitive functions.

Remarkably, the representations of temporal orders of related sequences display a tree-like structure [7]. For example, in speech recognition, spoken words are represented by temporal sequences formed by primitive syllables or phonemes. As illustrated in Fig.1A, the words "water" and "wash" share a common syllable "w-ao" at the first item and differ at the second syllable, and so do the words "your" and "year". Thus, the temporal orders of these four sequences form a two-layer tree: starting from the root, the words "water" and "wash" form a branch in the first layer as they share the same first syllable, and the words "your" and "year" form the other branch; in the second layer, the words "water" and "wash", and the words "your" and "year", are further branched due to their difference at the second syllable. Experimental data has indicated that such tree-structured representation of ordinal information is employed in the frontal cortex to support hierarchical motor control [8].

B

A

D1

D2

D3

Time

/w-ao/

/sh/

/dx-axr/

/axr/

/ih-axr/

/y/

wash

water

your

year

...

...

Input:

Output:

RNN

Despite that the representation of ordinal information in the brain has been well recognized in the field, the neural mechanism of implementing this representation remains largely unclear [1, 7]. A number of computational models has been proposed to mimic how temporal sequences are stored

in neural circuits, and they typically consider that neurons are asymmetrically connected along the path of a sequence of network states representing the temporal sequence [10-12, 15]. Among the proposed models, the framework of heteroclinic channel is inspiring [10], which considers that a temporal sequence is stored as a sequence of saddle points connected by unstable paths in the state space of the network, forming the so-called a heteroclinic channel, and under the drive of external inputs, the network state evolves along the channel to activate stored items in the right time order. In most previous works [11, 13-15], the asymmetric neuronal connections accounting for storing temporal sequences are either designed by hand or constructed by applying an asymmetric Hebbian learning rule (note that the application of a Hebbian learning rule implicitly assumes that neural representations of sequential items are known, which does not really model the learning process), and they did not address how a neural circuit learns to represent the temporal order structure from data. A few studies explored how temporal sequences are learned in a recurrent neural network [15], but they only considered the simple chain-structured temporal order and did not study the more general tree-structured orders. The machine learning society also studied the learning of temporal sequences (such as speech) by using artificial neural networks, but they have not considered the representation of disentangled ordinal information [9].

In this work, we explore how biologically plausible recurrent neural circuits learn to represent the treestructured orders of temporal sequences and how this disentangled representation of order structure facilitates the processing of temporal sequences. We show that with an appropriate training protocol, a recurrent neural circuit can learn a set of tree-structured attractor states to encode the corresponding tree-structured orders of temporal sequences, where each path from the root to a leaf represents a pattern of temporal order, with nodes on the path indicating the order hierarchy (see Fig.1A). This abstract order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. To demonstrate this advantage, we carry out two experiments. First, in a transfer learning task, we show that the reuse of the order template accelerates the acquisition of new temporal sequences, if these sequences share the same or partial ordinal structure. Second, in a key-word spotting task, we show that using the tree-structured attractor dynamics improves the robustness of temporal sequence discrimination, if the ordinal structure is the key to differentiate these sequences. We hope that this study facilitates our understanding of how temporal sequences are represented and processed in the brain, and helps us to develop brain-inspired algorithms for temporal sequence processing.

2 Learning the ordinal structure of temporal sequences

2.1 The neural circuit model

The model we consider consists of three parts (Fig.1B): an input layer which conveys the information of temporal sequences, a recurrent neural network (RNN) which stores the ordinal structure of temporal sequences, and an output layer which discriminates temporal sequences. Our idea is that by training the model to perform a temporal sequence discrimination task, the RNN learns to represent the ordinal structure of given temporal sequences in its dynamics. The details of the model are introduced below.

Denote I ( t ) = { I m ( t ) } , for m = 1 , . . . , M , with M the dimension, the activity of the input layer conveying the temporal sequence. The input layer is connected to the RNN through a feedforward connection matrix W in = { W in nm } , for m = 1 , . . . , M and n = 1 , . . . , N , with N the number of neurons in the RNN. Neurons in the RNN are recurrently connected with each other with a matrix W rec = { W rec nj } , for n, j = 1 , . . . , N , and they also receive a background input I b = { I b n } , for n = 1 , . . . , N . Denote x n ( t ) the total input received by the n th neuron in the RNN at time t and r n ( t ) the corresponding neuronal activity. The dynamics of x n ( t ) is written as,

where τ is the time constant. The nonlinear relationship between x n ( t ) and r n ( t ) is given by r n ( t ) = tanh [ x n ( t )] .

The RNN is connected to the output layer through a feedforward matrix W out = { W out kn } , for n = 1 , . . . , N and k = 1 , . . . , K , with K the number of neurons in the output layer. K is also the

number of temporal sequences, since each neuron reads out one class of sequences. Denote y k ( t ) the activity of the k th neuron in the output layer, whose dynamics is given by,

2.2 The training protocol

The details of the training protocol are presented in Supplementary Information (SI). Here, we highlight two settings which are critical for the RNN to extract the ordinal structure of temporal sequences.

Firstly, we augment training data by varying the duration between neighbouring items in a temporal sequence (see illustration in Fig.2A). Specifically, this is done by adding noisy signals of varied duration between neighbouring primitive items of the temporal sequence (see more details below). The underlying mechanism is intuitively understandable. The augmented data imposes that the model needs to realize invariant recognition with respect to the warping of a temporal sequence, which forces the RNN to extract the abstract ordinal structure of the temporal sequence. This is also biologically reasonable, as in speech recognition [2, 3], the varied speed of spoken words forces the brain to process temporal sequences relying mainly on their ordinal information.

Secondly, we set the target function, i.e., the target activity of a read-out neuron in response to the temporal sequence it encodes, to be continuous in time, and its value increases step by step as items in the temporal sequence unfold one by one (see illustration in Fig.2A). The underlying mechanism is intuitively understandable. Since not all temporal sequences are differentiable by early items, neuronal responses should be low initially, and their values increase along with the exposure of followed items to reflect that sequence recognition is a process of evidence accumulation over time. Interestingly, such ramping activity of read-out neurons agree with the experimental finding on decision-making neurons in monkeys [1-3].

After constructing input data and target functions, we apply the standard algorithm, back-propagation through time, to train the model, which optimizes the parameters ( W in , W rec , W out , I bg ) . For details, see Sec. A in SI.

2.3 A synthetic task and the model behavior

To demonstrate that our model works, we first apply it to a synthetic task of discriminating four temporal sequences ( K = 4 ). These four temporal sequences are constructed by combining two of three primitive items with varied orders (like syllables forming spoken words). As illustrated in Fig.2A, the three primitive items are 3-dimensional ( M = 3 ) continuous-time vectors within an interval ∆ t , denoted as a = [1 , 1 , 0] T , b = [1 , 0 , 1] T , and c = [0 , 1 , 1] T , respectively. The contents of four temporal sequences are expressed as S k = { S k 1 , S k 2 } , with k the index of the sequence and S k 1 , S k 2 the first and second items. Since in this study, the exact duration of a temporal sequence is not important, for clearance, we also use S k to denote a sequence having the contents { S k 1 , S k 2 } with unspecified duration. The four classes of temporal sequences are therefore expressed as, S 1 = { a , b } , S 2 = { a , c } , S 3 = { b , a } , S 4 = { b , c } , and their temporal orders form a two-layer tree structure as shown in Fig.2B. To augment the training data, we carry out two operations. Firstly, we randomly set the duration between neighboring primitive items to be ∆ T , which satisfies a uniform distribution in the range of [0 , ∆ T max ] . Secondly, we add noises in the duration, which are given by η i ( t ) = σξ i ( t ) , for i = 1 , · · · , M and t ∈ (0 , ∆ T ) , where ξ i ( t ) denotes Gaussian white noises of zero mean and unit variance, and σ the noise strength.

After training, we observe that the model learns to discriminate four temporal sequences in a coarse-tofine manner. An example of discriminating the temporal sequence S 1 is presented in Fig.2C. Initially, both activities of neurons 1 (red) and 2 (green) increase, since S 1 and S 2 are not differentiable by their first item, i.e., S 1 1 = S 2 1 ; while neurons 3 (orange) and 4 (blue) keep silent. When the second item appears, only the activity of neuron 1 keeps increasing, while the activity of neuron 2 drops, indicating that the model recognizes S 1 . The above results can be generalized to more than four temporal sequences, see Sec. A in SI for K = 8 , 12 .

2.4 Visualizing the learned tree-structured attractor dynamics

D

A

B

E

F

augmenting

C

t

We carry out intensive analysis to visualize what have been learned in the dynamics of RNN.

2.4.1 Tree-structured representation trajectories

To visualize the high-dimensional dynamics of the RNN ( N = 50 ), we apply principle component analysis (PCA) to reduce its dimensionality. PCA is done by using the activities of RNN neurons in response to 1000 testing temporal sequences (for details, see Sec. B.1 in SI). It turns out the top 3 PCs explain nearly 90% of the variance of neuronal responses (Fig.2D, upper panel), indicating that after training, the RNN performs computation in a low-dimensional subspace. By projecting RNN activities onto the top 3 PCs, we obtain the trajectories of the network state in responses to the four temporal sequences (averaged over 25 trials), and observe that they display a tree structure, agreeing with the ordinal structure of the learned four temporal sequences (Fig.2E, trajectories for different sequences are marked by different colours).

2.4.2 Tree-structured attractor states

We continue to analyze the state space of the RNN, and find that it has totally 7 stable points (attractors), which are the nodes of the tree: the root, branching points, and end points (marked by crosses in Fig.2E), indicating that primitive items are stored as attractors in the RNN. The detailed analysis is presented in Sec. B.2 in SI. Here, we sketch the main idea. We first use an optimization method to find out all fixed points of the RNN dynamics [4], which turn out to coincide with the 7 nodes of the tree. We then analyze the stability of each node. An example of stability analysis of node S 1 , 2 1 is depicted in Fig.2D (lower panel), which shows that the real parts of all eigenvalues of the Jacobian matrix are small than 1 , indicating that the node is stable with respect to noise perturbation.

2.4.3 Tree-structured attractor dynamics

We further analyze the transition dynamics of attractor states (nodes). The detailed analysis is presented in Sec. B.3 in SI. Here, we sketch the main idea. Consider node S 1 , 2 1 in the first layer of the tree (Fig.2E). We linearize the RNN dynamics around the node and compute all eigenvectors of the Jacobian matrix. The larger the eigenvalue is, the more unstable the network state along the eigenvector under noise perturbation. As shown in Fig.2F, the most two unstable directions (the length of eigenvector reflecting the eigenvalue) point to nodes S 1 2 and S 2 2 in the second layer, respectively. This implies that starting from the node S 1 , 2 1 , the network state can travel to the node S 1 2 or S 2 2 depending on the second item, but has difficulty to travel to other nodes.

In summary, the above results reveal that after training, the tree-structured attractor dynamics emerges in the RNN, which encode the ordinal structure of the learned temporal sequences. Upon receiving a temporal sequence, the network state evolves from the root to the leaf, node by node, following the order hierarchy of the temporal sequence. Notably, unlike the framework of heteroclinic channel, our model stores primitive items as stable attractors rather than saddle points, and the transition between two nodes requires the followed item to be the input drive. As demonstrated below, this attractor dynamics enables our model to discriminate temporal sequences robustly against temporal warping.

3 Processing temporal sequences with the ordinal structure

3.1 Disentangled representation of ordinal structure facilitates transfer learning

The key advantage of disentangled representation of order structure is that it allows flexible generation to new temporal sequences via conjunctive code, that is, the neural system can combine the stored order template with different contents to form different temporal sequences. In our model, the RNN learns to extract the abstract ordinal structure of temporal sequences in the form of tree-structured attractor dynamics. After training, the association between the RNN and the training data is released, and the RNN can be reused to encode new temporal sequences having the same or partial temporal structure. Mathematically, this can be done by freezing the recurrent connections in the RNN (and hence the tree-structured attractor dynamics), while only learning the feedforward connections from the input layer to the RNN and from the RNN to the output layer, when a new temporal sequence processing task arises. In such a way, it is expected that the pre-trained RNN helps the model to speed up the acquisition of new temporal sequences having the same or partial ordinal structure, achieving the so-called transfer learning.

To demonstrate the above idea, we design a transfer learning task. We consider that the RNN has extracted the ordinal structure of the synthetic data described in Sec.2.3, and we now apply the model to solve a new task of discriminating new four temporal sequences. These four temporal sequences are composed by three primitive phonemes chosen from the TIMIT dataset, which are "pcl", "tcl", and "pau" (for the details of sequence construction, see Sec. C in SI), and the four sequences are written as "pcl-tcl", "pcl-pau", "tcl-pcl" and "tcl-pau", which have the same order structure as the synthetic data (Fig.3A-B). Following the same procedure as described in Sec.2.2, we augment the training data by inserting time intervals between neighboring items, and choose their ramping target functions accordingly. During transfer learning, we freeze the recurrent connections in the RNN and only optimize the feedforward connections. As comparison, we also train the model without freezing the RNN. The results are shown in Fig.3A-B. We observe that: 1) after transfer learning, the RNN retains the tree-structured representation trajectories for the learned temporal sequences, indicating that the model adjusts the feedforward connections to match the tree-structured attractors of the

New Sequence

Ordinal Structure

pcl

tcl

tcl

pau

A

B

C

D

Template

Partial Reuse

root

a

b

c

a

c

a

a

c

c

b

b

a

b

b

pcl

pau

tcl

root

tcl

pcl

tcl

pau

pcl

pcl

pau

Test Accuracy (%)

Test Accuracy (%)

Learned Dynamics

Template Dynamics

RNN (note that freezing recurrent connections in the RNN does not ensure that the representation trajectories are also fixed); 2) the speed of transfer learning is accelerated significantly compared to that without freezing the RNN.

We further consider a transfer learning task, in which the ordinal structure of new temporal sequences is only part of the order template stored in the RNN (more details see Sec. C in SI). Firstly, we create a task of discriminating eight sequences, which are constructed by the synthetic primitive items given in Sec.2.3. They are: S 1 = { a , b , a } , S 2 = { a , b , c } , S 3 = { a , c , a } , S 4 = { a , c , b } , S 5 = { b , a , b } , S 6 = { b , a , c } , S 7 = { b , c , a } , S 8 = { b , c , b } . We train the RNN to learn the three-layer tree structure of the temporal orders of these sequences. We then design a new task of discriminating six temporal sequences formed by the above three primitive phonemes, which are "pcl-tcl-pcl", "pcl-tcl-pau", "pcl-pau-pcl", "pcl-pau-tcl", "tcl-pau" and "tcl-pcl", and their ordinal structure only cover part of the stored template in the RNN (shown in Fig.3C, denoted by red line). Following the same transfer learning procedure as described above, we train the model to learn new sequences, and observe that 1) the model can reuse part of the stored template to represent new sequences; 2) the model speeds up the learning process significantly compared to the case of no transfer learning (Fig.3D).

In addition to the aforementioned, we also conduct an additional experiment to demonstrate that our model can also combine and reuse primitive tree attractor structures to rapidly solve longer sequence task, shown in Fig. S8. Overall, our study shows that disentangled representation of ordinal structure in recurrent neural circuits can be efficiently combined and reused to facilitate transfer learning, if new temporal sequences have the same or partial ordinal structure.

wash

['w-ao','sh']

water

['w-ao','dx-axr']

had

['hv','eh-dcl']

year

['y-ih','axr']

'w-ao'

'dx-axr'

'sh'

'hv'

'eh-dcl'

'y-ih'

'axr'

B

D

A

F

Accuracy (%)

C

Outputs

F

= 0.6

F

= 2

Channels

F

= 1

3.2 Robust temporal sequence discrimination using ordinal structure

A hallmark of temporal information processing in the brain is that the neural system can recognize temporally stretched or compressed signals, known as temporal warping, as the same pattern [20]. It remains unclear how neural circuits in the brain achieves this goal. Here, we argue that storing ordinal structure as tree-structured attractor dynamics in recurrent neural circuits provides a feasible solution.

To test this hypothesis, we apply our model to a real-world application called key-word spotting. Specifically, we design a four spoken words discrimination task, where the four words, "water", "wash", "year", and "had", are chosen from the TIMIT dataset. For each word, 10 original temporal sequences are randomly sampled from the dataset, and each sequence is chunked into two primitive items by syllables or phonemes, for example, the word "wash" is chunked into "w-ao" and "sh". Overall, the temporal orders of these four words form a two-layer tree (Fig.4A). Following the same training protocol, we first augment training data by inserting varied intervals between neighboring primitive items and construct their target functions accordingly (for details, see Sec. A in SI). We then train the model to extract the ordinal structure (see Fig.4B). In testing, we present temporal sequences of four words without inserting separation intervals, rather we stretch or compress the original temporal sequences to reflect the temporal warping effect in practice (for details, see Sec. D in SI). We use a variable F to denote the warping effect, with F = 1 representing no warping, F > 1 stretching, and F < 1 compressing. The results are shown in Fig.4B-D. We observe that: 1) the RNN extracts the ordinal structure of four temporal sequences; 2) the model realizes word discrimination

in a coarse-to-fine manner; 3) the mode achieves robust performance over a wide range of temporal warping.

4 Conclusion and Discussion

The brains is good at discovering and utilizing abstract representations, known as "schemas", from daily experiences. These schemas are suggested to be stored in the prefrontal cortex, and can be reinitialized and reused to afford efficient transfer learning to new tasks [21, 22]. However, there is still lack of deep understanding of how these schemas are represented and reused in neural circuits in the brain. In this work, we study the representation and reuse of a specific schema, the ordinal structure of temporal sequences, in neural circuits. We show that with an appropriate training protocol, a recurrent neural circuit can learn tree-structured attractor dynamics to represent the ordinal structure of temporal sequences. This abstract order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. We showcase this advantage by using a transfer learning and a key-word spotting tasks.

Our model may have some far-reaching implications to the representation of abstract schemas in the brain. Recently, researchers have conducted interesting studies on the format of neural schemas. For example, Guntupalli et al. proposed a graph model [23]. Similar to our work here, by learning new associations between new sensory features and schema representations, this graph model can utilize common abstract knowledge to facilitate transfer learning. However, their schema representations employ a high-order Markov dynamic model rather than the attractor dynamic model, which raises the question of how the brain biologically represents such schema representations. On the other hand, Goudar et al. trained a recurrent neural network model on various common sensor-motor mapping tasks and discovered a low-dimensional attractor dynamics which emerges and acts as an abstract structure to accelerate new decision-making tasks [24]. In contrast to Goudar's work, our research primarily focuses on abstract temporal order representation in the context of a real-world task involving key-word spotting. This task provides a more intricate and complicated setting for exploring abstract schema representations than what has been considered in previous studies. Overall, studying various attractor dynamics for different schema representations in the neural system would offer fascinating avenues for future research.

Although the focus of the present study is on unveiling the biological mechanism for representing the ordinal structure of temporal sequences, the computational principle we have gained can be applied to other brain-inspired computations. To test this idea, we replace the RNN in the model with two popular recurrent networks in machine learning, which are GRU and LSTM. Applying to the key-word spotting task, we observe that (Table.1): if the same training protocol proposed in this work is used, both GRU and LSTM achieve good performances; while if the conventional training protocol (i.e., no data augmentation and using the cross-entropy loss function) is used, they can still discriminate original temporal sequences but fail to be robust to temporal warping. These results suggest that our method (including the model structure and the training protocol) is general and has potential to be applied to brain-inspired computing.

Table 1: Model performances using different recurrent networks and different training protocols in a key-word spotting task. We replace the RNN in our model by GRU or LSTM. We train the model following either the same training protocol used in this work (marked by*) or the conventional protocol, i.e., no data augmentation and the cross-entropy loss function. Both models are trained on the dataset without warping ( F = 1 ) and tested on a dataset with warping ( F = 0 . 6 , 1 . 0 , 1 . 4 , 1 . 8 , 2 . 2 , 2 . 6 , 3 . 0 ). The classification accuracy of each model is obtained by averaging over 8 trials with randomly split training and testing data, as well as network initialization. For each task, we have 10 samples per class in the train dataset and 300 samples per class in the test dataset. The word sequences are sampled from the TIMIT dataset. For more details, see Sec. D, F in SI.

In order to learn a tree-like structure in our model, we require precise ordinal details about continuous sequences. Several mechanisms can be employed in our brain to access this kind of information. Firstly, a large volume of experimental studies has shown that in the brain, continuous sequences are often chunked into discrete items to support high-level cognition [25]. For examples, speech sequences can be hierarchically chunked into words and syllables [1]; neurons in the hippocampus have been shown to detect event boundaries when watching continuous movie videos [26]. This chunking/segmentation process naturally outputs the ordinal structure of a temporal sequence. Additionally, computational models have been proposed for the brain performing sequence chunking, such as self-supervised learning [27] and oscillation [28]. Secondly, the development path of our brain also indicates the brain can naturally acquire sequence chunking. For example, in language acquisition, young children learn primitive phoneme categories around 2 months [29]; around 7 months, they learn sequences of either an ABB or ABA form [30], which can be represented as tree-like attractor structures in our model; subsequently, children learn to recognize spoken words around 12 months. Thus, in language acquisition, phoneme learning serves as a building block for word learning, which defines the ordinal structure of language sequences.

Incorporating a wide range of diverse temporal intervals presents another challenge for our model. How does the neural system handle this in a biologically plausible manner? Firstly, in our brain, motor and speech sequences are often generated with a large variability in speed, and hence they exhibit a large variability in separations between motor motifs and speech chunks. This large variability in separations enables the brain to learn the tree-like attractor structure, as demonstrated in our model. Secondly, in our network training, we do not really need very large intervals. For the clean synthetical data, we can actually train the tree-structured attractors using fixed interval values, shown in Fig. S10A. For the noisy spoken words, we do need an amount of variations to achieve good performances, but the range is only about 2 times of the item length. Overall, the requirement for a wide range of temporal intervals can be largely relaxed in practice.

Finally, there are several interesting directions worth exploring further. First, our current model is currently limited to small sequences. Figuring out how to effectively expand and scale our approach for more complex scenarios, like sequences involving large classes and longer sequences, will be a crucial focus for our research. Second, in the context of transfer learning, we employ a stochastic gradient descent method to learn the binding between sensory features and abstract representations. However, this technique is not biologically plausible and efficient. Experimental studies have shown the existence of a fast time-scale Hebbian rule in the brain, such as in the hippocampus [31], which is suggested to mediate rapid binding of distinct neural populations. By incorporating this fast Hebbian rule, our model could learn new associations between contents and the ordinal structure in an one-shot manner. Third, beyond ordinal structure and contents, the timing structure in a temporal sequence is also crucial for some tasks. Recent experiments have shown that the human brain could integrate abstract order template, timing structure, and contents to generate diverse finger action sequences [6]. Thus, integrating the timing structure into our model could potentially enhance its power for processing more complicated temporal sequence processing tasks. Last, the ordinal structure also serves as the basis for some more high-level abstract structures, such as algebraic patterns in motor sequence representation and recursive syntactic structures in language representation [1, 32]. How these abstract representations are concretely represented in neural circuits is largely unknown. Our model may serve as a starting point for exploring these mysteries.

Acknowledgments and Disclosure of Funding

This work was supported by the National Natural Science Foundation of China (N0: T2122016, Y.Y.Mi), National Science and Technology Innovation 2030 Major Program (No. 2021ZD0203700 / 2021ZD0203705, Y.Y. Mi, No. 2021ZD0200204, Si Wu).

References

Dehaene, S., Meyniel, F., Wacongne, C., Wang, L., & Pallier, C. (2015). The neural representation of sequences: from transition probabilities to algebraic patterns and linguistic trees. Neuron, 88(1), 2-19.

Gwilliams, L., King, J. R., Marantz, A., & Poeppel, D. (2022). Neural dynamics of phoneme sequences reveal position-invariant code for content and order. Nature communications , 13(1), 6606.

Yi, H. G., Leonard, M. K., & Chang, E. F. (2019). The encoding of speech sounds in the superior temporal gyrus. Neuron, 102(6), 1096-1110.

Xie, Y., Hu, P., Li, J., Chen, J., Song, W., Wang, X. J., ... & Wang, L. (2022). Geometry of sequence working memory in macaque prefrontal cortex. Science, 375(6581), 632-639.

Ninokura, Y., Mushiake, H., & Tanji, J. (2004). Integration of temporal order and object information in the monkey lateral prefrontal cortex. Journal of neurophysiology, 91(1), 555-560.

Kornysheva, K., Bush, D., Meyer, S. S., Sadnicka, A., Barnes, G., & Burgess, N. (2019). Neural competitive queuing of ordinal structure underlies skilled sequential action. Neuron, 101(6), 1166-1180.

Pitti, A., Quoy, M., Lavandier, C., Boucenna, S., Swaileh, W., & Weidmann, C. (2022). In search of a neural model for serial order: A brain theory for memory development and higher level cognition. IEEE Transactions on Cognitive and Developmental Systems, 14(2), 279-291.

Badre, D., & Nee, D. E. (2018). Frontal cortex and the hierarchical control of behavior. Trends in cognitive sciences, 22(2), 170-188.

López-Espejo, I., Tan, Z. H., Hansen, J. H., & Jensen, J. (2021). Deep spoken keyword spotting: An overview. IEEE Access, 10, 4169-4199.

Rabinovich, M., Huerta, R., & Laurent, G. (2008). Transient dynamics for neural processing. Science, 321(5885), 48-50.

Yildiz, I. B., von Kriegstein, K., & Kiebel, S. J. (2013). From birdsong to human speech recognition: Bayesian inference on a hierarchy of nonlinear dynamical systems. PLoS computational biology, 9(9), e1003219.

Gillett, M., Pereira, U., & Brunel, N. (2020). Characteristics of sequential activity in networks with temporally asymmetric Hebbian learning. Proceedings of the National Academy of Sciences, 117(47), 29948-29958.

Kiebel, S. J., von Kriegstein, K., Daunizeau, J., & Friston, K. J. (2009). Recognizing sequences of sequences. PLoS computational biology, 5(8), e1000464.

Cone, I., & Shouval, H. Z. (2021). Learning precise spatiotemporal sequences via biophysically realistic learning rules in a modular, spiking network. Elife, 10, e63751.

Frölich, S., Markovi´ c, D., & Kiebel, S. J. (2021). Neuronal sequence models for Bayesian online inference. Frontiers in Artificial Intelligence, 4, 530937.

Wang, X. J. (2008). Decision making in recurrent neuronal circuits. Neuron, 60(2), 215-234.

Okazawa, G., & Kiani, R. (2023). Neural Mechanisms that Make Perceptual Decisions Flexible. Annual Review of Physiology, 85, 191-215. S

Shadlen, M. N., & Newsome, W. T. (1996). Motion perception: seeing and deciding. Proceedings of the national academy of sciences, 93(2), 628-633.

Sussillo, D., & Barak, O. (2013). Opening the black box: low-dimensional dynamics in highdimensional recurrent neural networks. Neural computation, 25(3), 626-649.

Lerner, Y., Honey, C. J., Katkov, M., & Hasson, U. (2014). Temporal scaling of neural responses to compressed and dilated natural speech. Journal of neurophysiology, 111(12), 2433-2444.

Tse, D., Langston, R. F., Kakeyama, M., Bethus, I., Spooner, P. A., Wood, E. R., ... & Morris, R. G. (2007). Schemas and memory consolidation. Science, 316(5821), 76-82.

Zhou, J., Jia, C., Montesinos-Cartagena, M., Gardner, M. P., Zong, W., & Schoenbaum, G. (2021). Evolving schema representations in orbitofrontal ensembles during learning. Nature, 590(7847), 606-611.

Guntupalli, J. S., Raju, R. V., Kushagra, S., Wendelken, C., Sawyer, D., Deshpande, I., ... & George, D. (2023). Graph schemas as abstractions for transfer learning, inference, and planning. arXiv preprint arXiv:2302.07350.

Goudar, V., Peysakhovich, B., Freedman, D. J., Buffalo, E. A., & Wang, X. J. (2023). Schema formation in a neural population subspace underlies learning-to-learn in flexible sensorimotor problem-solving. Nature Neuroscience, 1-12.

Dehaene, S., Al Roumi, F., Lakretz, Y., Planton, S., & Sablé-Meyer, M. (2022). Symbols and mental programs: a hypothesis about human singularity. Trends in Cognitive Sciences.

Hahamy, A., Dubossarsky, H., & Behrens, T. E. (2023). The human brain reactivates contextspecific past information at event boundaries of naturalistic experiences. Nature neuroscience, 1-10.

Asabuki, T., & Fukai, T. (2020). Somatodendritic consistency check for temporal feature segmentation. Nature communications, 11(1), 1554.

Giraud, A. L., & Poeppel, D. (2012). Cortical oscillations and speech processing: emerging computational principles and operations. Nature neuroscience, 15(4), 511-517.

Kuhl, P. K. (2004). Early language acquisition: cracking the speech code. Nature reviews neuroscience, 5(11), 831-843.

Marcus, G. F., Vijayan, S., Bandi Rao, S., & Vishton, P. M. (1999). Rule learning by sevenmonth-old infants. Science, 283(5398), 77-80.

Bittner, K. C., Milstein, A. D., Grienberger, C., Romani, S., & Magee, J. C. (2017). Behavioral time scale synaptic plasticity underlies CA1 place fields. Science, 357(6355), 1033-1036.

Shima, K., Isoda, M., Mushiake, H., & Tanji, J. (2007). Categorization of behavioural sequences in the prefrontal cortex. Nature, 445(7125), 315-318.

Appendices

A The Training Protocol

a

b

a

'dx-axr'

'w-ao'

{a,c,a}

{a,b,c}

{a,b,a}

{a,c,b}

{b,a,b}

{b,a,c}

{b,c,a}

{b,c,b}

water

wash

had

year

carry

suit

oily

dark

To train the model, we first augment training data and construct their target functions (see illustration in Fig.2A of maintext), after that we apply backpropagation-through-time (BPTT) algorithm to optimize the model parameters { W in , W rec , W out , I b } . Denote y ( t ) the actual model output sequence and ˆ y ( t ) the corresponding target function. The loss function is given by

where b = 1 , . . . , B denote the index of temporal sequences in the training data, T b the length of the b th temporal sequence, and k = 1 , . . . , K the index of the output neuron. We adopt the ADAM optimizer, with the learning rate varying in the range of lr = [1 e -4 , 1 e -2 ] , and the decay rates of the first and second moments are 0 . 9 and 0 . 999 .

Inspired by the ramping activity of decision making neurons observed in the experiment [1-3], we set the target function of a read-out neuron increases with the unfold of the items in the temporal

sequence this neuron encodes. Consider the temporal sequence of the first class, denoted as S 1 , is presented to the network. The target functions of read-out neurons are constructed as below.

(1) The task of discriminating four synthetic temporal sequences ( K = 4 ), Fig.2 in the main text.

In this task, S 1 consists of two consecutive items { S 1 1 , S 1 2 } with the duration ∆ T , as shown in Fig.2A-B in the main text. Before the temporal sequence is presented, all read-out neurons are at the spontaneous state, whose target functions are set to be ˆ y k = O init , for k = 1 , ..., K . When the first item S 1 1 appears, both the target functions of neuron 1 and 2 increase to ˆ y 1 , 2 = O 1 , and the target functions of other neurons decay to the silent state, ˆ y 3 , 4 = O sile . This lasts for the duration ∆ T , until the second item appears. When S 1 2 appears, the target function of neuron 1 further increases to ˆ y 1 = O 2 , while the target function of neuron 2 decreases to O sile , and the target functions of other neurons keep the same, i.e., ˆ y 2 , 3 , 4 = O sile . In this task, the parameters used are O init = 0 . 25 , O 1 = 0 . 5 , O 2 = 1 , O sile = 0 .

For other tasks of discriminating four temporal sequences, the target functions are designed similarly.

(2) The task of discriminating eight synthetic temporal sequences ( K = 8 ), Fig.3C in the main text

In this task, S 1 consists of three consecutive items { S 1 1 , S 1 2 , S 1 3 } with the duration ∆ T , as shown in Fig.3C in the main text. The design of target functions is illustrated in Fig.S1A, similar to the task with K = 4 . Before the sequence is presented, all target functions of all read-out neurons are set to be ˆ y k = O init for k = 1 , ..., 8 . When the first item S 1 1 appears, the target functions of neurons 1 -4 increase to be ˆ y k = O 1 , for k = 1 , 2 , 3 , 4 , since the four neurons share the same first item; while the target functions of other neurons decay to the silent state, i.e., ˆ y k = O sile for k = 5 , 6 , 7 , 8 . When S 1 2 appears, the target functions of neurons 1 -2 keep increasing to ˆ y k = O 2 for k = 1 , 2 ; while the target functions of neurons 3 -4 decay to silent, and the target functions of other neurons keep silent, i.e., ˆ y k = O sile , for k = 3 , ..., 8 . When S 1 3 appears, only the target function of neuron 1 keep increasing to ˆ y 1 = O 3 , while all other neurons are in the silent state, i.e., ˆ y k = O sile , for k = 2 , ..., 8 . The parameters used are O init = 0 . 25 , O 1 = 0 . 5 , O 2 = 0 . 75 , O 3 = 1 , and O sile = 0 .

For other tasks of discriminating eight temporal sequences, the target functions are designed similarly.

(3) The key-word spotting task of discriminating eight or twelve temporal sequences ( K = 8 , 12 ), Table 1 in the main text.

In this task, S 1 denotes the temporal sequence of the key word 'water', which is chunked into two different items by syllables, i.e., 'w-ao' and 'dx-axr'. The training data is augmented according to the learning protocol described in the main text, as illustrated in Fig.S1B (upper panel). Before the sequence is presented, all read-out neurons are at the spontaneous state, i.e., ˆ y k = O init , for k = 1 , ..., K , with K = 8 , 12 . When the first item ('w-ao') appears, the target functions of neurons 1 -2 increase to O 1 , while the target functions of other neurons decay to O sile . When the second item appears, only the target function of neuron 1 keep increasing to O 2 , while the target functions of other neurons are, ˆ y k = O sile for k = 2 , ..., K . The design of target functions for S 1 in 8-classes task is illustrated in Fig.S1B (lower panel). The parameters used are O init = 0 . 25 , O 1 = 1 , O 2 = 2 , O sile = 0 .

B The Visualization Method

B.1 Principle Component Analysis (PCA) of neural activities in the RNN (Sec.2.4.1 in the Main text)

We apply PCA to find out the low dimensional state space which captures the major variance of neural activities in the RNN. Denote r k ( t ) the activity vector of the RNN in response to the k th temporal sequence at moment t , for k = 1 , ..., K , whose element is given by r k i ( t ) = r i ( t ) , for i = 1 , ..., N , with N the number of neurons in the RNN. Combining the activity of the RNN at all moments, we obtain a martix R k = { r k ( t ) } , whose dimension is N × T , with T the length of k th temporal sequence. Define the covariance matrix X = ∑ N k i =1 ∑ K k =1 ( R k -< R > )( R k -< R > ) T , where < R > is obtained by averaging activity vectors over time and the total testing trials, K × N k , with N k the number of trails in each class. We apply PCA to the matrix X and obtain a set of eigenvectors with decreasing eigenvalues. The eigenvalue of each PC reflects the variability of RNN response over time and trials along the corresponding eigenvector. As shown in Fig.2D in the main text, the first

3 PCs account for nearly 90% of the variance of RNN activities. On the basis of this analysis, we project the RNN activities onto the first 3 PCs to visualize the evolution of the RNN state over time for different temporal sequences, as shown in Fig.2E in the main text.

B.2 Stability analysis of tree-structured attractors (Sec.2.4.2 in the main text)

To uncover the structure of the state space of the learned RNN, we firstly find fixed points of the RNN dynamics, and then analyze their stability.

(1) Finding fixed points in the RNN dynamics

Identifying fixed points or slow points in the dynamics of the RNN helps us to understand the dynamical properties of our model. Specifically, a fixed point is a state that remains unchanged over time to a constant input, while a slow point is a state that changes very slow over time to a constant input. The dynamics of the RNN is given by τd x ( t ) /dt = -x ( t ) + W rec r ( t ) + W in I ( t ) + I b , which can be approximately solved using the first order Euler method with the simulation time step ∆ t = 1 , and the neural activity r ( t ) at time t +1 can be approximately written as r ( t +1) = F ( r ( t ) , I ( t +1)) = tanh ((1 -∆ t/τ ) ∗ arctan ( r ( t )) + ∆ t/τ ( W rec r ( t ) + W in I ( t ) + I b )) , where F ( · ) is the state update function. Denote r ∗ a fixed point in the state space of the RNN, which satisfies the equation, r ∗ = F ( r ∗ , I ) for a particular input I . To get a better understanding of the network dynamics, we are also interested in slow points, which approximately satisfy r ∗ ≈ F ( r ∗ , I ) .

To identify these fixed or slow points, we follow the approach developed by Sussillo and Barak [4] by solving an optimization problem. We focus only on the fixed and slow points when the network receives no external input (i.e., I = 0 ). We define q as an variable describing the speed at which the network state evolves over time, which is given by,

The smaller q is, the slower the network dynamics evolves. To optimize q over the RNN state r , we use BPTT and initialize from many conditions sampled from the distribution of network states during training. The neural states satisfying q < 10 -7 are considered to be fixed points. For the synthetic task as shown in Fig. 2 in the main text, we identify 7 fixed points in the RNN dynamics.

(2) Stability analysis of fixed points

For each fixed point r ∗ , we analyze its stability using the perturbation method [5]. Specifically, we apply perturbations to a fixed point r ∗ , denoted as ∆ r ( t ) and ∆ I ( t ) , respectively. The network state at t +1 can be updated with the first-order Taylor expansion, which is given by,

where { J rec , J inp } are Jacobian matrices given by J rec ij ( r ∗ , I ∗ ) = ∂F ( r , I ) i ∂r j | ( r ∗ , I ∗ ) and J inp ij ( r ∗ , I ∗ ) = ∂F ( r , I ) i ∂I j | ( r ∗ , I ∗ ) .

The eigendecomposition of the recurrent Jocabian matrix J rec can be expressed as J rec = RΛL , where R and L are matrices that contain the right and left eigenvectors of J rec , respectively, and Λ is a diagonal matrix containing complex-valued eigenvalues. The value of the real part of an eigenvalue determines the stability of the corresponding eigenmode, with those smaller than 1 representing stable modes and those larger than 1 representing unstable modes. As shown in Fig.S6, the maximums of real parts of all eigenvalues of each fixed point are all smaller than 1 , indicating that all these fixed points are stable attractors of the RNN.

B.3 Analyzing the transition dynamics of attractor states (Sec.2.4.3 in the main text)

We take the transitions from stable points S 1 , 2 1 to S 2 1 and S 2 2 as examples to illustrate the transition dynamics of the RNN. We combine the network activity trajectories in response to the k th class temporal sequence, for k = 1 , ..., K , denoted as ˆ R k = { r k ( t 1 : t 2 ) } , to construct the matrix ˆ X as in Sec.B.1 of SI, where t 1 and t 2 denote the moments of the appearance of item S 1 , 2 1 and the ending of the sequence, respectively. Define the covariance matrix ˆ X = ∑ N k i =1 ∑ K k =1 ( ˆ R k -< ˆ R > )( ˆ R k -< ˆ R > ) T , where < ˆ R > is the average of network activities over time from t 1 to t 2 and over K × N k

trails, with N k the number of trail in each class. We perform PCA on the matrix ˆ X and obtain a set of eigenvectors with decreasing eigenvalues. The first two eigenvectors (denoted as PC 1 and PC 2 ) account for the major variance of neural responses.

We further analyze the right eigenvectors of the Jocabian matrix J rec around the stable fixed point S 1 , 2 1 . The larger the eigenvalue is, the more unstable the network activity state is along the eigenvector under noise perturbation. We project the right eigenvectors onto the hyperplane spanned by PC 1 and PC 2 , with their lengths determined by the corresponding eigenvalues. We observe that the two most unstable directions among all eigenvectors point to nodes S 1 2 and S 2 2 , as shown in Fig.2E-F. This indicates that it is much easier for the network state transferring from node S 1 , 2 1 to nodes S 1 2 and S 2 2 than to other states.

C Transfer Learning Tasks

C.1 The construction of phoneme Sequences (Sec.3.1 in the main text)

The three primitive items, 'pcl', 'tcl' and 'pau' used in Fig.3 in the main text, are randomly selected from the TIMIT dataset [6]. We transform the raw waves of three phoneme items into Mel-frequency cepstral coefficients (MFCC) with a window size of 25 ms and stride of 10 ms, roughly mimicking the information processing in the cochlea in the brain. The obtained MFCC feature vectors have 16 channels, and the MFCC vectors are further normalized by subtracting their mean and dividing by their standard deviation, along the time dimension, which are used as inputs to the RNN. Using the four-class classification task as an example, as shown in Fig.2B in the main text, we construct four temporal sequences, which are "pcl-tcl", "pcl-pau", "tcl-pcl", and "tcl-pau". We augment these sequence by inserting varied intervals between primitive items and adding Gaussian noises. We then train the model using ramping target functions. In the task, the total number of augmented data is 10000 , which are randomly divided into 8000 training and 2000 testing examples.

C.2 The performance measurement (Fig.3A in the main text)

In transfer learning, the model learns to allocate phoneme contents to the pre-trained tree-like attractor template. To evaluate the effect of reusing the ordinal structure, we calculate the discrepancy between the learned attractor structure and the template, measuring by the Euclidean distance between them. Each item and its followed duration can be seen as a temporal chunk. If the model learns to reuse the pre-trained attractor dynamics, the last state of each temporal chunk should exactly stay at the corresponding attractor state. To test this, as an example, for the sequence S 1 = { S 1 1 , S 1 2 } in the four-class discrimination task, the learned state vector of S 1 1 item is obtained by averaging the state vectors over the last five time steps before the item S 1 2 is presented. Similarly, the learned state vector of S 1 2 item is obtained by averaging the state vectors over the last five time steps before the end of the sequence. The learned state vectors of other sequence items are computed similarly. After obtaining

Discrepancy

A

B

these learned state vectors, we calculate the Euclidean distance between them and the corresponding tree-structured attractor state vectors in the template. The final discrepancy is computed by averaging the results of seven attractor states. As shown in Fig.3A and Fig.SS7, as the training progresses, the discrepancy between the learned attractor states and the template attractor states decreases gradually to zero, indicating that the model learns to allocate the input contents to the ordinal template stored in the RNN.

In transfer learning setting, We also compare our model with tree attractor structure to a pretrained recurrent neural network without using tree structrue. As shown in Fig. S7B, we train a control model having the same network structure as our model, except that the cross-entropy loss, rather than the ramping target function, is used. By this, the control model learns the sequence discrimination task but no longer acquires the tree-like attractors. We then apply the control model to the same transfer learning task (following the same protocol of freezing the connections in the recurrent and read-out layers). The results demonstrate that our model exhibits significantly accelerated learning process compared to the control model. This experiment demonstrates that the tree-structured attractor dynamics (the schema) is indeed indispensable for transfer learning. Together with our other analyses in Fig.3A and Fig. S7A), which verify that the attractor template is indeed reused after transfer learning, we come to the conclusion that the tree-structured attractor dynamics plays a critical role in transfer learning.

As shown in Fig. S8, we show that our model can learn to process sequences of depth 3 by combining two primitive templates of depth 2. In this experiment, for simplicity, we consider two independent recurrenct networks and there are no recurrent connections between them. Firstly, two RNNs learn and form same tree attractor structures of depth 2 through training on a four-class synthetical task. Then, following the transfer learning procedure as described in Sec.3.1, we freeze the recurrent connections in two RNNs and only optimize the feedforwad and readout connections, as shown in Fig. S8A. New eight-class phoneme task are constructed, written as "pcl-tcl-pcl", "pcl-tcl-pau", "pcl-pau-pcl", "pcl-pau-tcl", "tcl-pcl-tcl", "tcl-pcl-pau", "tcl-pau-tcl" and "tcl-pau-pcl". Our findings are as follows: 1) the pretrained attractor structures significantly accellerate the learning speed in a new sequence task, as depcited in Fig. S8C; 2) the network indeed combines and reuses two primitive tree attractor dynamics to facilitate the learning speed, as shown in Fig. S8D. These results demonstrate that our model has the capability to adapt to processing temporal sequences of varying lengths.

A

B

C

subRNN2

root

'pcl'

'pcl'

'pau'

'pau'

'pcl'

subRNN1

...

...

subRNN2

subRNN1

(%)

Combine and Reuse

D

D Key-word spotting tasks

D.1 Dataset Construction for Key-Word Spotting Task (Sec.3.2 in the maintext)

To construct the training and testing datasets for the key-word spotting task in Sec.3.2 in the main text, we randomly select twelve key words from the TIMIT dataset based on the word labeling. The TIMIT dataset comprises 6300 English sentences read by 630 speakers. The twelve key words are illustrated in Table.S1, which can be chunked into two different primitive items according to syllables. For example, 'wash' is chunked into 'w-ao' and 'sh'.

In the four-class key-word spotting task, we use the four words, "water, wash, had, year". In the eight-class task, we add another four words, "carry, oily, dark, suit". In the twelve-class task, we use all the selected words. In each task, the total number of temporal sequences for each word is nearly 310 , which are randomly divided into 10 training and 300 testing examples. The raw wave of each temporal sequence is transformed into MFCC features using a time window size of 25 ms and stride of 10 ms. The obtrained MFCC vectors of each word is further normalized by subtracting their means and divided by their standard deviations along the time dimension.

In the training data, following our training protocol described in the main text, we chunk the MFCC vectors of a sequence into two primitive items according to the syllables of the key word, and then augment the data by inserting varied intervals between primitive items and adding Gaussian white noises. We then train the augmented temporal sequences of key words using ramping target functions, more details see Sec. A in SI. In the testing data, we present the MFCC vectors of each sequence without augmenting, rather we stretch or compress the MFCC sequences. The warping effect is specified by a variable F , with F = 1 denoting no warping, F > 1 stretching and F < 1 compressing.

During training, both our model and the control model are trained on data with F = 1 . During testing, we apply a warping factor F to compress or stretch the original MFCC feature vectors using linear interpolation while retaining their spatial structure (particularly, we use numpy.interp function to realize temporal warping, which works by finding the two nearest data points surrounding each value in the target data, constructing a straight line between these two points, and calculating the stretched or compressed target data along this line). After temporal warping, we scale the total strength of MFCC feature vectors by dividing a scaling factor √ F , mimicking the divisive normalization process in the neural system [7]. Finally, the scaled versions of testing words are used to evaluate the model's classification robustness. Notably, scaling the strength of a MFCC feature vector slightly improves models' performance during testing, but in any case, our model outperforms other models significantly.

D.2 Ablation study (Sec.4 in the main text)

In our training protocol, the introduction of varied temporal intervals between phonemes or syllables has the potential to generate spectral structures that closely resemble those present in the test examples. Consequently, the apparent efficacy of our model may be attributed to the exposure to training data that closely mirrors the test data, rather than leveraging tree attractor structures. To alleviate this concern and demonstrate the importance of the training protocol, we compare our model with several control models in the key-word spotting task. It demonstrates that the tree-structured attractor dynamics (the schema) contributes to the robustness of our model to warping sequences.

In our model, each training sequence is augmented to be 1.5 times of its original length by inserting varied intervals between phoneme or syllable chunks, and the model is trained using the ramping target function. In control model 2 , training examples adopt the same augmentation protocol as in our model, but the model is trained using cross-entropy loss. In the control model 1 , training examples are just warped sequences with the warping value chosen randomly in the range of ( 1 , 1 . 5 ). This makes the training and test data even more similar in the control model 1 than that in our model by data augmentation (in both models, the lengths of sequences are kept to be in the same range). We used the cross-entropy loss to train the control model 2 . Thus, both control model 1 and 2 would not acquire proper tree-structured attractors.

We find that when the temporal warping value falls in the range of [1, 1.5], i.e., the range of training sequences, both models exhibit similar good test accuracies. However, when the warping value is outside of the training range, our model outperforms the control model significantly, as shown in Fig. S9.

This supports our statement that the tree-structured attractor dynamics (the schema) contributes to the robustness of our model to warping sequences. The underlying mechanism can be attributed to two factors: 1) attractors ensure the stable responses of our model to stretching/compressing inputs; 2) the tree-structured attractor dynamics further enhances averaging out noises in inputs over time, behaving like an evidence-accumulation process (see Fig.4C)

We also apply our training protocol to other recurrent networks used in machine learning, including Long Short-Term Memory (LSTM) [9] and Gated Recurrent Unit (GRU)[10]. Using the datasets in the key-words spotting task, both GRU and LSTM are trained by using either our learning protocol or the conventional learning method (which involves data augmentation and the cross-entropy loss function). The results are compared in Table.1 in the main text.

E Experiments with different training Settings and model architectures

We find that the tree-structured attractors also emerge when training using fixed interval values for the clean synthetical data, as shown in Fig. S10A. However, for the noisy spoken words, we do need an amount of variations to achieve good performances.We also examine our method on GRU network to test its university across model architectures. As shown in Fig. S10B, GRU network can also learn a tree-structured attractors successfully.

F Model Parameters used in this study

All models are trained using PyTorch 2.0 [8].

Parameters in Fig.2 in the main text:

The parameters of the RNN are: M = 3 , N = 50 , K = 4 , τ = 2 . The parameters for data augmentation are: ∆ t = 3 , ∆ T max = 30 , the noise strength inserted between neighboring items is σ = 0 . 01 . The recurrent connection weights W rec are initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ √ N , with g representing the 'gain' of the network, g = 0 . 2 . For the design of target functions, see Sec. A. The batch size is B = 16 , and the total number of training epochs is 100 .

(2) Parameters in Fig.3 in the main text:

A

B

The parameters of the RNN are: M = 16 , N = 50 , K = 4 in Fig.4A-B and K = 8 in Fig.4C-D, τ = 2 . The parameters for data augmentation are: ∆ T max = 30 , the noise strength inserted between neighboring items is σ = 0 . 01 . For details of the construction of phoneme temporal sequences, see Sec. C in SI. W rec is initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ √ N , g = 0 . 2 . For the design of target functions, see Sec. A in SI. The batch size is B = 16 , and the total number of training epochs is 100 .

Parameters in Fig.4 in the main text:

The parameters of the RNN are: M = 16 , N = 100 , K = 4 , τ = 2 . The parameters for data augmentation are: ∆ T max = 60 , the noise strength inserted between neighboring items is σ = 0 . 01 . The recurrent connection weights W rec are initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ √ N , with g representing the 'gain' of the network, g = 0 . 2 . For the design of target functions, see Sec. A. The batch size is B = 16 , and the total number of training epochs is 200 .

Parameters in Table 1 in the main text:

The parameters of LSTM, GRU and RNN are: M = 16 , N = 100 , K = 4 . The time constant of neurons in the RNN is τ = 2 . The parameters for data augmentation are: ∆ T max = 60 , the noise strength inserted between neighboring items is σ = 0 . 01 . The recurrent connection weights W rec are initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ √ N , with g representing the 'gain' of the network, g = 0 . 2 . The connections of LSTM and GRU adopt Glorot are initialized by an uniform distribution. For the design of target functions, see Sec. A. The batch size is B = 16 , and the total number of training epochs is 200 .

References

Wang, X. J. (2008). Decision making in recurrent neuronal circuits. Neuron, 60(2), 215-234.

Okazawa, G., & Kiani, R. (2023). Neural Mechanisms that Make Perceptual Decisions Flexible. Annual Review of Physiology, 85, 191-215.

Shadlen, M. N., & Newsome, W. T. (1996). Motion perception: seeing and deciding. Proceedings of the national academy of sciences, 93(2), 628-633.

Sussillo, D., & Barak, O. (2013). Opening the black box: low-dimensional dynamics in highdimensional recurrent neural networks. Neural computation, 25(3), 626-649.

Maheswaranathan, N., Williams, A., Golub, M., Ganguli, S., & Sussillo, D. (2019). Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. Advances in neural information processing systems, 32.

Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S. & Dahlgren, N. L. (1993). DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM NIST

Carandini, M., & Heeger, D. J. (2012). Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1), 51-62.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., ... & Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.

Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript the authors use a custom training regime to force simple recurrent neural networks (RNNs) to learn the ordinal structure of sequential inputs. Specifically, they train the network to learn the order of sequences by presenting the elements of a sequence with variable durations and variable intervals between elements. They demonstrate the utility of these networks in both a toy transfer learning task and a more realistic \"key-word spotting\" task.\n\nSTRENGTHS:\nStrengths:\n\n1. The manuscript presents the key ideas in a straightforward and lucid manner\n2. The core idea – training RNNs to recognize the ordinal structure of its inputs through varying the duration of its constituent elements – is interesting with potential downstream applications in machine learning and computational neuroscience\n\nWEAKNESSES:\nWeaknesses:\n\nMajor:\n\n1. The results in Figure 3 are difficult to interpret without a more systematic bank of control models. For instance, in the transfer learning task, as far as I can tell the authors compare transfer learning of their particular model to a model with no transfer learning. This does not tell us if the tree structure learned by their model is doing the heavy lifting. Specifically, have the authors compared their \"tree-structure\" model to an RNN that learns the task without using tree structure? Without this key control we cannot assess whether tree structure is what confers performance in this task, or if it is simply a matter of their model having been trained to solve *any* task prior to transfer learning.\n1. I am a bit confused about what Figure 4 is showing. Here, the authors train their network using varied intervals between \"chunks\" of a given word. One issue that I can see with this training regime is that, in addition to enforcing tree structure in network dynamics, it is also presenting a \"warped\" version of the inputs. In other words, does inserting a delay between phonemes create spectral structure more similar to the test inputs than the phonemes *without* gaps? If true, this would imply that the network is simply seeing training data more similar to the test data, rather then leveraging tree structure. Further controls and analysis would alleviate this concern.\n\nMinor:\n\n1. Typos throughout, the manuscript needs a pass on the writing.\n\nQUESTIONS:\n1. Can the authors provide more detail about the control models in Figures 3 and 4? My concerns could be simple misunderstanding.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how recurrent neural circuits learn to represent the abstract order structure of temporal sequences and how the disentanglement facilitates sequence processing. The main objectives were better understand the brain's mechanisms for representing temporal sequence ordinal information and contributing to the development of brain-inspired sequence processing algorithms. \n\nThe authors found that given suitable training, a recurrent neural circuit can learn tree-structured attractor dynamics to encode the tree-structured orders of temporal sequences. They show that reusing a temporal order template aids the learning of new sequences sharing the same or partial ordinal structure and that the tree-structured attractor dynamics improve the robustness of temporal sequence discrimination.\n\nSTRENGTHS:\nThe authors demonstrated that in a supervised learning task, recurrent neural circuit can learn tree-structured attractor dynamics to encode the corresponding tree-structured orders of temporal sequences. They used a transfer learning task to show that once the network has learned the temporal structure, it can apply that knowledge to different temporal inputs - this was demonstrated by freezing the recurrent weights and training only weights in a feedforward layer that followed the recurrent layer. They also showed that data augmentation can lead to invariance to temporal rescaling. These results are consistent with several neuroscience studies.\n\nWEAKNESSES:\nThe evaluation is relatively limited and uses only short sequences and supervised learning. If the brain indeed uses a similar mechanism, then it should successfully scale to much longer sequences.\n\nIf the model is capturing cognitive properties of human sequence memory, then it should also be consistent with results from behavioral studies, which are characterized by effects such as primacy, recency and contiguity.\n\nThe authors might consider reflecting on or comparing their approach with other work related to modeling sequences in the brain, for example:\n\nCui et al. 2016 Continuous online sequence learning with an unsupervised neural network model\n\nGraves et al. 2014 Neural Turing machines \n\nVoelker et al. 2019 Legendre memory units: Continuous-time representation in recurrent neural networks \n\nEliasmith et al. 2013 A large-scale model of the functioning brain \n\nHoward et al. 2014 A unified mathematical framework for coding time, space, and sequences in the hippocampal region \n\nWhittington et al. 2020 The Tolman-Eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation\n\nAlso, for recent work on biologically inspired neural networks robust to temporal rescaling see \n\nJacques et al. 2022 A deep convolutional neural network that is invariant to time rescaling\n\nQUESTIONS:\nWhen you used data augmentation and varied the duration between neighboring items, how many steps of BPTT did you have to do and have you encountered issues with vanishing/exploding gradients? \n\nThe authors reflected on the biological implausibility of backpropagation and proposed the possibility of using the fast Hebbian rule. It is not clear whether this could work in the context of backpropagation through time since it would require that the biological system stores a copy of the temporal input. Do the authors have a possible solution for this? \n\nDo you think the observed properties could emerge if the training was done in a self-supervised way?\n\nLIMITATIONS:\nThe authors addressed some of the limitations, especially regarding the backpropagation (but see my earlier comments on how I suggest expanding those).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a method for training RNNs that is used to extract ordinal sequences.  There are two variations on the training that make this possible.  First, the network is trained on sequences with a wide range of temporal delays, so that only ordinal position is relevant.  Second, the training signal is the location of each sequence on a tree structure given a priori to describe the set of sequences.  \n\nThe model is tractable and interpretable and two classes of findings are described.  First, networks that have learned a tree structure for a particular problem can generalize rapidly to new problems with the same structure by freezing the recurrent weights and relearning the output weights.  Second, the model is tested on time-warped versions of a set of spoken words and it generalizes better than a control model.\n\nSTRENGTHS:\nThe exposition is extremely clear.\n\nWEAKNESSES:\nThe requirement that the training set provides information about the true ordinal structure seems very strict.  \n\nThe requirement for training on a wide range of temporal intervals is a serious limitation as a model of the brain.  \n\nI find the colored lines in Fig 1A,C very difficult to distinguish.\n\nQUESTIONS:\nUnder what circumstances would we expect the network has access to a training signal with information about the ordinal position in a tree structure?  What is the use case for this network?  How could one discover that structure without training it in?\n\nA simple way to build in a model of ordinal position is to have an RNN where the rhs is modulated by a gating factor:\n$\ndx/dt = \\alpha(t) [ \\ldots ] \n$\nwhere $\\alpha(t)$ can be learned.  If $\\alpha(t) = 0$ between relevant triggering stimuli, one can say it's learned an ordinal code.  Is that possible in a GRU? \n\nHow does this model compare in its time warping performance to classic algorithms (e.g., Sekoe & Chiba, 1978)?  On a related note, a recent approach (Jacques, et al., 2022, ICML) shows effectively perfect generalization over a wide range of warping factors without data augmentation.  How does this model relate to that work?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe canonical biological neural circuit model, described by equation (1) in this work, primarily relies on attractor dynamics to perform cognitive tasks involving temporal sequences. Facilitating the emergence of appropriate attractors during training is a difficult task that challenges neuroscientists even today. The authors alleviate this problem by first training RNNs on simple abstract tasks to allow tree-structured attractor templates to appear within the network, and then further training the networks on more complex tasks, such as a key-word spotting task which will effectively reuse the existing templates. This idea is similar to the emerging field of 'schemas' in network neuroscience.  \n\nOn a technical standpoint, I am convinced that this work represents novel and significant advances in the field of biological RNNs, specifically in the subfield of schema formation. However, I have certain doubts on the neuroscientific elements of this work, especially since the paper is thoroughly cited with neuroscience literature. None of these concerns are a major detriment to my evaluation, and hence I recommend an accept for this work. However, I feel that the authors can considerably improve their discussion and neuroscientific arguments regarding the reusage of tree-structured templates. As such, I hope the authors can adequately address my concerns in the weaknesses section to further solidify this work.\n\nSTRENGTHS:\nThe model and training methods have been extensively explained, which makes their objective clear. The underlying intuition that is being conveyed is straightforward and easy to understand, which is the direct consequence of a well-written introduction from pages 1 to 4. Subsequent sections retain the same quality of writing and scientific rigor.\n\nWEAKNESSES:\nAs mentioned before, the key issues that I have about this work stems from neuroscientific plausibility. I do not have specific questions or requests regarding each point, and thus I would recommend the authors simply address this points by arguing for or against these statements.\n\n1. It seems like in this work, the network needs to forget about the previous task in order to use an existing tree-structured template. This seems unrealistic for such a model, and seems to suggest a slot-based usage for such attractors.\n\n2. There exists a lesser-known question in biological RNNs: suppose a network is tasked to remember 3 two-digit numbers (including 00) in sequence. The number of attractors required for this task would be $100^3 = 1,000,000$, which is exaggeratingly large for such a simple task that most humans are able to do. This deters the approach of using tree-structured attractors. \n\n3. Discrete tree-structured attractors are also unable to account for continuous variables, such as color, intensity, speed, temperature etc. Yet, there is no simple generalization for this work to account for continuous variables.\n\n4. Reusing a template also implies that the depth of the tree is fixed. While a tree can indeed grow deeper after subsequent training on more complex tasks, or become shorter due to lack of usage, it does not agree with the logical viewpoint that cognition is highly adaptive to sequence length at much shorter timescales compared to actually learning and developing tree-structures of modified depths.\n\n5. Tree-structures may also lead to unintended behaviors if the wrong inputs are provided, especially if the inputs are not within the scope of the task. This leads to involuntary usage of these tree structures, thus presenting an inherent inflexibility in tree structures.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nI was not able to find explicit statements regarding limitations, although the typical shortcomings of a biological RNNs have been addressed as future research directions in lines 294-309. Points addressed in the weaknesses may possibly be added as limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}