Learning and processing the ordinal information of temporal sequences in recurrent neural circuits
Authors: Xiaolong Zou
Zhikun Chu
Qinghai Guo
Jie Cheng
Bo Hong
Si Wu
Yuanyuan Mi
Venue: NeurIPS 2023 poster (2023)
Keywords: temporal sequence processing
temporal order structure
tree-structured attractor
TL;DR: We trained a RNN to learn a tree-like attractor structure for representing temporal ordinal structure and investigated its computational advantages in temporal information processing.

Abstract: Temporal sequence processing is fundamental in brain cognitive functions. 
Experimental data has indicated that the representations of ordinal information and contents of temporal sequences are disentangled in the brain, but the neural mechanism underlying this disentanglement remains largely unclear. Here, we investigate how recurrent neural circuits learn to represent the abstract order structure of temporal sequences, and how this disentangled representation of order structure from that of contents facilitates the processing of temporal sequences. We show that with an appropriate learn protocol, a recurrent neural circuit can learn a set of tree-structured attractor states to encode the corresponding tree-structured orders of given temporal sequences. This abstract temporal order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. Using a transfer learning task, we demonstrate that the reuse of a temporal order template facilitates the acquisition of new temporal sequences of the same or similar ordinal structure. Using a key-word spotting task, we demonstrate that the attractor representation of order structure improves the robustness of temporal sequence discrimination, if the ordinal information is the key to differentiate different sequences. We hope this study gives us insights into the neural mechanism of representing the ordinal information of temporal sequences in the brain, and helps us to develop brain-inspired temporal sequence processing algorithms.

Learning and Processing the Ordinal Information of Temporal Sequences in Recurrent Neural Circuits
Xiaolong Zou 1,4,5 youxiaoloogg@giiuanlabb com
Zhikun Chu Hhukunzhi@outloor com
Qinghai Guo" guoqinghai www.hingha@hhawii....m @hwawei.
com
Jie Cheng" jiecheng20090gmaii.c com
Bo Hong 1.6
Si Wu siwu@pku edu. cn
hongbo@tsinghua. edu. cn
Yuanyuan M
miyuanyuanetsinghuu edu.
1, Qiyuan Lab, Beijing, China.
2, Center for Neurointelligence, School of Medicine, Chongqing University.
3, Department of Psychology, Tsinghua University
4, School of Psychological and Cognitive Sciences,
Beijing Key Laboratory of Behavior and Mental Health,
IDG/McGovern Institute for Brain Research,
Peking-Tsinghua Center for Life Sciences. Center of Quantitative Biology, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China
Beijing Academy of Artificial Intelligence, Beijing, China.
6, Biomedical Engineering.Schoo of Medicine, Tsinghua University Huawei Technologies.
*: Equal contributions. T: Corresponding authors.
Abstract
Temporal sequence processing fundamental in brain cognitive functions. Ex- perimental data has indicated that the representations of ordinal information and contents of temporal sequences are disentangled in the brain, but the neural mecha- nism underlying this disentanglement remains largely unclear. Here, investigate how recurrent neural circuits learn to represent the abstract order structure temporal sequences, and how the disentangled representation of order structure facilitates the processing of temporal sequences. We show that with an appropriate training protocol, recurrent neural circuit can learn tree-structured attractor dy- namics to encode the corresponding trec-structured orders of temporal sequences. This abstract temporal order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. Using a transfer learning task, demonstrate that the reuse of temporal order template facilitates the acquisition new temporal sequences, these sequences share the same or partial ordinal structure. Using key-word spotting task, we demonstrate that the tree-structured attractor dynamics improves the robustness of temporal sequence discrimination, if the ordinal information the key differentiate these sequences. We hope that this study gives insights into the mechanism of representing the ordinal information of temporal sequences in the brain, and helps us develop brain-ine iain-insspire temporal sequence processing algorithms.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
temporal sequence involves a set items or events that unfold in a specific order over time. Temporal sequence processing is fundamental in many cognitive functions [. such speech recognition, language comprehension, motor control, and memory formation. Effective temporal sequence processing relies on extracting the temporal structure of a sequence, which includes particularly the ordinal information, i.e., the order of events' occurrence. A large volume of studies has suggested that the representations of ordinal information and contents of temporal sequence are disentangled in the brain [1+6]. For instances, studies using single-umit recording in primates have found that significant proportion of neurons the dorsolateral prefrontal and intraparictal cortexes are sensitive to the temporal order of presented visual items, but not to their physical attributes [5]. and similar temporal order representations were also observed in the auditory and linguistic sequence processing 22. The imaging study has shown that the human brain can encode the phonetic order during speech processing, regardless of the phonetic contents (2). Recent large-scale neuron recordings have revealed that the temporal order structure can be robustly encoded in low-dimensional space formed by coordinated neuron population activities 4 The advantage of
disentangled representation of order structure is that it enables the brain to flexibly and rapidly process temporal sequences via conjunctive coding []]. i.c, the brain can casily generate various temporal sequences by combining the temporal order with different contents. This was evident in finger sequence learning task [6), where the brain stored a temporal order template and reused to combine actions to generate various finger action sequences. Understanding how ordinal information is acquired and stored in the brain is crucial for us to understand how temporal sequences are processed in cognitive functions.
Remarkably, the representations of temporal orders related sequences display a tree-like struc- ture (7. For example, in speech recognition, spoken words are represented by temporal sequences formed by primitive syllables or phonemes.
illustrated in Fig the words "water" and "wash" share common syllable "w-ao" at the first item and differ at the second syllable, and so do the words "your" and "year". Thus, the temporal orders of these four sequences form two-layer tree: starting from the root, the words "water" and "wash" form a branch in the first layer they share the same first syllable, and the words "your" and "year" form the other branch; in the second layer, the words "water" and "wash", and the words "your" and "year", are further branched due to their difference at the second syllable. Experimental data has indicated that such rre-structured representation of ordinal information is employed in the frontal cortex support hierarchical motor control (8).
water
wash
your
lsh/
fxx-axri
laxil
M
:
K
/w-aol
D1
ilh-axt
D2
tyl
year
Input: /(t)
RNN
Output: y(t)
Time
D3
Figure 1: (A) A schematic diagram illustrating the tree-structured temporal orders of four spoken words. Words 'water' and 'wash' form branch in the first layer because of the shared syllable 'w-ao', and they are branched in the second layer due to the different second syllables, 'dx-axr' and 'sh'. Similarly, words 'your' and 'year' form other branches.
(B) A schematic diagram of our neural circuit model. It consists of three parts: an input layer of dimension M conveying temporal sequences, a recurrent neural network (RNN) of N neurons encoding the ordinal structure, and an output layer discriminating K different temporal sequences.
Despite that the representation of ordinal information in the brain has been well recognized in the field, the neural mechanism of implementing this representation remains largely unclear 7 number of computational models has been proposed to mimic how temporal sequences stored

---

in neural circuits, and they typically consider that neurons are asymmetrically connected along the path of sequence of network states representing the temporal sequence [2 L5]. Among the proposed models, the framework of heteroclinic channel is inspiring [00, which considers that temporal sequence stored as sequence of saddle points connected by unstable paths the state space of the network, forming the so-called a heteroclinic channel, and under the drive of external inputs, the network state evolves along the channel to activate stored items in the right time order. In most previous works [T] IHIS), the asymmetric neuronal connections accounting for storing temporal sequences are either designed by hand or constructed by applying an asymmetric Hebbian learning rule (note that the application of a Hebbian learning rule implicitly assumes that neural representations of sequential items are known, which does not really model the learning process), and they did not address how neural circuit learns to represent the temporal order structure from data. few studies explored how temporal sequences are learned in a recurrent neural network [[5], but they only considered the simple chain-structured temporal order and did not study the more general tree-structured orders. The machine learning society also studied the learning of temporal sequences (such speech) by using artificial neural networks, but they have not considered the representation of disentangled ordinal information [9]
In this work, we explore how biologically plausible recurrent neural circuits learn to represent the tree- structured orders of temporal sequences and how this disentangled representation of order structure facilitates the processing of temporal sequences. We show that with an appropriate training protocol, a recurrent neural circuit can learn a set of tree-structured attractor states to encode the corresponding tree-structured orders of temporal sequences, where each path from the root to leaf represents pattern of temporal order, with nodes on the path indicating the order hierarchy (see TA). This abstract order template then be bound with different contents, allowing for flexible and robust temporal sequence processing. To demonstrate this advantage, we carry out two experiments. First, in transfer learning task, we show that the reuse of the order template accelerates the acquisition of new temporal sequences, if these sequences share the same or partial ordinal structure. Second, in key-word spotting task, we show that using the tree-structured attractor dynamics improves the robustness of temporal sequence discrimination. if the ordinal structure is the key to differentiate these sequences. We hope that this study facilitates our understanding of how temporal sequences are represented and processed in the brain, and helps us to develop brain inspired algorithms for temporal sequence processing.
2
Learning the ordinal structure of temporal sequences
2.1 The neural circuit model
The model we consider consists of three parts [Fig 3): an input layer which conveys the information of temporal sequences, recurrent neural network (RNN) which stores the ordinal structure of temporal sequences, and an output layer which discriminates temporal sequences. Our idea is that by training the model to perform a temporal sequence discrimination task, the RNN learns to represent the ordinal structure of given temporal sequences in its dynamics. The details of the model are introduced below.
Denote I(t) = {Im (t)), for m 1, 1,...,Au M.. with M the dimension, the activity of the input layer conveying the temporal sequence. The input layer is connected to the RNN through a feedforward connection matrix W = [Win ). for neurons in the RNN. Neurons in the RNN are recurrently connected with each other with matrix Wree- (Wr5e), ). for j 1,.... N, and they also receive a background input Is = (7)). for n = ,....,,,, N. Denote xn (t) the total input received by the rith neuron in the RNN at time t and Tr (t) the corresponding neuronal activity. The dynamics of xn (f) is written as,
1..... . and n I....,N, NN, with N the number of
dr,, d) xt(( Σ Wree
M Σ
Win mm mm (1) + 1a
(1)
dt
where T is the time constant. The nonlinear relationship between xx((t) and (t) is given by rn (1) tanh [x., ()).
The RNN is connected to the output layer through a feedforward matrix Wout [Wext), for 1,..., N and k = I,..., K, with K the number of neurons in the output layer. K is also the
3

---

number of temporal sequences, since each neuron reads out one class of sequences. Denote yıe the activity of the kth neuron in the output layer, whose dynamics is given by,
N
(1:(t)
Σ n=1
Whn' In (t).
(2)
2.2 The training protocol
The details of the training protocol are presented in Supplementary Information (SI). Here, we highlight two settings which are critical for the RNN to extract the ordinal structure of temporal sequences.
Firstly, we augment training data by varying the duration between neighbouring items in temporal sequence (see illustration in Fig ZAg Specifically, this is done by adding noisy signals of varied duration between neighbouring primitive items of the temporal sequence (see more details below). The underlying mechanism intuitively understandable. The augmented data imposes that the model needs to realize invariant recognition with respect to the warping of a temporal sequence, which forces the RNN to extract the abstract ordinal structure of the temporal sequence. This is also biologically reasonable, as in speech recognition [2] 3
the varied speed of spoken words forces the brain to process temporal sequences relying mainly on their ordinal information.
Secondly, we set the target function, i.c., the target activity of a read-out neuron in response to the temporal sequence it encodes, to be continuous in time, and its value increases step by step as items in the temporal sequence unfold one by one (see illustration in Fig A The underlying mechanism is intuitively understandable. Since not all temporal sequences are differentiable by early items, neuronal responses should be low initially, and their values increase along with the exposure of followed items to reflect that sequence recognition is process of evidence accumulation over time. Interestingly. such ramping activity of read-out out neurons agree with the experimental finding on decision-making neurons in monkeys [
After constructing input data and target functions, we apply the standard algorithm, back-propagation through time, train the model, which optimizes the parameters (W'm, Wree, Wout I For details, see Sec. in SI.
2.3 A synthetic task and the model behavior
To demonstrate that our model works, we first apply it synthetic task of discriminating four temporal sequences (K = 4). These four temporal sequences are constructed by combining two of three primitive items with varied orders (like syllables forming spoken words). As illustrated in ZAg the three primitive items are 3-dimensional (M = 3) continuous-time vectors within an interval At. denoted as a [1, 1, 0) of four temporal sequences are expressed S S the first and second items. Since in this study, the exact duration of temporal sequence is not important, for clearance, we also use S to denote sequence having the contents {s* S with unspecified duration. The four classes of temporal sequences are therefore expressed as, S (a. b}. S - {a, c). S {b, a). St (b, c). and their temporal orders form a two-layer tree structure as shown Fig B. augment the training data, we carry out two operations. Firstly, we randomly set the duration between neighboring primitive items to be AT, which satisfies a uniform distribution in the range of [0, ]. Secondly, we add noises in the duration, which are given by
[1. 0. 1]T, and c [0. 1]", respectively. The contents
with k the index of the sequence and
(t( = (t), for = 1, ... . and t E (0, AT',, where (t) denotes Gaussian white noises of zero mean and unit variance, and a the noise strength.
After training, we observe that the model learns to discriminate four temporal sequences in a coarse-to- fine manner. An example of discriminating the temporal sequence S' is presented in Fig Initially, both activities of neurons (red) and 2 (green) increase, since S' and 8 are not differentiable by their first item, i.e., St = S7: while neurons (orange) and (blue) keep silent. When the second item appears, only the activity of neuron keeps increasing, while the activity of neuron 2 drops, indicating that the model recognizes St. The above results can be generalized more than four temporal sequences, Sec. in SI for K 12.

---

2.4
Visualizing the learned tree-structured attractor dynamics
A
b
B
1.0
AT
root
augmenting
Outputs
0.5
b
Targets
k
k
k 4
b
co
Oa
Oc
50
100
Time Step
Time Step
E
variance
100
$3
54
S
S
2
50
# principal component
2
-3
2
S
PC
1
2
5
3,
4
0
Imag
root
S
52
PCd
0
3
-2.5
0.0
2.5
-1
0 pC
PC 1
Real
-2
-3
Figure 2: Unveiling the performance and the mechanism of the model using synthetic data. (A) Illustrating the constructions of temporal sequences and target functions. Two primitive items, (a, b). are combined form the temporal sequence S with an interval AT' inserted between where AT is sampled from uniform distribution in the range of [0, AT Noises are added in the interval. Along with the unfold of two items in S the target function of read-out neuron (red) increases step by step, the target function of read-out neuron 2 (green) first increases and then decreases, and the target functions of read-out neurons (orange) and 4 (blue) always keep silent. (B) The orders of four temporal sequences (S², S S S4) form two-layer tree. (C) An example of the model performance when recognizing the temporal sequence S1. operates in a coarse-to-fin manner. When the first item appears, both the activities of read-out neurons I and 2 increase, since s' and S are not differentiable at this stage; when the second item appears, only the activity of read-out neuron keeps increasing, indicating the model accomplishes the discrimination task. (D) (upper panel) PCA analysis shows that the top 3 PCs explain nearly 90% the variance of neuronal responses in the RNN. (lower panel) The eigenvalue distribution of the attractor state S/.² Fig. (E) The visualization of the structuud neural response trajectories to four temporal sequences, ss', S*), in the reduced space spanned by 3 PCs. Each item in temporal sequence is stored as an attractor in the RNN (marked by cross). (F) The transition probabilities from node S to other states, which are quantified by the eigenvalues of the corresponding Jacobian matrix. The two eigenvectors having the largest two eigenvalues (reflected by their lengths) point to nodes S and S3. The parameters used are given in Sec. F SI.
items,
We carry out intensive analysis to visualize what have been learned in the dynamics of RNN.
2.4.1 Tree-structured representation trajectories
To visualize the high-dimensional dynamics of the RNN (N = 50), we apply principle component analysis (PCA) to reduce its dimensionality. PCA is done by using the activities of RNN neurons in response to 1000 testing temporal sequences (for details, see Sec. B.I in SI). It turns out the top 3 PCs explain nearly 90% of the variance of neuronal responses (Fig (Fig2p, upper panel), indicating that after training, the RNN performs computation in a low-dimensional subspace. By projecting RNN activities onto top 3 PCs, we obtain the trajectories of the network state responses to the four temporal sequences (averaged over 25 trials), and observe that they display tree structure, agreeing with the ordinal structure of the learned four temporal sequences (Fig 2 trajectories for different sequences marked by different colours).
5

---

2.4.2 Tree-structured attractor states
We continue to analyze the state space of the RNN, and find that it has totally 7 stable points (attractors), which are the nodes of the tree: the root, branching points, and end points (marked by crosses in 2ig indicating that primitive items are stored as attractors in the RNN. The detailed analysis is presented in Sec. B.2 in SI. Here, we sketch the main idea. We first use an optimization method to find out all fixed points of the RNN dynamics 4. which turn out to coincide with the 7 nodes of the tree. We then analyze the stability of each node. An example of stability analysis of node $1.2 is depicted in Fig 2D (lower panel), which shows that the real parts of all eigenvalues of the Jacobian matrix are small than 1, indicating that the node is stable with respect to noise perturbation.
2.4.3 Tree-structured attractor dynamics
We further analyze the transition dynamics of attractor states (nodes). The detailed analysis is presented in Sec. B.3 in SI. Here, we sketch the main idea. Consider node 5 the first layer of the tree (Fig 2)) We linearize the RNN dynamics around the node and compute all eigenvectors of the Jacobian matrix. The larger the eigenvalue is, the more unstable the network state along the eigenvector under noise perturbation. As shown in Fi the most two unstable directions (the length of eigenvector reflecting the eigenvalue) point to nodes S} and S in the second layer, respectively. This implies that starting from the node S the network state can travel to the node S/2 or 53 depending the second item, but has difficulty travel to other nodes.
In summary, the above results reveal that after training, the tree-structured attractor dynamics emerges in the RNN, which encode the ordinal structure of the learned temporal sequences. Upon receiving temporal sequence, the network state evolves from the root to the leaf, node by node, following the order hierarchy of the temporal sequence. Notably, unlike the framework of heteroclinic channel, our model stores primitive items as stable attractors rather than saddle points, and the transition between two nodes requires the followed item to be the input drive. As demonstrated below, this attractor dynamics enables our model to discriminate temporal sequences robustly against temporal warping.
3 Processing temporal sequences with the ordinal structure
3.1 Disentangled representation of ordinal structure facilitates transfer learning
The key advantage of disentangled representation of order structure is that it allows flexible generation to new temporal sequences via conjunctive code, that is, the neural system can combine the stored order template with different contents to form different temporal sequences. In our model, the RNN learns extract the abstract ordinal structure of temporal sequences in the form of tree-structured attractor dynamics. After training, the association between the RNN and the training data is released, and the RNN can be reused to encode new temporal sequences having the same or partial temporal structure. Mathematically. this can be done by freezing the recurrent connections in the RNN (and hence the tree-structured attractor dynamics), while only learning the feedforward connections from the input layer to the RNN and from the RNN to the output layer, when a new temporal sequence processing task arises. In such a way, it expected that the pre-trained RNN helps the model to speed up the acquisition of new temporal sequences having the same or partial ordinal structure, achieving the so-called transfer learning.
To demonstrate the above idea, we design a transfer learning task. We consider that the RNN has extracted the ordinal structure of the synthetic data described in Sec.2.3, and now apply the model to solve a new task of discriminating new four temporal sequences. These four temporal sequences are composed by three primitive phonemes chosen from the TIMIT dataset, which are "pel", "tel". and "pau" " (for the details of sequence construction, see Sec. Sec.[[]in SI), and the four sequences are written as "pcl-tel", "pel-pau", "tcl-pel" and "tcl-pau", which have the same order structure as the synthetic data BB). Following the same procedure as described in Sec.2.2, we augment the training data by inserting time intervals between neighboring items, and choose their ramping target functions accordingly. During transfer learning, we freeze the recurrent connections in the RNN and only optimize the feedforward connections. As comparison, we also train the model without freezing the RNN. The results are shown in Fig 3BB We observe that: 1) after transfer learning, the RNN retains the tree-structured representation trajectories for the learned temporal sequences, indicating that the model adjusts the feedforward connections to match the tree-structured attractors of the

---

A
Leamed Dynamics
New Sequence
Template
PC
root
Ordinal Structure
PCa
Discrepancy 1o to
root
PC
1
Template Dynamics
palp
tcl
O
PC 3
10
20
30
40
Training Epoch
Zol
pau pal
pau
PC2
PC
Aesuray (%)
100
rood
Partial Reuse
D
Test Accuracy (95)
100
80
tal
60-
with Sandfor
pau
pel
Aau
40
40
sonsfer
20
40
80
40
Training Epoch
IN
Training Epoch
Figure 3: Disentangled representation of ordinal structure facilitates transfer learning. (A-B) Illustrat- ing a transfer learning task, which new temporal sequences have the same ordinal structure as the template. (A) From left to right. Four new sequences constructed by using three primitive items (phonemes), and they have the same tree-structured orders as the template stored in the RNN. By freezing the recurrent connections in the RNN, the model learns the same tree-structured representa- tion trajectories as the old ones, and their discrepancy is negligible after training (for the measurement of the discrepancy, see Sec. in ST). (B) The learning of the model is accelerated significantly compared to the case of no transfer learning. (C) A schematic diagram of transfer learning task, in which the ordinal structure of new temporal sequences only covers part of the template. (D) The learning of the model in (C) is accelerated significantly compared to the case of no transfer learning. For the details of model parameters and performances, see Sec. in SI.
RNN (note that freezing recurrent connections in the RNN does not ensure that the representation trajectories are also fixed); 2) the speed of transfer learning accelerated significantly compared to that without freezing the RNN.
We further consider a transfer learning task, which the ordinal structure of new temporal sequences is only part of the order template stored in the RNN (more details see Sec. Cln SI). Firstly, we create a task of discriminating eight sequences, which are constructed by the synthetic primitive items given in 2.3. They are: S' [a, b, a), S = (a, c) S = fa. a). S {a. c. b}. 5 (b. a, b). S (b, a, c}. S [b, [b. a), S [b. cb. We train the RNN learn the three-layer tree structure of the temporal orders of these sequences. We then design new task of discriminating six temporal sequences formed by the above three primitive phonemes, which are "pel-tel-pel", "pel-tel-pau., "pel-pau-pell, "pel-pau-tel", "tel-pau" and "tcl-pel", and their ordinal structure only cover part of the stored template in the RNN (shown in Fig denoted by red line). Following the same transfer learning procedure described above, train the model to learn new sequences, and observe that 1) the model can reuse part of the stored template to represent new sequences; 2) the model speeds up the learning process significantly compared to the case of no transfer learning (Fig BD).
In addition to the aforementioned, we also conduct an additional experiment to demonstrate that our model can also combine and reuse primitive tree attractor structures to rapidly solve longer sequence task, shown in Fig. S8 Overall, our study shows that disentangled representation of ordinal structure in recurrent neural circuits can be efficiently combined and reused to facilitate transfer learning, new temporal sequences have the same or partial ordinal structure.
7

---

A
W-ao'
hhv'
y-ih'
-5
O
-5
'dx-axr'
'sh'
'eh-dcl'
'axr'
PO2
4
0
water ['waao' :dx-xxr]
had ["hv',ee--de "eh-ddl]] year ['y-ih',axr"]
5
4
PC
1
wash ['w-ao', 'sh']
D
k=
k=3
0.6
k-2
km4
Accuracy (%)
85
Channels
Outputs
F=1
10
0
an
70
Our Model
F=
2
10
2
3
20
40
60
80
100
0
20
40
60
80
100
Temporal Warping
Time Step
Time Step
Figure 4: Robust key-word spotting with the ordinal structure. (A) The tree-structured orders of four words, 'water', 'wash', 'year", 'had'. (B) The visualization of the learned rree-structured neural representations of four temporal sequences in the reduced state space of the RNN. (C) Examples of the model discriminating the word 'water' with different warping effects (F 0.6, 2). Notably, the time for the model recognizing the word also stretched or compressed accordingly. (D) With respect to temporal, our model achieves robust performances; whereas, control model which does not follow the training protocol proposed in this work (and hence does not learn the tree-structured attractor dynamics) fails to do so. For the details of model parameters and performances, see Sec. D in SI.
3.2 Robust temporal sequence discrimination using ordinal structure
A hallmark of temporal information processing in the brain is that the neural system can recognize temporally stretched compressed signals, known as temporal warping, as the same pattern [20). It remains unclear how neural circuits in the brain achieves this goal. Here, we argue that storing ordinal structure as tree-structured attractor dynamies in recurrent neural circuits provides a feasible solution.
To test this hypothesis, we apply our model real-world application called key-word spotting. Specifically, we design a four spoken words discrimination task, where the four words, "water", "wash", "year", and "had", chosen from the TIMIT dataset. For each word, 10 original temporal sequences are randomly sampled from the dataset, and each sequence is chunked into two primitive items by syllables or phonemes, for example, the word "wash" is chunked into "w-ao" and "sh". Overall, the temporal orders of these four words form two-layer tree (Fig training protocol, we first augment training data by inserting varied intervals between neighboring primitive items and construct their target functions accordingly (for details, see Sec. A in SI). We then train the model extract the ordinal structure (see Fig BB). testing, we present temporal sequences of four words without inserting separation intervals, rather we stretch or compress the original temporal sequences to reflect the temporal warping effect practice (for details, see Sec. [ in SI). We use variable F to denote the warping effect, with F representing no warping, F >>> stretching, and F compressing. The results are shown in Fig We observe that: 1) the RNN extracts the ordinal structure of four temporal sequences; 2) the model realizes word discrimination
(FigMA). 4A). Following the same

---

in coarse-to to-see-o--fin manner, 3) the mode achieves robust performance over a wide range of temporal warping.
4 Conclusion and Discussion
The brains is good at discovering and utilizing abstract representations, known "schemas", from daily experiences. These schemas are suggested to be stored in the prefrontal cortex, and can be reinitialized and reused to afford efficient transfer learning to new tasks [21 p2] However, there is still lack of deep understanding of how these schemas are represented and reused in neural circuits in the brain. In this work, we study the representation and reuse of a specific schema, the ordinal structure of temporal sequences, in neural circuits. We show that with an appropriate training protocol, recurrent neural circuit can learn rrc-sstruttrre attractor dynamics to represent the ordinal structure of temporal sequences. This abstract order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. We showcase this advantage by using a transfer learning and a key-word spotting tasks.
Our model may have some far-reaching implications to the representation of abstract schemas in the brain. Recently, researchers have conducted interesting studies on the format of neural schemas. For example, Guntupalli et al. proposed graph model [23]. Similar to our work here, by learning new associations between new sensory features and schema representations, this graph model can utilize common abstract knowledge to facilitate transfer learning. However, their schema representations employ a high-order Markov dynamic model rather than the attractor dynamic model, which raises the question of how the brain biologically represents such schema representations. On the other hand, Goudar et al. trained a recurrent neural network model on various common sensor-motor mapping tasks and discovered a low-dimensions attractor dynamics which emerges and acts as an abstract structure to accelerate new peciision-mainng tasks (24). In contrast to Goudar's work, our research primarily focuses on abstract temporal order representation in the context of real-world task involving key-word spotting. This task provides a more intricate and complicated setting for exploring abstract schema representations than what has been considered in previous studies. Overall, studying various attractor dynamics for different schema representations in the neural system would offer fascinating avenues for future research.
Although the focus of the present study is on unveiling the biological mechanism for representing the ordinal structure of temporal sequences, the computational principle we have gained can be applied to other brain-inspired computations. To test this idea, we replace the RNN in the model with two popular recurrent networks in machine learning, which are GRU and LSTM. Applying to the key-word spotting task, we observe that (Table. I): if the same training protocol proposed in this work is used, both GRU and LSTM achieve good performances; while if the conventional training protocol (i.c., no data augmentation and using the cross-entropy loss function) is used, they can still discriminate original temporal sequences but fail to be robust temporal warping. These results suggest that our method (including the model structure and the training protocol) is general and has potential to be applied to brain- mean-insspreed computing.
Table I: Model performances using different recurrent networks and different training protocols in a key- word spotting task. We replace the RNN in our model by GRU LSTM. We train the model fol- lowing either the same training protocol used in this work (marked by*) or the conventional protocol, i.e., data augmentation and the cross-entropy loss function. Both models are trained on the dataset without warping (F = 1) and tested on dataset with warping (F = 0.6, 1.0, 1.4, 1.8, 2.2, 2.6, 3.0). The classification accuracy of each model is obtained by averaging over 8 trials with randomly split training and testing data, as well as network initialization. For each task, we have 10 samples per class in the train dataset and 300 samples per class in the test dataset. The word sequences are sampled from the TIMIT dataset. For more details, see Sec. D Fin in SI.
classes
classes
12 classes
Model
no warping
warping
no warping
warping
no warping
warping
GRU
94.0% + 2.5 88.9% + 6.2 95.1% + 2.5 87.9% + 7.6 94.3% + 1.2 84.4% 9.0
GRU
97.4% + 1.5 96.6% + 1.8 97.2% + 0.6 95.6% + 1.8
96.4% + 0.9 94.8% = 1.8
LSTM
94.1%
2.2 82.8% + 13.7
93.8%
+ 1.6 80.5% + 12.3
92.1%
+ 0.7 75.9% + 14.8
LSTM2
95.1% + 1.8 92.5%
94.0% + 0.8 00.1%
92.6%
2.1 89.5%

---

In order to learn a tree-like structure in our model, we require precise ordinal details about continuous sequences. Several mechanisms can be employed in our brain to access this kind of information. Firstly, a large volume of experimental studies has shown that in the brain, continuous sequences are often chunked into discrete items to support high-level cognition [25]. For examples, speech sequences can be hierarchically chunked into words and syllables M1 neurons in the hippocampus have been shown to detect event boundaries when watching continuous movie videos [26]. This chunking/eegeentation process naturally outputs the ordinal structure of temporal sequence. Addi- tionally, computational models have been proposed for the brain performing sequence chunking, such as self-supervised learning (27] and oscillation (28). Secondly, the development path of our brain also indicates the brain can naturally acquire sequence chunking. For example, in language acquisition, young children learn primitive phoneme categories around 2 months [9);; around 7 months, they learn sequences of either an ABB or ABA form [30]. which can be represented as tree-like attractor structures in our model; subsequently, children learn recognize spoken words around 12 months. Thus, in language acquisition, phoneme learning serves as a building block for word learning, which defines the ordinal structure of language sequences.
Incorporating a wide range of diverse temporal intervals presents another challenge for our model. How does the neural system handle this in biologically plausible manner? Firstly, in our brain, motor and speech sequences are often generated with a large variability in speed, and hence they exhibit large variability in separations between motor motifs and speech chunks. This large variability in separations enables the brain to learn the tree-like attractor structure, as demonstrated in our model. Secondly, in our network training, we do not really need very large intervals. For the clean synthetical data, we can actually train the tree-structured attractors using fixed interval values, shown in Fig. (STOAA For the noisy spoken words, do need amount of variations achieve good performances, but the range is only about 2 times the item length. Overall, the requirement for a wide range of temporal intervals can be largely relaxed in practice.
Finally, there are several interesting directions worth exploring further. First, our current model is currently limited to small sequences. Figuring out how to effectively expand and scale our approach for more complex scenarios, like sequences involving large classes and longer sequences, will be a crucial focus for our research. Second, in the context of transfer learning, we employ a stochastic gradient descent method to learn the binding between sensory features and abstract representations. However, this technique is not biologically plausible and efficient. Experimental studies have shown the existence of a fast time-scale Hebbian rule in the brain, such as in the hippocampus [3I, which is suggested to mediate rapid binding of distinct neural populations. By incorporating this fast Hebbian rule, our model could learn new associations between contents and the ordinal structure in an one-shot manner. Third, beyond ordinal structure and contents, the timing structure in temporal sequence is also crucial for some tasks. Recent experiments have shown that the human brain could integrate abstract order template. timing structure, and contents generate diverse finger action sequences (6). Thus, integrating the timing structure into our model could potentially enhance its power for processing more complicated temporal sequence processing tasks. Last, the ordinal structure also serves as the basis for some more high-level abstract structures, such as algebraic patterns in motor sequence representation and recursive syntactic structures in language representation [ (22. How these abstract representations are concretely represented in neural circuits largely unknown. Our model may serve as starting point for exploring these mysteries.
Acknowledgments and Disclosure of Funding
This work supported by the National Natural Science Foundation of China (NO: T2122016. Y.Y.Mi). National Science and Technology Innovation 2030 Major Program (No. 202 2021ZID003305, Y.Y. Mi, No. 2021ZD0200204, Si Wu).
10

---

References
[1] Dehaene, S., Meyniel, F., Wacongne, C., Wang. I.., Pallier, C. (2015). The neural representation of sequences: from transition probabilities algebraic patterns and linguistic trees. Neuron, 88(1), 2-19.
[2] Gwilliams, L.,, King. J. R., Marantz, A., Poeppel, D. (2022). Neural dynamics of phoneme sequences reveal position-invariant code for content and order. Nature communications, 13(1), 6606.
[3] Yi, H. G., Leonard, M. K., Chang. F. (2019). The encoding of speech sounds in the superior temporal gyrus. Neuron, 102(6), 1096-1110.
[4] Xie, Y., Hu, P., Li, J., Chen, J., Song, W., Wang, X. & Wang. L. (2022). Geometry of sequence working memory in macaque prefrontal cortex. Science, 375(6581), 632-639.
[5] Ninokura, Y.. Mushiake, H., Tanji. (2004). Integration of temporal order and object informa- tion in the monkey lateral prefrontal cortex. Journal of neurophysiology. 91(1), 555-560.
[6] Kornysheva, K., Bush, D., Meyer, S. S., Sadnicka, A., Barnes, G., & Burgess, N. (2019). Neural competitive queuing of ordinal structure underlies skilled sequential action. Neuron, 101(6), 1166-1180.
[7] Pitti, A., Quoy, M., Lavandier, C.,, Boucenna, S., Swaileh, W., & Weidmann, C. (2022). In search of neural model for serial order: brain theory for memory development and higher level cognition. IEEE Transactions on Cognitive and Developmental Systems, 14(2), 279-291.
[8] Badre, D., & Nee, D. E. (2018). Frontal cortex and the hierarchical control of behavior. Trends in cognitive sciences, 22(2), 170-188.
[9] López-Expejo, I. Tan, Z. H., Hansen, J. H., & Jensen, J. (2021). Deep spoken keyword spotting: An overview. IEEE Access, 10, 4169-4199.
[10] Rabinovich, M., Huerta, R., & Laurent, G. (2008), Transient dynamics for neural processing. Science, 321 321 48-50.
[11] Yildiz, I. B., von Kriegstein, K., & Kiebel, S. J. (2013). From birdsong to human speech recog- nition: Bayesian inference on a hierarchy of nonlinear dynamical systems. PLoS computational biology, 9()), e1003219.
[12] Gillett, M., Pereira, U., & Brunel, N. (2020). Characteristics of sequential activity in networks with temporally asymmetric Hebbian learning. Proceedings of the National Academy of Sciences, 117(47), 29948-29958.
[13] Kiebel, S. J., von Kriegstein, K., Daunizeau, J., & Friston, K. J. (2009). Recognizing sequences of sequences. PLoS computational biology. 5(8), e1000464.
[14] Cone, I. Shouval, H. Z. (2021). Learning precise spatiotemporal sequences via biophysically realistic learning rules in modular, spiking network. Elife, 10, e63751.
[15] Frölich, S., Marković, D., & Kiebel, S. J. (2021). Neuronal sequence models for Bayesian online inference. Frontiers in Artificial Intelligence, 4, 530937.
[16] Wang, X. J. (2008). Decision making in recurrent neuronal circuits. Neuron, 60(2), 215-234.
[17] Okazawa, G., & Kiani, R. (2023). Neural Mechanisms that Make Perceptual Decisions Flexible. Annual Review of Physiology, 85, 191-215.5 S
[18] Shadlen, M. N., & Newsome, W. T. (1996). Motion perception: seeing and deciding. Proceedings of the national academy of sciences, 93(2), 233-633.
[19] Sussillo, D., & Barak, O. (2013). Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3), 626-649.
[20] Lemner, Y., Honey, C. J., Katkov, M., & Hasson, (2014). Temporal scaling of neural responses to compressed and dilated natural speech. Journal of neurophysiology. 111(12), 2433-2444.

---

[21] Tse. D., Langston, R. F., Kakeyama. M., Bethus, I., Spooner, P. A.. Wood, E. R., & Morris, R.
G. (2007). Schemas and memory consolidation. Science, 316(5821), 76-82.
[22] Zhou, J., Jia, C., Montesinos-Cartagenn M., Gardner, M. P., Zong, W., Schoenbaum, G. (2021). Evolving schema representations in orbitofrontal ensembles during learning. Nature, 590(7847), 606-611.
[23] Guntupalli, J. S., Raju, R. V., Kushagra, S., Wendelken, C., Sawyer, D., Deshpande, ... & George, D. (2023). Graph schemas abstractions for transfer learning, inference, and planning. arXiv preprint arXiv:2302.07350.
[24] Goudar, V., Peysakhovich, B., Freedman, D. J., Buffalo, E. A., & Wang, X. J. (2023). Schema formation in neural population subspace underlies learning-to-eean flexible sensorimotor problem-solving. Nature Neuroscience, 1-12.
[25] Dehaene, S., Al Roumi, F., Lakretz, Y., Planton, S., & Sablé-Meyer, M. (2022). Symbols and mental programs: a hypothesis about human singularity. Trends in Cognitive Sciences.
[26] Hahamy, A., Dubossarsky, H., & Behrens, E. (2023). The human brain reactivates context- specific past information at event boundaries of naturalistic experiences. Nature neuroscience, 1-10.
[27] Asabuki, T., & Fukai, T. (2020). Somatodendritic consistency check for temporal feature segmentation. Nature communications, 11(1), 1554.
[28] Giraud, A. L., & Poeppel, D. (2012). Cortical oscillations and speech processing: emerging computational principles and operations. Nature neuroscience, 15(4). 511-517.
[29] Kuhl, P. K. (2004). Early language acquisition: cracking the speech code. Nature reviews neuroscience, 5(11), 831-843.
[30] Marcus, G. F., Vijayan, S., Bandi Rao, S., & Vishton, P. M. (1999). Rule learning by seven- month-old infants. Science, 283(5398), 77-80.
[31] Bittner, K. C., Milstein, A. D., Grienberger, C., Romani, S., Magee, J. C. (2017). Behavioral time scale synaptic plasticity underlies CA1 place fields. Science, 357(6355), 1033-1036.
[32] Shima, K., Isoda, M., Mushiake, H., & Tanji, J. (2007). Categorization of behavioural sequences in the prefrontal cortex. Nature, 445(7125), 315-318.
12

---

Appendices
A
The Training Protocol
"w-ao"
"dx-axr"
A
a
Inputs
Inputs
O3
01
02
21
01
0.5
init
{a
Onii
water
0.0
1.0
(a.b.c)
Osile
wash
Osile
0.0
100
(a,c,a)
had
0.0
1.0
(a.c,b)
(t(t)
0.0
(t(
year
2
0.5
(b,a,b)
carry
0
21
0.5
(b,a,c)
1
oily
1.0
21
(b,c.a)
dark
0.0
1.
21
(b.c.b)
suit
0.0
10
15
20
20
40
80
100
Time Step
Time Step
Figure S5: Illustration of the augmentation of temporal sequences and the setting of ramping target functions. (A) An example temporal sequence, b a, in the synthetic task with K 8, which used to train the model shown in Fig.3C of maintext. The upper panel is augmented input stimuli by inserting duration and adding Gaussian noise, the lower panel is ramping target functions of readout neurons. (B) An example sequence, "water", in the key-word spotting task with K = 8 (used in Tab. of maintext ). which includes the follwing words: "water, wash, had, year, carry, oily, dark, suit". The upper panel is augmented MFCC feature vector of "water", the lower is the corresponding target functions read-out neurons.
To train the model, first augment training data and construct their target functions (see illustration in Fig.22 2A of maintext), after that we apply packpropagation-thoou----ii optimize the model parameters (Won Wiree Wout I³). Denote (t) the actual model output sequence and ýtt) the corresponding target function. The loss function is given by
(BPTT) algorithm to
L
TB ELE
v2(1) =2)())
where b = 1...., denote the index of temporal sequences in the training data, T₁ the length of the bth temporal sequence, and k 1...., K the index of the output neuron. We adopt the ADAM optimizer, with the learning rate varying in the range of = [le le 12, and the decay rates the first and second moments are 0.9 and 0.999.
Inspired by the ramping activity decision making neurons observed in the experiment 1+ we set the target function of read-out neuron increases with the unfold of the items in the temporal
13

---

sequence this neuron encodes. Consider the temporal sequence of the first class, denoted as S' is presented to the network. The target functions of read-out neurons are constructed as below.
(1) The task of discriminating four synthetic temporal sequences (K = 4), Fig.2 2 in the main text.
In this task, S consists of two consecutive items {S). 522 with the duration AT, as shown in Fig. in the main text. Before the temporal sequence is presented, all read-out neurons are at the spontaneous state, whose target functions are set to be U Oinit, for k = 1,.., K. When the first item Si appears, both the target functions of neuron and 2 increase to U1.2 = O2, and the target functions of other neurons decay to the silent state, Us, Ozile. This lasts for the duration AT, until the second item appears. When Sj appears, the target function of neuron 1 further increases to ý1 O2. while the target function of neuron 2 decreases to Osites and the target functions of other neurons keep the same, i.e. 32,3,,1 Osile- In this task, the parameters used are Oniit 0.25,
O = 0.5, O2 I, Osite = O.
For other tasks of discriminating four temporal sequences, the target functions are designed similarly,
(2) The task of discriminating eight synthetic temporal sequences (K 8), Fig.3C in the main text
In this task, S' consists of three consecutive items { ..2.SS) S3}} with the duration AT, as shown in Fig.3C in the main text. The design of target functions is illustrated in Fig.S1 A, similar to the task with K = 4. Before the sequence is presented, all target functions of all read-out neurons are set to be ÿk- = Oinit for k- = 1....,& 8. When the first item Si appears, the target functions of neurons 1 - 4 increase to be iss = O₁, for k = 1, 2, 3, 4, since the four neurons share the same first item; while the target functions of other neurons decay to the silent state, i.c., Dk Osite for k = 5, 6, 7,8. 8. When Si appears, the target functions of neurons
keep increasing to ÿ) = O2 for k = 1, while the target functions of neurons 3 4 decay silent, and the target functions of other neurons keep silent, ie., Oh= = site, for k = 3, .... 8. When S appears, only the target function of neuron keep increasing ý1 O3, while all other neurons are in the silent state, i... DK O for l = 2, .... 8. The parameters used are Oimit = 0.25, O = 0.5, O₂ = 0.75, O₂- = 1, and Oxile = 0.
For other tasks of discriminating eight temporal sequences, the target functions are designed similarly.
(3) The key-word spotting task of discriminating eight or twelve temporal sequences (K 8, 12). Table 1 in the main text.
In this task, 1 denotes the temporal sequence of the key word 'water', which is chunked into two different items by syllables, i.e., 'w-ao' and 'dx-axr'. The training data is augmented according to the learning protocol described in the main text, as illustrated in Fig.SIB S1B (upper panel). Before the sequence is presented, all read-out neurons are at the spontaneous state, i.e., ÿk Oini, for k == ..... K, with K- 8, 12. When the first item ('w-ao') appears, the target functions of neurons 1 - 2 increase to O,, while the target functions of other neurons decay to Osile- When the second item appears, only the target function of neuron 1 keep increasing O2, while the target functions of other neurons are, U/ = Osite for k = 2, 2.... K. The design of target functions for S in 8-classes
task is illustrated in FIB (lower panel). The parameters used are O 0.25, O₁ = 02,02 = 2.
Osite 0.
B
The Visualization Method
B.1 Principle Component Analysis (PCA) of neural activities in the RNN (Sec.2.4.1 in the Main text)
We apply PCA to find out the low dimensional state space which captures the major variance of neural activities in the RNN. Denote (1) the activity vector of the RNN in response to the kthh temporal sequence at moment t, for = L..., K. whose element is given by (t) = rt). for i = 1,..., N, with N the number of neurons in the RNN. Combining the activity of the RNN at all moments, we obtain a martix R = (r* (t)). whose dimension is N x T. with T the length of kth temporal sequence. Define the covariance matrix X = ΣN
No Σ ii= 2k=1 (R* < R (R < R >T,, where < R > is obtained by averaging activity vectors over time and the total testing trials, x Nr. with N₂ the number of trails in each class. We apply PCA to the matrix X and obtain set of eigenvectors with decreasing eigenvalues. The eigenvalue of each PC reflects the variability of RNN response over time and trials along the corresponding eigenvector. As shown in Fi 2D in the main text, the first
14

---

3 PCs account for nearly 90% of the variance of RNN activities. On the basis of this analysis, we project the RNN activities onto the first 3 PCs to visualize the evolution of the RNN state over time for different temporal sequences, as shown in Fig.2E in the main text.
B.2 Stability analysis of tree-structured attractors (Sec.2.4.2 in the main text)
To uncover the structure of the state space of the learned RNN, we firstly find fixed points of the RNN dynamics, and then analyze their stability.
(I) Finding fixed points in the RNN dynamies
Identifying fixed points or slow points the dynamies of the RNN helps us understand the dynamical properties of our model. Specifically, fixed point is state that remains unchanged over time to a constant input, while a slow point is state that changes very slow over time to a constant input. The dynamics of the RNN is given by rdx(t)/dt which can be approximately solved using the first order Euler method with the simulation time step At = 1. and the neural activity r(t) at time + 1 can be approximately written as r(t + 1) F(r(t), 1))
-x(t) + Wte + I() + I
tanh ((1 At/T) arctan (r(t)) + At/r(W"rr(t) + W'][()) + I))), where F(-) the state update function. Denote r fixed point in the state space of the RNN, which satisfies the equation, r* F(r", I) for a particular input I. To get a better understanding of the network dynamies, we are also interested in slow points, which approximately satisfy r' as F(r", I).
To identify these fixed or slow points, we follow the approach developed by Sussillo and Barak [4] by solving an optimization problem. We focus only on the fixed and slow points when the network receives no external input (i.e., = 0). We define as an variable describing the speed at which the network state evolves over time, which is given by,
1 2 F -
(4)
The smaller q is, the slower the network dynamics evolves. To optimize over the RNN state r, we use BPTT and initialize from many conditions sampled from the distribution of network states during training. The neural states satisfying < 10-7 are considered to be fixed points. For the synthetic task as shown in Fig. 2 in the main text, identify fixed points in the RNN dynamics.
(2) Stability analysis of fixed points
For each fixed point r", we analyze its stability using the perturbation method []] Specifically, we apply perturbations to a fixed point r", denoted as Ar(t) and AT(t), respectively. The network state at & + can be updated with the first-order Taylor expansion, which is given by,
r(t 11 = F(r + Ar(t), + AT())) = F(r (') J + J +J"PAAI(().
(5)
where {J Jimp) are Jacobian matrices given by (x Ji (r",I*)) aF(r.)) S1, ( p.)
OF
((r)))) and Øry
The eigendecomposition of the recurrent Jocabian matrix Jee can be expressed as rec = RAL. where R and L matrices that contain the right and left eigenvectors of J'ee respectively, and A is a diagonal matrix containing complex-valued eigenvalues. The value of the real part of an eigenvalue determines the stability of the corresponding eigenmode, with those smaller than 1 representing stable modes and those larger than representing unstable modes. As shown in (ig the maximums of real parts of all eigenvalues of each fixed point are all smaller than 1, indicating that all these fixed points are stable attractors of the RNN.
B.3 Analyzing the transition dynamics of attractor states (Sec.2.4.3 in the main text)
We take the transitions from stable points s,1.2 to SP and $2 as examples to illustrate the transition dynamics of the RNN. We combine the network activity trajectories in response the kth class temporal sequence, for k 1, ..., K, denoted as R² {r² : t2)], construct the matrix X as in See B.1 of SI. where t and to denote the moments of the appearance of item S7.2 and the ending of
the sequence, respectively. Define the covariance matrix X En Σ (R << R (f
R >)", where R > is the average of network activities over time from to to and over K x No.
15

---

A
root
B
C
S2
Imag
Imag
Imag
.0.0.
mto
0
0
Real
Real
Real
2
5
S
E
Imag
Imag
Imag
0 Real
Real
Real
Figure S6: The distributions of complex eigenvalues for six attractor states of the RNN used in the synthetic task. The result for the other fixed point is shown in Fig.2D.
trails, with NE the number of trail in each class. We perform PCA on the matrix X and obtain set of cigenvectors with decreasing eigenvalues. The first two eigenvectors (denoted as PC and PC2) account for the major variance of neural responses.
We further analyze the right eigenvectors of the Jocabian matrix J around the stable fixed point $7.2. The larger the eigenvalue is, the more unstable the network activity state is along the eigenvector under noise perturbation. We project the right eigenvectors onto the hyperplane spanned by PC1 and PC2, with their lengths determined by the corresponding eigenvalues. We observe that the two most unstable directions among all eigenvectors point to nodes 8 and S2. as shown in Fig.2E-FF. This indicates that it is much easier for the network state transferring from node s1.2 to nodes S and S than to other states.
C Transfer Learning Tasks
C.1 The construction of phoneme Sequences (Sec.3.1 in the main text)
The three primitive items, 'pel', 'tel' and 'pau' used in Fig.3 in the main text, are randomly selected from the TIMIT dataset (6). We transform the raw waves of three phoneme items into Mel-frequency cepstral coefficients (MFCC) with a window size of 25ms and stride of 10ms, roughly mimicking the information processing in the cochlea in the brain. The obtained MFCC feature vectors have 16 channels, and the MFCC vectors are further normalized by subtracting their mean and dividing by their standard deviation, along the time dimension, which are used as inputs to the RNN. Using the four-class classification task as an example, as shown in Fig.2B in the main text, we construct four temporal sequences, which are "pel-tel", "pel-pau", "tcl-pel", and "tcl-pau". We augment these sequence by inserting varied intervals between primitive items and adding Gaussian noises. We then train the model using ramping target functions. In the task, the total number of augmented data is 10000, which are randomly divided into 8000 training and 2000 testing examples.
C.2 The performance measurement (Fig.3A in the main text)
In transfer learning. the model learns allocate phoneme contents to the pre-trained tree-like attractor template. To evaluate the effect of reusing the ordinal structure, we calculate the discrepancy between the learned attractor structure and the template, measuring by the Euclidean distance between them. temporal chunk. If the model learns to reuse
Each item and its followed duration can be seen the pre-trained attractor dynamics, the last state of each temporal chunk should exactly stay the corresponding attractor state. To test this, as an example, for the sequence S' {S}, S in the four-class discrimination task, the learned state vector of S item obtained by averaging the state vectors over the last five time steps before the item $J is presented. Similarly, the learned state vector of S item is obtained by averaging the state vectors over the five time steps before the end the sequence. The learned state vectors of other sequence items computed similarly. After obtaining
16

---

A
B
100
4
Accuracy (%)
Discrepancy
80
3
2
60
1
our model
40
control model
0
20
40
0
20
40
60
Training Epoch
Training Epoch
Figure S7: (A) The discrepancy between the learned dynamics and the template dynamics in the task of partially reusing the ordinal template. (B) In the transfer learning scenario, our model exhibits significantly accelerated learning compared the control model using the cross-entropy loss.
these learned state vectors, we calculate the Euclidean distance between them and the corresponding tree-structured attractor state vectors in the template. The final discrepancy is computed by averaging the results of seven attractor states. As shown in Fig.3A and Fig.SS7 as the training progresses, the discrepancy between the learned attractor states and the template attractor states decreases gradually to zero, indicating that the model learns to allocate the input contents to the ordinal template stored in the RNN.
In transfer learning setting, We also compare our model with tree attractor structure to a pretrained recurrent neural network without using tree structrue. As shown in Fig. (7B, train control model having the same network structure as our model, except that the cross-entropy loss, rather than the ramping target function, is used. By this, the control model learns the sequence discrimination task but no longer acquires the tree-like attractors. We then apply the control model to the same transfer learning task (following the same protocol of freezing the connections in the recurrent and read-out layers). The results demonstrate that our model exhibits significantly accelcrated learning process compared to the control model. This experiment demonstrates that the tree-structured attractor dynamics (the schema) is indeed indispensable for transfer learning. Together with our other analyses in Fig.3A and Fig. A),), which verify that the attractor template indeed reused after transfer learning, we come to the conclusion that the tree-structured attractor dynamics plays a critical role in transfer learning.
As shown in Fig. S8, we show that our model can learn to process sequences of depth 3 by combining two primitive templates of depth 2. In this experiment, for simplicity, consider independent recurrenct networks and there are no recurrent connections between them. Firstly, two RNNs learn and form same tree attractor structures of depth 2 through training on four-class synthetical task. Then, following the transfer learning procedure as described in Se we freeze the recurrent connections in two RNNs and only optimize the feedforwad and readout connections, shown in Fig. S New eight-class phoneme task are constructed, written as "pel-tcl-pel", "pel-tcl-ppu "pel-pau-pel", "pcl-par pacl-pau-tell, "tcl-pel-tel", scl-pcl-pauu" "tel-pau-tel" and "tel-pau-pel". Our findings are as follows: 1) the pretrained attractor structures significantly accellerate the learning speed in a new sequence task, as depcited Fig. SSC: 2) the network indeed combines and reuses two primitive tree attractor dynamics to facilitate the learning speed, as shown in Fig, SSD. These results demonstrate that our model has the capability to adapt to processing temporal sequences of varying lengths.
17

---

A
B
SubRNN1
Test Acsuracy (%)
100
Combine and Reuse
80
40
with tramsfer
20
10
20 Training Epoch
30
40
SubRNN2
D
root
"pcl"
"pcl"
"pau"
"pau" "pcl"
SubRNN1
5
S
PC2
x
2 x
x
S
$
root
SubRNN2
x
X
x
X
PC
x
0
-2
2
2
0
PC
PC
PC 1
Figure S8: An example demonstrating primitive attractor template can be combined and reused to represent longer sequence. A. Network structure: two parts of a RNN (subRNNI and subRNN2) encoding two depth- 2 tree-structured attractors. B. Schematic diagram: illustrating the combination and reuse of two tree- structured primitive templates. C. Sigmificant acceleration in learning compared to the case without transfer learning. Attractor transition dynamics in the two subRNNs when phoneme sequence "pel"-"pau" "pel" is presented. The transition dynamics of subRNNI (the upper pannel) and (uhR lower panel) are shown. In root-"pel when the network receives the 1st sequence item "pel", subRNN1's state remain the root node (red circles), while that of subRNN2 transits from the root node to S² (shown in red arrow). In 1 "pel"-"pau" when given the 2nd item "pau," subRNN1's 's state moves from the root node S².4 while that of subRNN2 stay state $8.4 In "pau"-"pcl when given the 3rd item "pel," subRNNI's state further transits from S to Sg. Thus, in transfer learning, the network combines two tree- structured attractor 12 templates to represent a longer sequence.
D Key-word spotting tasks
D.1 Dataset Construction for Key-Word Spotting Task (Sec.3.2 in the maintext)
To construct the training and testing datasets the key-word spotting task in Sec.3.2 in the main text, we randomly select twelve key words from the TIMIT dataset based on the word labeling. The TIMIT datasct comprises 6300 English sentences read by 630 speakers. The twelve key words are illustrated in S1, which can be chunked into two different primitive items according to syllables. For example, 'wash" is chunked into 'w-ao' and 'sh".
In the four-class key-word spotting task, we use the four words, "water, wash, had, year". In the eight-class task, add another four words, "carry, oily, dark, suit". the twelve-class task, we use all the selected words. In each task, the total number of temporal sequences for each word is nearly 310, which are randomly divided into 10 training and 300 testing examples. The raw wave of each temporal sequence is transformed into MFCC features using time window size of 25ms and stride of 10ms. The obtrained MFCC vectors of each word is further normalized by subtracting their means and divided by their standard deviations along the time dimension.
18

---

In the training data, following our training protocol described in the main text, we chunk the MFCC vectors of sequence into two primitive items according to the syllables of the key word, and then augment the data by inserting varied intervals between primitive items and adding Gaussian white noises. We then train the augmented temporal sequences of key words using ramping target functions, more details see Sec. A in SI. In the testing data, we present the MFCC vectors of each sequence without augmenting, rather we stretch or compress the MFCC sequences. The warping effect is specified by a variable F, with F denoting no warping, F > stretching and / < 1 compressing.
Table S2: 12 words and their chunks, each chunk consists of phonemes or syllables.
Words
Chunks
water wash
['w-ao", ddx-xrr"
('w-ao'''sh) ["hv",'h-ddf"]]
had
year
['y-ih','axr')
carry oily
['kel-k', 'ch-r-iy'] ['oy",I-iy"]
dark
['del-d",aaa--kclla
suit
['s', ['s','ux-eel]]
greasy rag
['gel', (ar'g'ae-gel-'']
ask
['ae','s-kel] [PP]ay-kel']
like
During training, both our model and the control model trained on data with F 1. During testing, we apply warping factor F to compress or stretch the original MFCC feature vectors using linear interpolation while retaining their spatial structure (particularly, we use numpy. interp function to realize temporal warping, which works by finding the two nearest data points surrounding each value in the target data, constructing a straight line between these two points, and calculating the stretched or compressed target data along this line). After temporal warping, we scale the total strength of MFCC feature vectors by dividing scaling factor VF, mimicking the divisive normalization process in the neural system 7.. Finally, the scaled versions of testing words are used to evaluate the model's classification robustness. Notably, scaling the strength of a MFCC feature vector slightly improves models' performance during testing, but any case, our model outperforms other models significantly.
D.2 Ablation study (Sec.4 in the main text)
In our training protocol, the introduction of varied temporal intervals between phonemes syllables has the potential to generate spectral structures that closely resemble those present in the test examples. Consequently, the apparent efficacy of our model may be attributed to the exposure to training data that closely mirrors the test data, rather than leveraging tree attractor structures. To alleviate this concern and demonstrate the importance of the training protocol, we compare control models in the key-word spotting task. demonstrates that the tree-structured attractor dynamics (the schema) contributes to the robustness of our model to warping sequences.
model with several
In our model, each training sequence is augmented to be 1.5 times of its original length by inserting varied intervals between phoneme syllable chunks, and the model is trained using the ramping target function. In control model 2. training examples adopt the same augmentation protocol as in our model, but the model is trained using cross-entropy loss. In the control model 1, training examples are just warped sequences with the warping value chosen randomly in the range of (1, 1.5). This makes the training and test data even more similar in the control model 1 than that in model by data augmentation (in both models, the lengths of sequences are kept to be in the same range). We used the cross-entropy loss to train the control model 2. Thus, both control model 1 and 2 would not acquire proper rre-structured attractors.
We find that when the temporal warping value falls in the range of [1, 1.5], i.e., the range of training sequences, both models exhibit similar good test accuracies. However, when the warping value is out- side of the training range, model outperforms the control model significantly, as shown in Fig. S9
19

---

Accuracy (%)
85
70
Our Model
55
Control model
Control model
1
2
3
4
Temporal Warping (F)
Figure 89: Performance comparison of our model with two control models. In our model, the training data augmented by introducing varied intervals between "chunks" of a given word. Each sequence is augmented to be 1.5 times of its original length. Following the same augmentation protocol as in our model, Control model (green line) is trained using the cross-entropy loss. In Control model 1bblack line), the cross-entropy loss is used, and the training data are warping sequences with the warping value chosen randomly the range of (1, 1.5). The test accuracy of each model is calculated by averaging over 8 trials at each warping value F.
This supports our statement that the strustruet attractor dynamics (the schema) contributes to the robustness of our model to warping sequences. The underlying mechanism can be attributed to ipputs;
two factors: 1) attractors ensure the stable responses of our model to stretching/comprestiie 2) the tree-structured attractor dynamics further enhances averaging out noises in inputs over time, behaving like an evidence- cediemeceaccummlatiio process (see Fig.4C)
We also apply our training protocol to other recurrent networks used in machine learning, including Long Short-Term Memory (LSTM) [9] and Gated Recurrent Unit (OR Using the datasets in the key-words spotting task, both GRU and LSTM are trained by using either our learning protocol the conventional learning method (which involves data augmentation and the cross-entropy loss function). The results are compared in Table. in the main text.
E Experiments with different training Settings and model architectures
We find that the tree-structured attractors also emerge when training using fixed interval values for the clean synthetical data, shown in Fig. [S10 However, for the noisy spoken words, we do need an amount of variations to achieve good performances. We also examine our method on GRU network test university across model architectures. As shown in Fig. STOB, GRU network can also learn tree-structured attractors successfully.
F Model Parameters used in this study
All models are trained using PyTorch 2.0 S
(1) Parameters in 2 in the main text:
The parameters the RNN are: = 3, N 50, K
2. The parameters for data augmentation At 3, AT 30, the noise strength inserted between neighboring items is a = 0.01. The recurrent connection weights Wrec are initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ /N, with g representing the 'gain' of the network, 0.2. For the design of target functions, Sec. A The batch size is B = 16, and the total number of training epochs is 100.
(2) Parameters in Fig.3 in the main text:
20

---

A
B
2
S 1,,2
3
2
X
2
X
S
3,
1
S 1
S 1
1,
2
X
root
2C
X 52
root
52
PC 3
52
PC1
53, 4
52
PC 2
PC
Figure S10: (A) In a 4-class synthetic sequence task, our model successfully learns tree-like attractor structure from sequence inputs with a fixed interval of 10 time steps. (B) A GRU model trained using method also learns tree-like attractor structure the 4-class synthetic task.
The parameters of the RNN are: M 16, N = 50, K 4 in Fig. and K 8 in Fig.4C-D, T = 2. The parameters for data augmentation are: ATmaz = 30, the noise strength inserted between neighboring items is 0.01. For details of the construction of phoneme temporal sequences, in SI. Wree is initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ / g = 0.2. For the design of target functions, see Sec. A in SI. The batch size is B 16, and the total number of training epochs 100.
Sec.
C
(3) Parameters in Fig.4 in the main text:
The parameters of the RNN are: M = 16. = 100, = 4. = 2. The parameters for data augmentation are: max 60, the noise strength inserted between neighboring items is o 0.01. The recurrent connection weights Wree are initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ o/,,, with g representing the 'gain' of the network, g 0.2. For the design of target functions, see Sec. The batch size is 16, and the total number of training epochs is 200.
(4) Parameters in Table 1 in the main text:
The parameters of LSTM, GRU and RNN are: M 16, 100, neurons in the RNN is 2. The parameters for data augmentation are: ATmaz 60, the noise strength inserted between neighboring items is a = 0.01. The recurrent connection weights Wree are initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to g/ /N,, with g representing the 'gain' of the network, 9 0.2. The connections of LSTM and GRU adopt Glorot are initialized by an uniform distribution. For the design of target functions, see Sec. A
The time constant of
The batch size is B 16, and the total number of training epochs is 200.
References
[1] Wang, J. (2008). Decision making in recurrent neuronal circuits. Neuron, 60(2), 215-234.
[2] Okazawa, G., & Kiani, R. (2023). Neural Mechanisms that Make Perceptual Decisions Flexible. Annual Review of Physiology, 85, 191-215.
[3] Shadlen, M. N., & Newsome, W. T. (1996). Motion perception: seeing and deciding. Proceedings of the national academy of sciences, 93(2), 628-633.
[4] Sussillo, D., & Barak, O. (2013). Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3), 626-649.
21

---

[5] Maheswaranathan. N., Williams, A., Golub, M., Ganguli, S., & Sussillo, D. (2019). Reverse engi- neering recurrent networks for sentiment classification reveals line attractor dynamics. Advances in neural information processing systems, 32.
[6] Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S. & Dahlgren, N. L. (1993). DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM NIST
[7] Carandini, M., & Heeger, D. J. (2012). Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1), 51-62.
[8] Paszke, A., Gross, S., Massa, Lerer, A., Bradbury, Chanan, G., & Chintala, S. (2019). Pytorch: An imperative style, figh-perffrmance deep learning library, Advances in neural information processing systems, 32.
[9] Hochreiter, S., Schmidhuber, (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
[10] Chung, J., Gulcchre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
22


Reviews:
[{"review_id": "fjhBew9LSb", "reviewer": "Reviewer_7Qop", "summary": "The canonical biological neural circuit model, described by equation (1) in this work, primarily relies on attractor dynamics to perform cognitive tasks involving temporal sequences. Facilitating the emergence of appropriate attractors during training is a difficult task that challenges neuroscientists even today. The authors alleviate this problem by first training RNNs on simple abstract tasks to allow tree-structured attractor templates to appear within the network, and then further training the networks on more complex tasks, such as a key-word spotting task which will effectively reuse the existing templates. This idea is similar to the emerging field of 'schemas' in network neuroscience.  \n\nOn a technical standpoint, I am convinced that this work represents novel and significant advances in the field of biological RNNs, specifically in the subfield of schema formation. However, I have certain doubts on the neuroscientific elements of this work, especially since the paper is thoroughly cited with neuroscience literature. None of these concerns are a major detriment to my evaluation, and hence I recommend an accept for this work. However, I feel that the authors can considerably improve their discussion and neuroscientific arguments regarding the reusage of tree-structured templates. As such, I hope the authors can adequately address my concerns in the weaknesses section to further solidify this work.", "questions": "See weaknesses.", "limitations": "I was not able to find explicit statements regarding limitations, although the typical shortcomings of a biological RNNs have been addressed as future research directions in lines 294-309. Points addressed in the weaknesses may possibly be added as limitations.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The model and training methods have been extensively explained, which makes their objective clear. The underlying intuition that is being conveyed is straightforward and easy to understand, which is the direct consequence of a well-written introduction from pages 1 to 4. Subsequent sections retain the same quality of writing and scientific rigor.", "weaknesses": "As mentioned before, the key issues that I have about this work stems from neuroscientific plausibility. I do not have specific questions or requests regarding each point, and thus I would recommend the authors simply address this points by arguing for or against these statements.\n\n1. It seems like in this work, the network needs to forget about the previous task in order to use an existing tree-structured template. This seems unrealistic for such a model, and seems to suggest a slot-based usage for such attractors.\n\n2. There exists a lesser-known question in biological RNNs: suppose a network is tasked to remember 3 two-digit numbers (including 00) in sequence. The number of attractors required for this task would be $100^3 = 1,000,000$, which is exaggeratingly large for such a simple task that most humans are able to do. This deters the approach of using tree-structured attractors. \n\n3. Discrete tree-structured attractors are also unable to account for continuous variables, such as color, intensity, speed, temperature etc. Yet, there is no simple generalization for this work to account for continuous variables.\n\n4. Reusing a template also implies that the depth of the tree is fixed. While a tree can indeed grow deeper after subsequent training on more complex tasks, or become shorter due to lack of usage, it does not agree with the logical viewpoint that cognition is highly adaptive to sequence length at much shorter timescales compared to actually learning and developing tree-structures of modified depths.\n\n5. Tree-structures may also lead to unintended behaviors if the wrong inputs are provided, especially if the inputs are not within the scope of the task. This leads to involuntary usage of these tree structures, thus presenting an inherent inflexibility in tree structures.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments, which are very helpful for us to improve the paper. Below are our replies the comments point-by-point.\n\nOn weaknesses:\n\n1. Thank for raising this important issue. Shortly speaking, reusing an existing tree-structured template for a new task in our model does not need to forget about the previous task. In our model, we consider a recurrent network reserved to store tree-structure templates, a kind of slot-based memory. This is motivated by experimental findings which indicate that the brain recruits independent resources to store ordinal structures of temporal sequences disentangled from contents [1]. The advantages of this disentangled ordinal structure representation are that: 1) it saves the overall resource for representing a large number of temporal sequences sharing ordinal structures; 2) it allows the brain to reuse the shared ordinal structures, the so-called schema, to process different sequences flexibly. Once upon a temporal sequence arrives, the neural system rapidly binds the ordinal template with the contents. To process a new task, the neural system needs not to forget about the previous one; rather the neural system learns a new set of feedforward connections from the sensory cortex through dynamical binding. It has been explored previously that dynamic binding can be implemented in the brain through different means [2], including the fast Hebbian rule [3] and neural oscillation [4]. Notably, in the language domain, this schema idea is consistent with the minimalist program in linguistics proposed by Chomsky, which suggests that the brain stores many fundamental syntactic structures and is able to generate arbitrary complex syntactic trees via combining and reusing these primitive structures, referred to as merge, to realize fast language acquisition and appreciation [5].  \n\n2. The reviewer raised an interesting issue about the memory storage problem. Actually, disentangling ordinal structure from contents as a schema is a way for the brain to avoid the memory storage explosion problem. For a large number of sequences sharing the same ordinal structure, the brain only needs to store the ordinal template; while when processing a sequence with specific contents, the brain can quickly bind the template with the contents via dynamical binding. \n\n3. We agree that our model can not account for continuous variables, but this is not a problem for the brain. A large volume of experimental studies has shown that the brain employs a different type of network model, called continuous attractor neural networks, to process continuous variables, such as head-direction [6] and spatial location [7]. This is not contradictory to our proposal of using tree-structured attractors to represent the ordinal structure of temporal sequences formed by discrete items.\n\n4. Thank the reviewer for raising this important issue we have not addressed adequately in the original manuscript. Actually, our model has the capability to adapt to processing temporal sequences of varying lengths. First, for a sequence shorter than the template, our study has already demonstrated that our model can adapt to this, see Fig.3C-D in the manuscript, in which the network learned new sequences only covering part of the stored tree structure. Second, for a sequence longer than the template, our model can learn to combine shorter primitive templates to form a longer one, a strategy similar to the one proposed by Chomsky for language processing [5]. To demonstrate this idea, we conduct an additional experiment, in which the neural system learns to process sequences of depth 3 by combining two primitive templates of depth 2, see the new Fig.R1 in the reply pdf.\n\n5. Thank the reviewer for pointing out this interesting issue which actually highlights an advantage of our model. In the conventional heteroclinic channel model, each node is an unstable saddle point, which can lead to unintended usage of the tree structure if the initial input is wrong. However, in our model, each node is a stable attractor, and to move to the next node, it needs the next input item to be correct in order to induce the transition (see lines 186-189 in the manuscript). As a result, our model displays the event-driven and evidence-accumulation behaviors (see Fig.4C-D in the manuscript). Overall, our model is rather robust to noises as it employs a sequence of attractors to represent the information.  \n\nOn limitations:\n\nWe appreciate the reviewer's insightful comments and will add discussions about the limitations as suggested by the reviewer in the revised manuscript.\n\nReferences\n\n[1]\tDehaene et al, Neuron, 2015\n\n[2]\tEngel Trends in cognitive sciences,2001\n\n[3]\tBittner Science,2017\n\n[4]\tKlimesch et al,Neuroscience & Biobehavioral Reviews, 2010\n\n[5]\tN. Chomsky, MIT Press, 1995\n\n[6]\tKim et al, Science, 2017\n\n[7]\tGiocomo et al, Neuron, 2011"}, {"review_id": "KrCK5hqOs5", "reviewer": "Reviewer_tPSE", "summary": "This paper describes a method for training RNNs that is used to extract ordinal sequences.  There are two variations on the training that make this possible.  First, the network is trained on sequences with a wide range of temporal delays, so that only ordinal position is relevant.  Second, the training signal is the location of each sequence on a tree structure given a priori to describe the set of sequences.  \n\nThe model is tractable and interpretable and two classes of findings are described.  First, networks that have learned a tree structure for a particular problem can generalize rapidly to new problems with the same structure by freezing the recurrent weights and relearning the output weights.  Second, the model is tested on time-warped versions of a set of spoken words and it generalizes better than a control model.", "questions": "Under what circumstances would we expect the network has access to a training signal with information about the ordinal position in a tree structure?  What is the use case for this network?  How could one discover that structure without training it in?\n\nA simple way to build in a model of ordinal position is to have an RNN where the rhs is modulated by a gating factor:\n$\ndx/dt = \\alpha(t) [ \\ldots ] \n$\nwhere $\\alpha(t)$ can be learned.  If $\\alpha(t) = 0$ between relevant triggering stimuli, one can say it's learned an ordinal code.  Is that possible in a GRU? \n\nHow does this model compare in its time warping performance to classic algorithms (e.g., Sekoe & Chiba, 1978)?  On a related note, a recent approach (Jacques, et al., 2022, ICML) shows effectively perfect generalization over a wide range of warping factors without data augmentation.  How does this model relate to that work?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The exposition is extremely clear.\n\n", "weaknesses": "The requirement that the training set provides information about the true ordinal structure seems very strict.  \n\nThe requirement for training on a wide range of temporal intervals is a serious limitation as a model of the brain.  \n\nI find the colored lines in Fig 1A,C very difficult to distinguish.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments, which are very helpful for us to improve the paper. Below are our detailed replies. \n\nWeaknesses\n\n**On the prior of the ordinal structure**\n\nThank for raising this concern, but we would like to point out that this is not a problem for the brain, although it may be a concern for machine learning tasks. \n\nFirst, experimental studies have shown that in the brain, continuous sequences are chunked into discrete items to support high-level cognition [1]. For examples, speech sequences can be hierarchically chunked into words and syllables [2]; neurons in the hippocampus have been shown to detect event boundaries when watching movie videos [3]. This chunking process naturally produces ordinal structures of temporal sequences. Computational models are also proposed for neural chunking, such as self-supervised learning [4] and oscillation [5]. \n\nSecond, the development path also indicates that the brain implements sequence chunking. For example, in language acquisition, young children learn primitive phonemes around 2 months [6], learn sequences of either ABB or ABA form around 7 months [7], and learn to recognize spoken words around 12 months. Here, phoneme learning serves as a building block for word learning, defining the ordinal structure of language sequences.\n\nSince our focus is on exploring how the brain learns the ordinal structure disentangled from contents, the so-called schema which is receiving increasing attention in theoretical neuroscience [8], we assume that the input sequences to our model have already been chunked in advance. While when applying our model to machine learning, it does need a method to first segment sequences based on the statistics of items.  \n\n**On the wide range of temporal intervals**\n\nThank for raising this concern, but we would like to argue that this is not a serious limitation to the brain.\n\nFirst, in the brain, motor and speech sequences are generated with large variability in speed, and they exhibit large variability in separations between motor motifs and speech chunks. This large variability enables the brain to learn the tree-like attractor structure. \n\nSecond, in our network training, we actually do not need very large intervals. For the clean synthetical data, we can actually train the tree-structured attractors using fixed intervals (see Fig.R2-A in the reply pdf). For noisy spoken words, we do need an amount of variations in intervals to achieve good performance, but the range is only about 2 times of the item length. Overall, it does not need the range of temporal intervals to be very large.\n\nQuestions\n\n**On the available of ordinal structure, the use of the model, and un-supervised learning**\n\n1) As replied above, the brain can access the ordinal structure through chunking. Experimental data has shown that continuous sequences such as motor and speech sequences are hierarchically chunked to form discrete sequence representations. \n\n2) Our network can be applied to model the learning of abstract ordinal structures of temporal sequences (schema), such as motor sequences, speech, and language. For example, a potential application is to model language appreciation (certainly many more works are needed), where primitive ordinal structures are stored to facilitate language understanding. Our model may also be applied to solve transfer learning in machine learning.\n\n3) To discover sequence structures without supervised signals, we may mimic the chunking mechanism in the brain. Incorporating a generative objective in unsupervised learning may also aid in extracting sequence structures. The chunking mechanism may use a self-supervised loss [4] or quantized representation trick [9]. Generative objective loss may employ the contrastive predictive coding method [10] or some other self-supervised methods. \n\n**On GRU**\n\nSorry, we do not get the meaning of the reviewer, “A simple way to build in a model of ordinal position is to have an RNN where the rhs is modulated by a gating factor…”.\nIn term of GRU, it can indeed learn the tree-like attractor structure if our learning protocol is used. The performance of GRU after learning tree-like attractors is shown in Table 1. We can visualize its tree-like attractor structure of neural activities in GRU, see Fig.R2-B in the reply PDF.\n\n**On the warping performance**\n\nThanks for pointing out these two references we have missed. We did not evaluate our model intensively on the warping performance. This is because that the goal of the current paper on exploring how the brain learns the disentangled ordinal structure (schema). Since the key of schema is on transferring learning, we have focused on investigating this issue. In future work, we will systematically evaluate the warping performance of our model and compare it with other methods.  \n\nConsidering the differences to the two works: 1) Sekoe & Chiba proposed a machine learning kind of method for representing the ordinal structure, while our method focuses on the neural representation of the ordinal structure; 2) the model of Jacques et al. was inspired by time cells in the brain, while our model was inspired by the disentangled representation of ordinal structure, manifested as tree-structured attractors. Jacques’ model may perform well for recognizing a given set of warping sequences, it likely has difficulty for transfer learning, since it does not extract the schema as done in our model. \n\nWe hope that we have addressed all concerns of the reviewer and could convince the reviewer to raise the score.\n\nReferences:\n\n[1]\tDehaene et al, Trends in Cognitive Sciences, 2022\n\n[2]\tDehaene et al, Neuron, 2015\n\n[3]\tHahamy et al, Nature neuroscience,2023\n\n[4]\tAsabuki et al, Nature communications, 2020\n\n[5]\tGiraud et al, Nature neuroscience,2012\n\n[6]\tKuhl et al, Nature reviews neuroscience,2004\n\n[7]\tMarcus et al, Science,1999\n\n[8]\tGoudar et al, Nature Neuroscience,2023\n\n[9]\tVan Den Oord et al, Neurips, 2017\n\n[10]\tOord et al, arXiv, 2018"}, {"review_id": "1uPfgoLuBm", "reviewer": "Reviewer_dJ2Y", "summary": "This paper investigates how recurrent neural circuits learn to represent the abstract order structure of temporal sequences and how the disentanglement facilitates sequence processing. The main objectives were better understand the brain's mechanisms for representing temporal sequence ordinal information and contributing to the development of brain-inspired sequence processing algorithms. \n\nThe authors found that given suitable training, a recurrent neural circuit can learn tree-structured attractor dynamics to encode the tree-structured orders of temporal sequences. They show that reusing a temporal order template aids the learning of new sequences sharing the same or partial ordinal structure and that the tree-structured attractor dynamics improve the robustness of temporal sequence discrimination.\n\n", "questions": "When you used data augmentation and varied the duration between neighboring items, how many steps of BPTT did you have to do and have you encountered issues with vanishing/exploding gradients? \n\nThe authors reflected on the biological implausibility of backpropagation and proposed the possibility of using the fast Hebbian rule. It is not clear whether this could work in the context of backpropagation through time since it would require that the biological system stores a copy of the temporal input. Do the authors have a possible solution for this? \n\nDo you think the observed properties could emerge if the training was done in a self-supervised way? ", "limitations": "The authors addressed some of the limitations, especially regarding the backpropagation (but see my earlier comments on how I suggest expanding those). ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors demonstrated that in a supervised learning task, recurrent neural circuit can learn tree-structured attractor dynamics to encode the corresponding tree-structured orders of temporal sequences. They used a transfer learning task to show that once the network has learned the temporal structure, it can apply that knowledge to different temporal inputs - this was demonstrated by freezing the recurrent weights and training only weights in a feedforward layer that followed the recurrent layer. They also showed that data augmentation can lead to invariance to temporal rescaling. These results are consistent with several neuroscience studies. ", "weaknesses": "The evaluation is relatively limited and uses only short sequences and supervised learning. If the brain indeed uses a similar mechanism, then it should successfully scale to much longer sequences.\n\nIf the model is capturing cognitive properties of human sequence memory, then it should also be consistent with results from behavioral studies, which are characterized by effects such as primacy, recency and contiguity.\n\nThe authors might consider reflecting on or comparing their approach with other work related to modeling sequences in the brain, for example:\n\nCui et al. 2016 Continuous online sequence learning with an unsupervised neural network model\n\nGraves et al. 2014 Neural Turing machines \n\nVoelker et al. 2019 Legendre memory units: Continuous-time representation in recurrent neural networks \n\nEliasmith et al. 2013 A large-scale model of the functioning brain \n\nHoward et al. 2014 A unified mathematical framework for coding time, space, and sequences in the hippocampal region \n\nWhittington et al. 2020 The Tolman-Eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation\n\nAlso, for recent work on biologically inspired neural networks robust to temporal rescaling see \n\nJacques et al. 2022 A deep convolutional neural network that is invariant to time rescaling\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments, which are very helpful for us to improve the paper. Below are our detailed replies. \n\nWeaknesses\n\n**On the scalability of the model** \n\nThanks for raising this important issue. As the first step of presenting the framework, we only evaluate the model with short sequences. The study of long sequences will be added in future work. Here, we would like to point out that our model has the capacity of processing long sequences, and it can achieve this in two biologically plausible ways.  \n\nFirst, the brain can combine short primitive templates to form long ones. To demonstrate this idea, we conduct a preliminary experiment, in which the network learns to represent sequences of length 3 by dynamically combing two shorter templates of length 2 (see Fig.R1 in the uploaded reply pdf). \n\nSecond, the brain can employ a hierarchical network to combine ordinal templates in each layer hierarchically. Notably, this way of combining primitive templates to encode long sequences was proposed by Chomsky for language processing, the so-called the minimalist program [1]. The program suggests that our human brain is capable of processing arbitrarily complex and lengthy language sentences through recursive dynamic binding of primitive linguistic units.\n\n**On the cognitive properties of human sequence memory**\n\nThanks for the suggestion. The effects of primacy, recency, and contiguity are often reported in sequential working memory tasks, and a potential computational mechanism is the short-term plasticity (STP) of synapses [2]. Here, our focus is on studying the learning of tree-structured attractors (the schema), but we presume that if we also consider a sequential working memory task and include STP in feedforward connections, we should observe similar effects. \n\nAlternatively, we may compare our model to those schema-related experimental studies. For example, a recent study [3] found that rats could store the odor sequence structure as a schema by using a low-dimensional neural code in the orbitofrontal cortex, and this schema facilitated learning of new similar tasks. Our model also shows that the ordinal structure is encoded as tree-structured attractors in a low-dimensional space, and this attractor dynamics facilitates transfer learning. In future work, we will compare our model with the experimental data in details.\n\n\n**On the differences to other papers**\n\nThanks for providing these references. We checked all of them and summarized their differences to our work as below: \n\nThe works by Cui et al. (2016), Graves et al. (2019), and Eliasmith et al. (2013) addressed sequence modeling using different ways, such as HTM with an unsupervised Hebbian rule, external memory modules, and cognitive architectures. These models differ from our work on that they do not explicitly disentangle ordinal structures from contents. Whittington et al. (2020) proposed TEM to disentangle structure from contents and integrate them via conjunctive coding. The self-supervised method used by TEM may learn a tree-like attractor structure in an unsupervised manner. However, TEM does not tell how to store the learned sequence structures and reuse them for new tasks. The LMU model by Voelker et al. (2019) efficiently captures long temporal dependencies. Our method can be applied to train the LMU model for learning complex tree-like structures, and potentially further leverage the LMU's efficiency in capturing long dependencies. Jacques et al. (2022) proposed a new deep convolution network that utilizes logarithmically compressed temporal representations, but does not consider the extraction of disentangled ordinal structure.\n\nOverall, our model is fundamentally different from these works on that we explore how the brain learns the disentangled ordinal structure (schema) by employing tree-structured attractor dynamics, and how this schema facilitates transfer learning. Nevertheless, our model is not contradictory to these works, rather they can be integrated together to perform complex sequence tasks.   \n\nQuestions:\n\n1. We typically performed 90 to 150 steps of backpropagation through time (BPTT). To mitigate the potential vanishing or exploding gradients, we employed gradient clipping and carefully initialized the weights. Thus, we did not encounter vanishing/exploding gradients in practice.\n\n2. As the fast Hebbian rule is differentiable, it can be applied in the context of backpropagation through time (BPTT). Recent works in both neuroscience and machine learning society have explored this issue [4,5,6,7]. In our model, we can replace the static feedforward connections from inputs to the recurrent network with context-controlled fast Hebbian weights and train the model using meta learning methods [8]. In such a way, context information can modulate feedforward connections, enabling fast binding or rebinding between contents and ordinal templates. Thus, the integration of fast Hebbian rule and BPTT may facilitate the learning and utilization of the tree-structured attractor dynamics.\n\n3. Yes, it is possible that the tree-like attractor structure emerges in the network through self-supervised learning. A possible solution is that: we first apply a self-supervised method [9] or a quantized representation method [10] to chunk raw input sequences into discrete items; we then apply a self-supervised learning method, such as the contrastive predictive loss [11], to train the recurrent network to get the tree-structured attractor dynamics.  \n\nReferences:\n\n[1]\tN. Chomsky, MIT Press, 1995\n\n[2]\tMi et al, Neuron, 2017\n\n[3]\tZhou et al, Nature, 2021\n\n[4]\tBa et al, Neurips, 2016\n\n[5]\tThangarasa et al, ICML, 2019\n\n[6]\tTyulmankov et al, Neuron, 2022\n\n[7]\tDekker et al, PNAS,2022\n\n[8]\tWang et al, Current Opinion in Behavioral Sciences, 2021\n\n[9]\tAsabuki et al, Nature communications, 2020\n\n[10]\tVan Den Oord et al, Neurips, 2017\n\n[11]\tOord et al, arXiv, 2018"}, {"review_id": "cUrWw927vY", "reviewer": "Reviewer_CeTG", "summary": "In this manuscript the authors use a custom training regime to force simple recurrent neural networks (RNNs) to learn the ordinal structure of sequential inputs. Specifically, they train the network to learn the order of sequences by presenting the elements of a sequence with variable durations and variable intervals between elements. They demonstrate the utility of these networks in both a toy transfer learning task and a more realistic \"key-word spotting\" task.", "questions": "1. Can the authors provide more detail about the control models in Figures 3 and 4? My concerns could be simple misunderstanding.", "limitations": "Limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Strengths:\n\n1. The manuscript presents the key ideas in a straightforward and lucid manner\n2. The core idea – training RNNs to recognize the ordinal structure of its inputs through varying the duration of its constituent elements – is interesting with potential downstream applications in machine learning and computational neuroscience\n", "weaknesses": "Weaknesses:\n\nMajor:\n\n1. The results in Figure 3 are difficult to interpret without a more systematic bank of control models. For instance, in the transfer learning task, as far as I can tell the authors compare transfer learning of their particular model to a model with no transfer learning. This does not tell us if the tree structure learned by their model is doing the heavy lifting. Specifically, have the authors compared their \"tree-structure\" model to an RNN that learns the task without using tree structure? Without this key control we cannot assess whether tree structure is what confers performance in this task, or if it is simply a matter of their model having been trained to solve *any* task prior to transfer learning.\n1. I am a bit confused about what Figure 4 is showing. Here, the authors train their network using varied intervals between \"chunks\" of a given word. One issue that I can see with this training regime is that, in addition to enforcing tree structure in network dynamics, it is also presenting a \"warped\" version of the inputs. In other words, does inserting a delay between phonemes create spectral structure more similar to the test inputs than the phonemes *without* gaps? If true, this would imply that the network is simply seeing training data more similar to the test data, rather then leveraging tree structure. Further controls and analysis would alleviate this concern.\n\nMinor:\n\n1. Typos throughout, the manuscript needs a pass on the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments, which are very helpful for us to improve the paper. Below are our detailed replies. \n\nWeaknesses\n\n**On the contribution of tree-structured attractors**\n\nThank for raising this concern. To address this concern, we conducted an additional experiment (see Fig.R2-C in the uploaded reply PDF), in which we trained a control model having the same network architecture, except that the cross-entropy loss, rather than the ramping target function, was used. By this, the control model learned the sequence discrimination task, but no longer acquired the tree-structured attractors. We then applied the control model to the same transfer learning task (following the same protocol of freezing the connections in the recurrent and read-out layers). As shown in Fig.R2-C, our model exhibits significantly accelerated learning speed compared to the control model. This experiment demonstrates that the tree-structured attractor dynamics (the schema) is indispensable for fast transfer learning. Together with our other analyses in Fig.3A and Fig.S3 (which verify that the attractor template is indeed reused during transfer learning), we conclude that the tree-structured attractor dynamics is critical for fast transfer learning.    \n\n**On the control of training data in warping performance**\n\nThanks for raising this concern. To clarify this concern, we conducted an additional experiment, in which we built a control model which just uses warped sequences as the training data. This makes the training and test data even more similar than that in our model. We adopted the cross-entropy loss to train the control model, such that the recurrent network did not acquire proper tree-structured attractors. We found that when the temporal warping value falls in the range of [1, 1.5], i.e., the range of the lengths of training sequences, both models exhibit similar good test accuracies. However, when the warping value is outside of the training range, our model outperforms the control model significantly (see Fig.R2-D in the uploaded reply pdf). This control experiment excludes the possibility that the similarity between training and testing data leads to the good performance of our model, and it supports that the tree-structured attractor dynamics contributes to the robustness of our model to warped sequences. The underlying mechanism can be attributed to two factors: 1) attractors ensure the robust responses of our model to stretching/compressing inputs; 2) the tree-structured attractor dynamics further average out noises over the sequence, behaving like an evidence-accumulation process (see Fig.4C).\n\nMinor\n\nWe will thoroughly improve the writing in the revised manuscript.\n\nQuestions\n\n**On the control models in Figures 3 and 4**\n\nIn Fig.3, the control model shares the same network architecture, data augmentation and target function as our model. The only difference is on the training protocol. In our model, the connections in the recurrent layer, which store the tree-structured attractors, are frozen during the transfer learning, and only feedforward connections are updated; whereas, in the control model, both recurrent and feedforward connections are re-trained (more details see lines 209-214 in the supplementary materials). This comparison allows us to demonstrate the effect of the schema on accelerating transfer learning.  \n\nIn Fig.4, the control model shares the same network architecture as our model. In the control model, no data argumentation is used, and the cross-entropy loss is adopted. By this setting, the control model can learn a given set of sequences, but does not form the tree-structured attractor dynamics. This comparison is to demonstrate that the tree-structured attractor dynamics enhances the robust of our model to warped sequences. However, as pointed out by the reviewer, since no data argumentation is used in the control model, it does not exclude the possibility that the improved performance of our model comes from data argumentation, as it tends to make the training and testing data more similar. To clarify this concern, we train another control model as described above (see Fig.R2-D in the reply PDF), which directly uses warped sequences as training examples, and hence it makes the training and test data even more similar than our model. Again, we observe that our model outperforms the control model to unseen warped sequences. This further strengthens our conclusion that the learned schema facilitates the robustness of our model to warped sequences.   \n  \nWe will expand the descriptions about control models (including the new ones) in the revised manuscript to make them more clearly.\n\nWe hope that we have addressed all the concerns of the reviewer and could convince the reviewer to raise the score. \n\n"}]

Author rebuttal:
We appreciate the valuable comments from all reviewers, which are very helpful for us to improve the work. We have addressed all concerns of the reviewers point-by-point. 

Attached please find the supplementary figures to answer the concerns of reviewers.

Meta-review: The paper investigates how recurrent networks can rapidly learn ordinal information, via pertaining a recurrent RNN such that it learns tree-structured templates that can be rapidly bound to new items.

Strengths:

-Clear and complete explanation of methods and objective

-The problem is important

-The proposed method achieves good performance and behaviour consistent with neuroscientific studies; it also provides a compelling intuition for transfer learning abilities

Weaknesses:

-The paper would benefit from additional discussion of related sequence models in the brain

-The neural plausibility is not clear

-How the brain could achieve training on diverse temporal intervals is unclear

-The evaluation focuses on small sequences and the scalability of the method is unclear
Decision: Accept (poster)