SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Learning and Processing the Ordinal Information of Temporal Sequences in Recurrent Neural Circuits
Xiaolong Zou<sup>1,4,5,*</sup>
zouxiaolong@qiyuanlab.com
Zhikun Chu2,\*
chukunzhi@outlook.com
Qinghai Guo
guoqinghai@huawei.com
Jie Cheng
jiecheng2009@gmail.com
Bo Hong<sup>1,6</sup>
hongbo@tsinghua.edu.cn
Si Wu4,5
Yuanyuan  $\mathbf{M}\mathbf{i}^{3,\top}$
miyuanyuan@tsinghua.edu.cn
1, Qiyuan Lab, Beijing, China.
2, Center for Neurointelligence, School of Medicine, Chongqing University.
3, Department of Psychology, Tsinghua University
4, School of Psychological and Cognitive Sciences,
Peking-Tsinghua Center for Life Sciences, Center of Quantitative Biology,
Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China
5, Beijing Academy of Artificial Intelligence, Beijing, China.
6, Biomedical Engineering, School of Medicine, Tsinghua University
7, Huawei Technologies.
*: Equal contributions.  $\top$  : Corresponding authors.
Temporal sequence processing is fundamental in brain cognitive functions. Experimental data has indicated that the representations of ordinal information and contents of temporal sequences are disentangled in the brain, but the neural mechanism underlying this disentanglement remains largely unclear. Here, we investigate how recurrent neural circuits learn to represent the abstract order structure of temporal sequences, and how the disentangled representation of order structure facilitates the processing of temporal sequences. We show that with an appropriate training protocol, a recurrent neural circuit can learn tree-structured attractor dynamics to encode the corresponding tree-structured orders of temporal sequences. This abstract temporal order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. Using a transfer learning task, we demonstrate that the reuse of a temporal order template facilitates the acquisition of new temporal sequences, if these sequences share the same or partial ordinal structure. Using a key-word spotting task, we demonstrate that the tree-structured attractor dynamics improves the robustness of temporal sequence discrimination, if the ordinal information is the key to differentiate these sequences. We hope that this study gives us insights into the mechanism of representing the ordinal information of temporal sequences in the brain, and helps us to develop brain-inspired temporal sequence processing algorithms.
A temporal sequence involves a set of items or events that unfold in a specific order over time. Temporal sequence processing is fundamental in many cognitive functions [1], such as speech recognition, language comprehension, motor control, and memory formation. Effective temporal sequence processing relies on extracting the temporal structure of a sequence, which includes particularly the ordinal information, i.e., the order of events' occurrence. A large volume of studies has suggested that the representations of ordinal information and contents of a temporal sequence are disentangled in the brain [1-6]. For instances, studies using single-unit recording in primates have found that a significant proportion of neurons in the dorsolateral prefrontal and intraparietal cortexes are sensitive to the temporal order of presented visual items, but not to their physical attributes [5], and similar temporal order representations were also observed in the auditory and linguistic sequence processing [2]. The imaging study has shown that the human brain can encode the phonetic order during speech processing, regardless of the phonetic contents [2]. Recent large-scale neuron recordings have revealed that the temporal order structure can be robustly encoded in a low-dimensional space formed by coordinated neuron population activities [4]. The advantage of disentangled representation of order structure is that it enables the brain to flexibly and rapidly process temporal sequences via conjunctive coding [1], i.e., the brain can easily generate various temporal sequences by combining the temporal order with different contents. This was evident in a finger sequence learning task [6], where the brain stored a temporal order template and reused it to combine actions to generate various finger action sequences. Understanding how ordinal information is acquired and stored in the brain is crucial for us to understand how temporal sequences are processed in cognitive functions.
Remarkably, the representations of temporal orders of related sequences display a tree-like structure [7]. For example, in speech recognition, spoken words are represented by temporal sequences formed by primitive syllables or phonemes. As illustrated in Fig.1A, the words "water" and "wash" share a common syllable "w-ao" at the first item and differ at the second syllable, and so do the words "your" and "year". Thus, the temporal orders of these four sequences form a two-layer tree: starting from the root, the words "water" and "wash" form a branch in the first layer as they share the same first syllable, and the words "your" and "year" form the other branch; in the second layer, the words "water" and "wash", and the words "your" and "year", are further branched due to their difference at the second syllable. Experimental data has indicated that such tree-structured representation of ordinal information is employed in the frontal cortex to support hierarchical motor control [8].
Figure 1: (A) A schematic diagram illustrating the tree-structured temporal orders of four spoken words. Words 'water' and 'wash' form a branch in the first layer because of the shared syllable 'w-ao', and they are branched in the second layer due to the different second syllables, 'dx-axr' and 'sh'. Similarly, words 'your' and 'year' form other branches. (B) A schematic diagram of our neural circuit model. It consists of three parts: an input layer of dimension  $M$  conveying temporal sequences, a recurrent neural network (RNN) of  $N$  neurons encoding the ordinal structure, and an output layer discriminating  $K$  different temporal sequences.
Despite that the representation of ordinal information in the brain has been well recognized in the field, the neural mechanism of implementing this representation remains largely unclear [1, 7]. A number of computational models has been proposed to mimic how temporal sequences are stored
in neural circuits, and they typically consider that neurons are asymmetrically connected along the path of a sequence of network states representing the temporal sequence [10-12, 15]. Among the proposed models, the framework of heteroclinic channel is inspiring [10], which considers that a temporal sequence is stored as a sequence of saddle points connected by unstable paths in the state space of the network, forming the so-called a heteroclinic channel, and under the drive of external inputs, the network state evolves along the channel to activate stored items in the right time order. In most previous works [11, 13-15], the asymmetric neuronal connections accounting for storing temporal sequences are either designed by hand or constructed by applying an asymmetric Hebbian learning rule (note that the application of a Hebbian learning rule implicitly assumes that neural representations of sequential items are known, which does not really model the learning process), and they did not address how a neural circuit learns to represent the temporal order structure from data. A few studies explored how temporal sequences are learned in a recurrent neural network [15], but they only considered the simple chain-structured temporal order and did not study the more general tree-structured orders. The machine learning society also studied the learning of temporal sequences (such as speech) by using artificial neural networks, but they have not considered the representation of disentangled ordinal information [9].
In this work, we explore how biologically plausible recurrent neural circuits learn to represent the tree-structured orders of temporal sequences and how this disentangled representation of order structure facilitates the processing of temporal sequences. We show that with an appropriate training protocol, a recurrent neural circuit can learn a set of tree-structured attractor states to encode the corresponding tree-structured orders of temporal sequences, where each path from the root to a leaf represents a pattern of temporal order, with nodes on the path indicating the order hierarchy (see Fig.1A). This abstract order template can then be bound with different contents, allowing for flexible and robust temporal sequence processing. To demonstrate this advantage, we carry out two experiments. First, in a transfer learning task, we show that the reuse of the order template accelerates the acquisition of new temporal sequences, if these sequences share the same or partial ordinal structure. Second, in a key-word spotting task, we show that using the tree-structured attractor dynamics improves the robustness of temporal sequence discrimination, if the ordinal structure is the key to differentiate these sequences. We hope that this study facilitates our understanding of how temporal sequences are represented and processed in the brain, and helps us to develop brain-inspired algorithms for temporal sequence processing.
2 Learning the ordinal structure of temporal sequences
2.1 The neural circuit model
The model we consider consists of three parts (Fig.1B): an input layer which conveys the information of temporal sequences, a recurrent neural network (RNN) which stores the ordinal structure of temporal sequences, and an output layer which discriminates temporal sequences. Our idea is that by training the model to perform a temporal sequence discrimination task, the RNN learns to represent the ordinal structure of given temporal sequences in its dynamics. The details of the model are introduced below.
Denote  $\mathbf{I}(t) = \{I_m(t)\}$ , for  $m = 1, \ldots, M$ , with  $M$  the dimension, the activity of the input layer conveying the temporal sequence. The input layer is connected to the RNN through a feedforward connection matrix  $\mathbf{W}^{in} = \{W_{nm}^{in}\}$ , for  $m = 1, \ldots, M$  and  $n = 1, \ldots, N$ , with  $N$  the number of neurons in the RNN. Neurons in the RNN are recurrently connected with each other with a matrix  $\mathbf{W}^{rec} = \{W_{nj}^{rec}\}$ , for  $n, j = 1, \ldots, N$ , and they also receive a background input  $\mathbf{I}^b = \{I_n^b\}$ , for  $n = 1, \ldots, N$ . Denote  $x_n(t)$  the total input received by the  $n$ th neuron in the RNN at time  $t$  and  $r_n(t)$  the corresponding neuronal activity. The dynamics of  $x_n(t)$  is written as,
$$
\tau \frac {d x _ {n} (t)}{d t} = - x _ {n} (t) + \sum_ {j = 1} ^ {N} W _ {n j} ^ {r e c} r _ {j} (t) + \sum_ {m = 1} ^ {M} W _ {n m} ^ {i n} I _ {m} (t) + I _ {n} ^ {b}, \tag {1}
$$
where  $\tau$  is the time constant. The nonlinear relationship between  $x_{n}(t)$  and  $r_n(t)$  is given by  $r_n(t) = \tanh [x_n(t)]$ .
The RNN is connected to the output layer through a feedforward matrix  $\mathbf{W}^{out} = \{W_{kn}^{out}\}$ , for  $n = 1, \ldots, N$  and  $k = 1, \ldots, K$ , with  $K$  the number of neurons in the output layer.  $K$  is also the
number of temporal sequences, since each neuron reads out one class of sequences. Denote  $y_{k}(t)$  the activity of the  $k$ th neuron in the output layer, whose dynamics is given by,
$$
y _ {k} (t) = \sum_ {n = 1} ^ {N} W _ {k n} ^ {\text {o u t}} r _ {n} (t). \tag {2}
$$
2.2 The training protocol
The details of the training protocol are presented in Supplementary Information (SI). Here, we highlight two settings which are critical for the RNN to extract the ordinal structure of temporal sequences.
Firstly, we augment training data by varying the duration between neighbouring items in a temporal sequence (see illustration in Fig.2A). Specifically, this is done by adding noisy signals of varied duration between neighbouring primitive items of the temporal sequence (see more details below). The underlying mechanism is intuitively understandable. The augmented data imposes that the model needs to realize invariant recognition with respect to the warping of a temporal sequence, which forces the RNN to extract the abstract ordinal structure of the temporal sequence. This is also biologically reasonable, as in speech recognition [2, 3], the varied speed of spoken words forces the brain to process temporal sequences relying mainly on their ordinal information.
Secondly, we set the target function, i.e., the target activity of a read-out neuron in response to the temporal sequence it encodes, to be continuous in time, and its value increases step by step as items in the temporal sequence unfold one by one (see illustration in Fig.2A). The underlying mechanism is intuitively understandable. Since not all temporal sequences are differentiable by early items, neuronal responses should be low initially, and their values increase along with the exposure of followed items to reflect that sequence recognition is a process of evidence accumulation over time. Interestingly, such ramping activity of read-out neurons agree with the experimental finding on decision-making neurons in monkeys [1-3].
After constructing input data and target functions, we apply the standard algorithm, back-propagation through time, to train the model, which optimizes the parameters  $\left(\mathbf{W}^{in},\mathbf{W}^{rec},\mathbf{W}^{out},\mathbf{I}^{bg}\right)$ . For details, see Sec. A in SI.
2.3 A synthetic task and the model behavior
To demonstrate that our model works, we first apply it to a synthetic task of discriminating four temporal sequences ( $K = 4$ ). These four temporal sequences are constructed by combining two of three primitive items with varied orders (like syllables forming spoken words). As illustrated in Fig.2A, the three primitive items are 3-dimensional ( $M = 3$ ) continuous-time vectors within an interval  $\Delta t$ , denoted as  $\mathbf{a} = [1,1,0]^T$ ,  $\mathbf{b} = [1,0,1]^T$ , and  $\mathbf{c} = [0,1,1]^T$ , respectively. The contents of four temporal sequences are expressed as  $\mathbf{S}^k = \{S_1^k,S_2^k\}$ , with  $k$  the index of the sequence and  $S_1^k,S_2^k$  the first and second items. Since in this study, the exact duration of a temporal sequence is not important, for clearance, we also use  $\mathbf{S}^k$  to denote a sequence having the contents  $\{S_1^k,S_2^k\}$  with unspecified duration. The four classes of temporal sequences are therefore expressed as,  $\mathbf{S}^1 = \{\mathbf{a},\mathbf{b}\}$ ,  $\mathbf{S}^2 = \{\mathbf{a},\mathbf{c}\}$ ,  $\mathbf{S}^3 = \{\mathbf{b},\mathbf{a}\}$ ,  $\mathbf{S}^4 = \{\mathbf{b},\mathbf{c}\}$ , and their temporal orders form a two-layer tree structure as shown in Fig.2B. To augment the training data, we carry out two operations. Firstly, we randomly set the duration between neighboring primitive items to be  $\Delta T$ , which satisfies a uniform distribution in the range of  $[0,\Delta T_{max}]$ . Secondly, we add noises in the duration, which are given by  $\eta_i(t) = \sigma \xi_i(t)$  for  $i = 1,\dots ,M$  and  $t\in (0,\Delta T)$ , where  $\xi_{i}(t)$  denotes Gaussian white noises of zero mean and unit variance, and  $\sigma$  the noise strength.
After training, we observe that the model learns to discriminate four temporal sequences in a coarse-to-fine manner. An example of discriminating the temporal sequence  $\mathbf{S}^1$  is presented in Fig.2C. Initially, both activities of neurons 1 (red) and 2 (green) increase, since  $\mathbf{S}^1$  and  $\mathbf{S}^2$  are not differentiable by their first item, i.e.,  $S_1^1 = S_1^2$ ; while neurons 3 (orange) and 4 (blue) keep silent. When the second item appears, only the activity of neuron 1 keeps increasing, while the activity of neuron 2 drops, indicating that the model recognizes  $\mathbf{S}^1$ . The above results can be generalized to more than four temporal sequences, see Sec. A in SI for  $K = 8, 12$ .
2.4 Visualizing the learned tree-structured attractor dynamics
C
F
Figure 2: Unveiling the performance and the mechanism of the model using synthetic data. (A) Illustrating the constructions of temporal sequences and target functions. Two primitive items, (a, b), are combined to form the temporal sequence  $\mathbf{S}^1$ , with an interval  $\Delta T$  inserted between two items, where  $\Delta T$  is sampled from a uniform distribution in the range of  $[0, \Delta T_{max}]$ . Noises are added in the interval. Along with the unfold of two items in  $\mathbf{S}^1$ , the target function of read-out neuron 1 (red) increases step by step, the target function of read-out neuron 2 (green) first increases and then decreases, and the target functions of read-out neurons 3 (orange) and 4 (blue) always keep silent. (B) The orders of four temporal sequences ( $\mathbf{S}^1$ ,  $\mathbf{S}^2$ ,  $\mathbf{S}^3$ ,  $\mathbf{S}^4$ ) form a two-layer tree. (C) An example of the model performance when recognizing the temporal sequence  $\mathbf{S}^1$ . It operates in a coarse-to-fine manner. When the first item appears, both the activities of read-out neurons 1 and 2 increase, since  $\mathbf{S}^1$  and  $\mathbf{S}^1$  are not differentiable at this stage; when the second item appears, only the activity of read-out neuron 1 keeps increasing, indicating the model accomplishes the discrimination task. (D) (upper panel) PCA analysis shows that the top 3 PCs explain nearly  $90\%$  of the variance of neuronal responses in the RNN. (lower panel) The eigenvalue distribution of the attractor state  $S_1^{1,2}$  in Fig.2E. (E) The visualization of the tree-structured neural response trajectories to four temporal sequences, ( $\mathbf{S}^1$ ,  $\mathbf{S}^2$ ,  $\mathbf{S}^3$ ,  $\mathbf{S}^4$ ), in the reduced space spanned by 3 PCs. Each item in a temporal sequence is stored as an attractor in the RNN (marked by cross). (F) The transition probabilities from node  $S_1^{1,2}$  to other states, which are quantified by the eigenvalues of the corresponding Jacobian matrix. The two eigenvectors having the largest two eigenvalues (reflected by their lengths) point to nodes  $S_2^1$  and  $S_2^2$ . The parameters used are given in Sec. F in SI.
We carry out intensive analysis to visualize what have been learned in the dynamics of RNN.
2.4.1 Tree-structured representation trajectories
To visualize the high-dimensional dynamics of the RNN ( $N = 50$ ), we apply principle component analysis (PCA) to reduce its dimensionality. PCA is done by using the activities of RNN neurons in response to 1000 testing temporal sequences (for details, see Sec. B.1 in SI). It turns out the top 3 PCs explain nearly  $90\%$  of the variance of neuronal responses (Fig.2D, upper panel), indicating that after training, the RNN performs computation in a low-dimensional subspace. By projecting RNN activities onto the top 3 PCs, we obtain the trajectories of the network state in responses to the four temporal sequences (averaged over 25 trials), and observe that they display a tree structure, agreeing with the ordinal structure of the learned four temporal sequences (Fig.2E, trajectories for different sequences are marked by different colours).
2.4.2 Tree-structured attractor states
We continue to analyze the state space of the RNN, and find that it has totally 7 stable points (attractors), which are the nodes of the tree: the root, branching points, and end points (marked by crosses in Fig.2E), indicating that primitive items are stored as attractors in the RNN. The detailed analysis is presented in Sec. B.2 in SI. Here, we sketch the main idea. We first use an optimization method to find out all fixed points of the RNN dynamics [4], which turn out to coincide with the 7 nodes of the tree. We then analyze the stability of each node. An example of stability analysis of node  $S_1^{1,2}$  is depicted in Fig.2D (lower panel), which shows that the real parts of all eigenvalues of the Jacobian matrix are small than 1, indicating that the node is stable with respect to noise perturbation.
2.4.3 Tree-structured attractor dynamics
We further analyze the transition dynamics of attractor states (nodes). The detailed analysis is presented in Sec. B.3 in SI. Here, we sketch the main idea. Consider node  $S_1^{1,2}$  in the first layer of the tree (Fig.2E). We linearize the RNN dynamics around the node and compute all eigenvectors of the Jacobian matrix. The larger the eigenvalue is, the more unstable the network state along the eigenvector under noise perturbation. As shown in Fig.2F, the most two unstable directions (the length of eigenvector reflecting the eigenvalue) point to nodes  $S_2^1$  and  $S_2^2$  in the second layer, respectively. This implies that starting from the node  $S_1^{1,2}$ , the network state can travel to the node  $S_2^1$  or  $S_2^2$  depending on the second item, but has difficulty to travel to other nodes.
In summary, the above results reveal that after training, the tree-structured attractor dynamics emerges in the RNN, which encode the ordinal structure of the learned temporal sequences. Upon receiving a temporal sequence, the network state evolves from the root to the leaf, node by node, following the order hierarchy of the temporal sequence. Notably, unlike the framework of heteroclinic channel, our model stores primitive items as stable attractors rather than saddle points, and the transition between two nodes requires the followed item to be the input drive. As demonstrated below, this attractor dynamics enables our model to discriminate temporal sequences robustly against temporal warping.
3 Processing temporal sequences with the ordinal structure
3.1 Disentangled representation of ordinal structure facilitates transfer learning
The key advantage of disentangled representation of order structure is that it allows flexible generation to new temporal sequences via conjunctive code, that is, the neural system can combine the stored order template with different contents to form different temporal sequences. In our model, the RNN learns to extract the abstract ordinal structure of temporal sequences in the form of tree-structured attractor dynamics. After training, the association between the RNN and the training data is released, and the RNN can be reused to encode new temporal sequences having the same or partial temporal structure. Mathematically, this can be done by freezing the recurrent connections in the RNN (and hence the tree-structured attractor dynamics), while only learning the feedforward connections from the input layer to the RNN and from the RNN to the output layer, when a new temporal sequence processing task arises. In such a way, it is expected that the pre-trained RNN helps the model to speed up the acquisition of new temporal sequences having the same or partial ordinal structure, achieving the so-called transfer learning.
To demonstrate the above idea, we design a transfer learning task. We consider that the RNN has extracted the ordinal structure of the synthetic data described in Sec.2.3, and we now apply the model to solve a new task of discriminating new four temporal sequences. These four temporal sequences are composed by three primitive phonemes chosen from the TIMIT dataset, which are "pcl", "tcl", and "pau" (for the details of sequence construction, see Sec. C in SI), and the four sequences are written as "pcl-tcl", "pcl-pau", "tcl-pcl" and "tcl-pau", which have the same order structure as the synthetic data (Fig.3A-B). Following the same procedure as described in Sec.2.2, we augment the training data by inserting time intervals between neighboring items, and choose their ramping target functions accordingly. During transfer learning, we freeze the recurrent connections in the RNN and only optimize the feedforward connections. As comparison, we also train the model without freezing the RNN. The results are shown in Fig.3A-B. We observe that: 1) after transfer learning, the RNN retains the tree-structured representation trajectories for the learned temporal sequences, indicating that the model adjusts the feedforward connections to match the tree-structured attractors of the