SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junfeng Zuo
zuojunfeng@pku.edu.cn
Xiao Liu
xiaoliu23@oku.edu.cn
Ying Nian Wu2
ywu@stat.ucla.edu
Si  $\mathbf{W}\mathbf{u}^{1}$
siwu@pku.edu.cn
Wen-Hao Zhang $^{3,4*}$
wenhao.zhang@outsouthwestern.edu
<sup>1</sup>Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies,
School of Psychological and Cognitive Sciences,
Beijing Key Laboratory of Behavior and Mental Health,
IDG/McGovern Institute for Brain Research,
Center of Quantitative Biology, Peking University.
$^{2}$ Department of Statistics, University of California, Los Angeles.
$^{3}$ Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center.
$^{4}$ O'Donnell Brain Institute, UT Southwestern Medical Center.
Time perception is fundamental in our daily life. An important feature of time perception is temporal scaling (TS): the ability to generate temporal sequences (e.g., movements) with different speeds. However, it is largely unknown about the mathematical principle underlying TS in the brain. The present theoretical study investigates temporal scaling from the Lie group point of view. We propose a canonical nonlinear recurrent circuit dynamics, modeled as a continuous attractor network, whose neuronal population responses embed a temporal sequence that is TS equivariant. We find the TS group operators can be explicitly represented by a time-invariant control input to the network, whereby the input gain determines the TS factor (group parameter), and the spatial offset between the control input and the network state on the continuous attractor manifold gives rise to the generator of the Lie group. The recurrent circuit's neuronal responses are consistent with experimental data. The recurrent circuit can drive a feedforward circuit to generate complex sequences with different temporal scales, even in the case of negative temporal scaling ("time reversal"). Our work for the first time analytically links the abstract temporal scaling group and concrete neural circuit dynamics.
We are living in a dynamic world and the brain can flexibly process sensory and motor events occurring at different time scales [1-4]. An example of such temporal flexibility is self-initiated movements, e.g., singing a song at normal or faster speed. The capability of temporally flexible movements implies the brain flexibly control the underlying neural circuit dynamics. Indeed, experimental data converges to an empirical principle of temporal scaling (TS) at the behavioral and neuronal levels [2-5]. Specifically, when generating movements with longer intervals, it was found the neuronal population activities evolve along the same (low-dimensional) manifold of the neuronal population
responses but at a slower speed [2-8]. Although this observation was reproduced in previous modeling studies that trained recurrent circuit models to achieve temporal scaling (e.g., [2, 6, 7, 9-15]), it is far from clear about the mathematical principle governing temporal scaling in brain's recurrent neural circuits.
Temporal scaling is also fundamental in machine learning (ML) and robotic research. Most of engineering approaches use time warping to achieve temporal scaling where sinusoidal oscillatory inputs are needed (e.g., [16-19]). However, no experiments support such oscillation-based signals are used to represent time information ranging from milliseconds to seconds in the brain [3-5]. Some other ML studies directly modulated time constants of artificial neurons in recurrent neural networks to realize TS (e.g., [20]), whereas real neurons' time constant is thought to be fixed due to biophysical properties. Earlier studies also introduced neurophysiological properties to achieve temporal scaling, e.g., using synaptic shunting to adapt the effective integrating time [21] and utilizing time-covariant time field sizes [22]. However, they ([21, 22]) focused on the scale invariance during the temporal scaling inference rather than generating temporal sequences with different speeds. Combined, current ML models adopt different mechanisms to realize TS than neural circuits in the brain.
To provide algebraic and dynamical understanding of temporal scaling in neural circuits, we rigorously study this question from temporal scaling (Lie) group. We contend that there are two aspects of realizing TS group transformations in recurrent neural circuits: One is explicitly representing a TS controller corresponding to TS group operators; and the other is representing a TS equivariant temporal sequence in spatiotemporal neuronal responses, referred to as TS equivariant representation. It is unknown how recurrent neural circuits represent abstract TS group operators, nor the equivariant representation of temporal sequences. And no previous ML study investigated the equivariance of a Lie group acting on the temporal domain, nor an explicit representation of group operators.
The present study analytically derives a canonical current circuit with continuous manifolds of attractors, so-called continuous attractor networks (CANs), to equivariantly represent generic sequences with different temporal scales. And the TS operators are realized by a time-invariant control input applied to the CAN: The TS factor is represented as the gain (magnitude) of the control input, and the TS (group) generator emerges from the spatial offset between the control input and the network state along the continuous attractor manifolds. Moreover, applying a negative (inhibitory) gain of control input enables the recurrent circuit to generate a time-reversed sequence. The proposed circuit model reproduces a wide range of experimental findings of temporal scaling. We also demonstrate the recurrent circuit can drive a feedforward circuit to generate arbitrary, complex temporal sequences with different time scales, e.g., hand-written sequences. It will shed light on understanding the flexible timing in the brain, and provides new building blocks to achieve timing tasks in engineering.
The contributions of our work are as follows: It for the first time analytically links the TS group equivariant representation to a biologically plausible recurrent neural circuit, and analytically identifies that TS group operators are represented by time-invariant control inputs applied to the circuit. The proposed circuit with fixed connectivity flexibly generates TS equivariant sequences by just modulating the gain of control inputs, which is a novel mechanism never used in ML studies (Discussion, comparison with other works). It also proposes a novel circuit mechanism generating time-flipped sequences ("time reversal") with receiving control inputs of negative gain.
2 The temporal scaling group
We formulate the temporal scaling transformation by using the Lie group. Consider a TS group  $\mathbb{S}$  whose group operator  $S(\alpha)\in \mathbb{S}$  will map a time scale  $t$  (the scale of the time axis) into a new scale  $\mathfrak{t}$ ,
$$
t \mapsto \mathbb {t} = S (\alpha) \cdot t = t _ {\text {s t a r t}} + \alpha t, \tag {1}
$$
where the symbol  $\cdot$  denotes the action of group operators, and  $\alpha$  is the temporal scaling factor.  $t_{\mathrm{start}}$  denotes the start timing, where the  $t_{\mathrm{start}} = t_0 = 0$  when  $\alpha \geq 0$ , and otherwise  $t_{\mathrm{start}} = t_{\infty}$  regards as the end timing of the original time scale. Across this study, we use  $t$  to represent a "standard" or reference physical time, and  $\mathbb{t}$  denotes the scaled time. Then we consider an arbitrary temporal sequence  $\mathbf{y}(t)$ , which can be high-dimensional and may be regarded as, e.g., hand movements, vocal sequences, etc. Changing the time scale  $t$  will transform  $\mathbf{y}(t)$  into a new sequence  $\mathbf{y}(\mathbb{t})$ ,
$$
\mathbf {y} (\mathbb {t}) = \hat {S} (\alpha) \cdot \mathbf {y} (t) = \mathbf {y} [ S (\alpha) \cdot t ] = \mathbf {y} (\alpha t), \tag {2}
$$
Figure 1: (A) An equivariant map between the time and a temporal sequence. (B) Left: A TS operator  $\hat{S}(\alpha)$  with  $\alpha > 0$  scales the time by a factor  $\alpha$  and then scale temporal sequences accordingly. Right: A TS operator  $\hat{S}(\alpha)$  with  $\alpha < 0$  flips and scales the sequence on time axis.
To simplify notation, we suppress  $t_{\mathrm{start}}$  in the last term in Eq. (2) and across the study, in that our focus is the temporal scaling of sequences rather than temporal alignment between them. Our actual calculations do align  $t_{\mathrm{start}}$  appropriately.  $\hat{S}(\alpha)$  is a TS operator acting on a sequence, in contrast to  $S(\alpha)$  acting on the time  $t$  directly. TS group operators are commutative, i.e., the effect of sequential actions of two operators can be composed into the same operator irrelevant to their order, i.e.,  $\hat{S}(\alpha) \cdot \hat{S}(\beta) = \hat{S}(\beta) \cdot \hat{S}(\alpha) = \hat{S}(\alpha\beta)$ . Other properties of TS group operators can be found in Supplementary Information (SI) (Eq. S1).
$\alpha > 1$  ( $0 < \alpha < 1$ ) corresponds to speed up (slow down) the time, analogous to compress (stretch) the sequence  $\mathbf{y}(t)$  along the time  $t$  axis (Fig. 1B, left).  $\alpha = 0$  is a particular case where the sequence remains at a constant value, analogous to "time freezing".  $\alpha < 0$  implies "time reversal", i.e., flipping the sequence over the time axis and then scaling it based on the absolute value of  $\alpha$  (Fig. 1B, right).
3 Disentangled neural representation of the sequence time and pattern
We focus on how temporal sequences  $\mathbf{y}(t)$  with different temporal scales can be flexibly generated by canonical neural circuits. There are two aspects of temporal sequence generation: One is representing the temporal pattern (or content) of sequences referred to "pattern" representation, e.g., representing a song A vs. song B; and another is representing the time and time scale which is referred to as "time" representation, e.g., singing a song with normal speed vs. 2x speed. We reason the time and pattern of sequences can be represented separately in neural circuit [8], in that sequences can be scaled over time whatever their temporal patterns are. Therefore, we posit a disentangled, two-stage circuit architecture (Fig. 2A): A recurrent circuit acts as an "neural clock" whose population responses of  $N$  neurons,  $u(t) = \{u_j(t)\}_{j=1}^N$ , represent the time information; and a feedforward circuit converts the spatiotemporal responses  $u(t)$  into the sequence with the desired pattern,  $\mathbf{y}(t)$ , e.g., a concrete song or the trajectory of hand-written digits. Mathematically, the disentangled circuit can be denoted as,
$$
\mathbf {y} (t) = F [ u (t) ], \tag {3}
$$
where  $F[\cdot ]$  is the feedforward circuit storing the pattern information. Since  $F[\cdot ]$  is memoryless over time and maps the instantaneous response  $u(t)$  into the instantaneous complex sequence value  $\mathbf{y}(t)$ , scaling the sequence  $\mathbf{y}(\mathfrak{t})$  over time can be realized by just scaling the recurrent circuit's response,
$$
\mathbf {y} (\mathbb {t}) = \hat {S} (\alpha) \cdot \mathbf {y} (t) = \hat {S} (\alpha) \cdot F [ u (t) ] = F \left[ \hat {S} _ {u} (\alpha) \cdot u (t) \right], \tag {4}
$$
where  $\hat{S}_u(\alpha)$  is the TS operator acting on neural responses  $u(t)$ , in contrast with  $\hat{S}(\alpha)$  acting on  $\mathbf{y}(t)$ .
In summary, realizing TS in the disentangled circuit only needs to modulate the recurrent circuit's response  $u(t)$  without changing the feedforward circuit  $(F[\cdot])$ . In contrast, generating different patterns only needs to change (functional) feedforward circuits, while keeping the recurrent circuit's response  $u(t)$  unchanged. The representation of time information in recurrent circuits was supported by a wide range of neurophysiology experiments in many brain areas, e.g., medial frontal cortex [6], hippocampus [24], striatum [25], and HVC in songbirds [26], etc. The disentangled architecture is also supported by a recent study on vocal motor circuits in a highly vocal rodent [27], where the motor cortex dictates the timing of songs, while the mid-brain or brain stem areas generate music notes (patterns).
Figure 2: (A-B) A disentangled neural circuit that represents the sequence's time and pattern information in the recurrent and feedforward circuits respectively. The recurrent circuit generates a neural sequence embedding the "time" manifold  $z$  (B top, adapted from [23]) regardless of the sequence pattern, and then the feedforward circuit maps the neural sequence into sequences with arbitrary patterns (B bottom). A control input (green circle) is supposed to modulate the temporal scale of the neural sequences generated by the recurrent circuit. (C) The recurrent circuit is modeled as a continuous attractor network (CAN) consisting of excitatory (E) and inhibitory (I) neurons. The E neurons are uniformly distributed in the one-dimensional  $z$  manifold of the generic sequence (y-axis in top figure, panel B). (D) The recurrent connections E neurons, which decays with the difference between two neurons' preferred values of the generic sequence. (E) The eigenvalues and eigenvectors of the CAN, where the eigenvector with the largest eigenvalue corresponds to the movement of neuronal responses along the "time" manifold. (F) Energy landscape of the CAN without receiving the time-scale control input. (G) A generic neural sequence is generated by the recurrent circuit when receiving a time-invariant control input.
3.1 A low-dimensional "time" manifold in recurrent circuit's responses
We next investigate how the time is represented in  $u(t)$  based on experimental observations. It was found that when generating movements with different time scales, recurrent circuit's responses  $u(t)$  evolve along the same low-dimensional manifold with different speeds [2-7]. In contrast, the temporal responses of single neurons under different time scales may not align together. Therefore, we posit the one-dimensional (1D) time  $t$  can be represented (embedded) in a 1D manifold in recurrent circuit responses  $u(t)$ . We denote by  $z$  the 1D manifold of time representation in the circuit, in order to distinguish with the reference time  $t$ . It is worth noting that the neural response evolution on the  $z$  manifold,  $z(t)$ , doesn't need to be linear with reference time  $t$ , and such a linear  $z(t)$  also lacks experimental support. Mathematically, it only requires that  $z(t)$  is a one-to-one (bijective) mapping with time  $t$  [3]. Combined, the recurrent circuit's response  $u(t)$  represents an internal "time"  $z$ , and in the rest of the paper, we will denote the response as a function of  $z(t)$ , i.e.,  $u[z(t)]$ .
What is the internal "time" manifold  $z(t)$  would look like? Experiments found there is a unique time for each neuron to reach its maximal response [2-7]. Sorting neurons based on the peak response timing, the population responses  $u(t)$  resemble a bump-profile activity traveling across neurons (Fig. 2B, top). Hence, the 1D "time" manifold  $z$  can be regarded as the axis of sorted neurons based on their peak timing (Fig. 2B, y axis). And the evolution of neurons' bump-profile responses on the  $z$  manifold forms a sequence  $z(t)$  representing the internal " elapsed time" [4, 5, 10, 14]. Eventually, the time can be explicitly read out by identifying the index of the neuron which reaches its maximal response, and the time scale ( $\alpha$  in Eq. 1) is reflected by the speed of the neural sequence.
3.2 Temporal scaling operator acting on recurrent circuit's responses
As experiments suggested the TS transformations scale the "time" manifold  $z$  rather than each individual neuron's temporal responses, we define  $\hat{S}_u(\alpha)$  in Eq. (4) as,
$$
u [ z (\mathfrak {t}) ] = \hat {S} _ {u} (\alpha) \cdot u [ z (t) ] = u [ z (S (\alpha) \cdot t) ] = u [ z (\alpha t) ]. \tag {5}
$$
And such a neural representation  $u[z(t)]$  is called equivariant with the temporal scaling group. To find the expression of  $\hat{S}_u(\alpha)$ , we consider an infinitesimal temporal scaling with  $\alpha$  close to one  $(\alpha \to 1)$ .
Reorganizing the scaled neural response as  $u[z(\alpha t)] = u[z(\exp (\ln t + \ln \alpha))]$ , and performing a first-order Taylor expansion with amount  $\ln \alpha$
$$
u [ z (\exp (\ln t + \ln \alpha)) ] \approx u [ z (t) ] + (\ln \alpha) \frac {d u [ z (t) ]}{d \ln t} = \left[ 1 + (\ln \alpha) t \frac {d z}{d t} \frac {\partial}{\partial z} \right] \cdot u [ z (t) ]. \tag {6}
$$
From the above equation, the TS generator acting on neural responses can be defined as,
$$
\hat {g} _ {u} = t \frac {d z}{d t} \frac {\partial}{\partial z} \equiv t \left(\partial_ {t} z\right) \partial_ {z}, \tag {7}
$$
which characterizes the tangential direction of the TS effect on neural responses. Note that  $\hat{g}_u$  cannot be simplified into  $\hat{g}_u' = td / dt$  by canceling  $dz$  and  $\partial z$ . This is because  $\hat{g}_u$  only scales the 1D "time" manifold  $z(t)$  in the high-dimensional neural response  $u[z(t)]$ , whereas  $\hat{g}_u'$  scales every neuron's responses. TS operators with arbitrary  $\alpha$  can be derived by composing many infinitesimal scaling transformations,
$$
\hat {S} _ {u} (\alpha) = \lim  _ {n \rightarrow \infty} \left[ \hat {S} _ {u} \left(\alpha^ {1 / n}\right)\right] ^ {n} = \lim  _ {n \rightarrow \infty} \left(1 + \frac {\ln \alpha}{n} \hat {g} _ {u}\right) ^ {n} = \exp \left(\ln \alpha \cdot \hat {g} _ {u}\right), \tag {8}
$$
Eq. (8) is also compatible with a negative  $\alpha$  by using the generalized logarithmic function (see Eq. S9). To find the dynamics of the scaled neural responses, taking the time derivative of the operator,
$$
\begin{array}{l} \partial_ {t} \hat {S} _ {u} (\alpha) = \partial_ {t} \exp \left(\ln \alpha \cdot \hat {g} _ {u}\right) = \sum_ {n} \frac {(\ln \alpha) ^ {n}}{n !} \partial_ {t} \hat {g} _ {u} ^ {n} = \sum_ {n} \frac {(\ln \alpha) ^ {n}}{n !} (1 + \hat {g} _ {u}) ^ {n} (\partial_ {t} z) \partial_ {z}, \tag {9} \\ = \exp [ (\ln \alpha) (1 + \hat {g} _ {u}) ] (\partial_ {t} z) \partial_ {z} = \alpha \hat {S} _ {u} (\alpha) (\partial_ {t} z) \partial_ {z}, \\ \end{array}
$$
where we utilize the result that  $\partial_t\hat{g}_u^n = (1 + \hat{g}_u)^n (\partial_tz)\partial_z$  (see Eq. S13 in SI). And then the dynamics of the scaled equivariant neural sequence is,
$$
\partial_ {t} u [ z (\mathfrak {t}) ] = \partial_ {t} \hat {S} _ {u} (\alpha) \cdot u [ z (t) ] = \alpha \hat {S} _ {u} (\alpha) (\partial_ {t} z) \partial_ {z} \cdot u [ z (t) ] = \alpha [ \partial_ {\mathfrak {t}} z (\mathfrak {t}) ] \partial_ {z} \cdot u [ z (\mathfrak {t}) ]. \tag {10}
$$
Scaling the neural dynamics will modulate its time derivative by  $\alpha$ , and the right-hand side (RHS) of the scaled dynamics (Eq. 10) depends on the scaled  $z(\mathbb{t})$  rather than the original  $z(t)$ . Although the TS effect on the dynamics can be also derived via the chain rule (e.g., [20]), deriving such an effect from Lie group gives us more insight on the algebraic structure of TS and its circuit representation.
3.3 A concrete neural representation of internal "time"
We next define a concrete neural representation of internal "time" manifold  $z$  based on experimental observations. We consider a parametric representation where each neuron is selective for the internal  $z$  value, and denote  $x_{j}$  as the preferred  $z$  value of the  $j$ -th neuron which implies the neuron will reach its peak firing rate when  $z(t) = x_{j}$ . Moreover, we consider the preferences of neurons,  $\{x_{j}\}_{j=1}^{N}$ , are uniformly distributed along the  $z$  manifold to simplify the math analysis. Since the population response has a bump-profile (Fig. 2B, top), we model the mean firing rate (tuning) of  $j$ -th neuron as a Gaussian function over  $z(t)$ , a widely used approach in neural coding studies (e.g., [28-30]),
$$
\bar {u} _ {j} [ z (t) ] = A _ {u} \exp \left[ - \left(x _ {j} - z (t)\right) ^ {2} / 4 a ^ {2} \right], \tag {11}
$$
where  $A_{u}$  is the peak firing rate, and  $a$  the tuning width. Note that  $\bar{u}_{j}$  regards as the mean neuronal response, in distinguish with the instantaneous  $u_{j}(t)$  which may deviate from the mean response. Since Lie group operators act on continuous functions (Eq. 5), to simplify the math analysis, we treat  $\bar{u}[z(t)]$  as a continuous function corresponding to an infinite number of neurons in the population  $(N \to \infty)$ , and all neurons' preferences become a continuous variable, i.e.,  $x_{j} \to x$ . Hence, the mean population firing rate of all neurons becomes a continuous function of the  $z(t)$ ,
$$
\{\bar {u} _ {j} [ z (t) ] \} _ {j = 1} ^ {N} \equiv \bar {u} [ z (t) ] = A _ {u} \exp \left[ - (x - z (t)) ^ {2} / 4 a ^ {2} \right]. \tag {12}
$$
4 A temporal scaling equivariant recurrent neural circuit model
We propose a biologically plausible recurrent circuit dynamics with fixed connectivity to flexibly generate TS equivariant neural sequences  $u[z(t)]$ , and study how the circuit can be controlled to scale the neural sequence. Specifically, the requirements specified by the TS group and neural representations (Eqs. 5-12) will be used as objectives of the proposed functional recurrent circuit.
4.1 The recurrent circuit dynamics
First, we consider a special case of freezing the internal "time" by applying  $S(0)$ , and hence  $z(t) = z(t_0)$  as the initial value. This can be regarded as, e.g., the arm position is fixed over time. Then the mean, stationary neuronal responses in the recurrent circuit given the  $z(t_0)$  need to match the Gaussian profile as indicated by Eq. (12). Moreover, since the initial value  $z(t_0)$  is arbitrary (e.g., the arm location can be stable at any location at any time), stationary neural responses are required to be stable at any value of  $z$ . This implies that the recurrent circuit dynamics should have a family of stationary states (attractors) along the  $z$  manifold. These kinds of networks are called continuous attractor networks (CANs), which have been widely used to explain the neural coding of continuous features (e.g., [31-33]). We firstly consider an autonomous CAN dynamics as follows [34-37],
$$
\tau \partial_ {t} u (x, t) = - u (x, t) + \rho \int J \left(x, x ^ {\prime}\right) r \left(x ^ {\prime}, t\right) d x ^ {\prime}, \tag {13a}
$$
$$
r (x, t) = G [ u (x, t) ] = \frac {[ u (x , t) ] _ {+} ^ {2}}{1 + k \rho \int [ u (x ^ {\prime} , t) ] _ {+} ^ {2} d x ^ {\prime}}. \tag {13b}
$$
where  $u(x,t)$  is the synaptic input received by the excitatory neuron preferring  $x$  (Fig. 2C, blue triangle), and  $\tau$  is the time constant.  $G(\cdot)$  is a nonlinear activation function modeled as divisive normalization, a canonical operation in the cortex [38], where  $k$  is the global inhibition strength coming from an inhibitory neuron pool (Fig. 2C, red disk).  $[\cdot]_{+}^{2}$  denotes rectifying the negative part followed by a square function.  $\rho$  is the neuronal density on the  $z$  manifold.  $J(x,x')$  denotes the synaptic weight from the pre-synaptic neuron preferring  $x'$  to the post-synaptic neuron preferring  $x$ , and is modeled as a Gaussian function over the difference  $|x - x'|$  (Fig. 2D),
$$
J \left(x, x ^ {\prime}\right) = J \left(x - x ^ {\prime}\right) = J _ {0} \exp \left[ - \left(x - x ^ {\prime}\right) ^ {2} / 2 a ^ {2} \right]. \tag {14}
$$
The  $J_{0}$  and  $a$  in Eq. (14) denote the peak recurrent weight and the connection width respectively. It can be checked that the autonomous recurrent circuit can self-consistently generate non-zero, desired Gaussian profile population responses  $\bar{u}(z)$  as specified in Eq. (12),
$$
\bar {u} (x - z) = A _ {u} \exp [ - (x - z)) ^ {2} / 4 a ^ {2} ] = \bar {u} (z), \tag {15}
$$