SlowGuess/NeurIPS_2023_OCR huggingface dataset:
as long as the peak recurrent weight  $J_{0}$  is larger than a critical value  $J_{c}$  (can be analytically solved, see Eq. S30 for details). Note that the  $z$  in the above equation is a free parameter, meaning the recurrent circuit model has a continuous family of attractors over the "time"  $z$  manifold (Fig. 2F). Here we use the notation  $\bar{u}(x - z)$  to denote the responses generated by the concrete recurrent circuit dynamics (Eqs. 13a-13b), in distinguishing with  $\bar{u}[z(t)]$  (Eq. 12) directly specified in earlier derivations.
To generate a moving neural sequence (dynamic  $z(t)$ ; Fig. 2B, top), one possibility is applying a time-invariant control input  $I(x)$  to the recurrent circuit (Fig. 2A, green disk), i.e., inserting  $I(x)$  into the RHS of Eq. (13a) (see Discussion for other mechanisms of generating moving neural sequences),
$$
I (x) = I \left(x \mid z _ {\infty}\right) = \mathrm {I} _ {0} \exp \left[ - \left(x - z _ {\infty}\right) ^ {2} / 4 a ^ {2} \right]. \tag {16}
$$
$\mathrm{I}_0$  is the peak controlling input intensity, and  $z_{\infty}$  is the final position of the neural sequence (the final location at y-axis of Fig. 2B, top). Then if the neural response is initiated at  $u(x,t_0) = \bar{u} [x - z(t_0)]$ , it will move along  $z$  manifold towards  $z_{\infty}$  and generates a neural sequence. Fig. 2G shows a moving neural sequence generated by the recurrent circuit model when receiving such a control input.
4.2 Identify temporal scaling operators in the recurrent circuit dynamics
We pursue a theoretical understanding of how the recurrent circuit dynamics represents temporal scaling operators. Since an TS operator  $\hat{S}_u(\alpha)$  is an exponential map based on the TS factor  $\alpha$  and the TS generator  $\hat{g}_u$  (Eq. 8), we specifically identify how the recurrent circuit represents  $\alpha$  and  $\hat{g}_u$ .
The TS generator (Eq. 10) requires the neural dynamics only preserves perturbations along the  $z$  manifold while filters out perturbations along other directions. This is because the time derivative of the neural response is only proportional to the partial derivative over the 1D  $z$  manifold, i.e., setting  $\alpha = 1$  in Eq. (10) yields  $\partial_t u[z(t)] = (\partial_t z)\partial_z u[z(t)]$ . To test whether this requirement can be satisfied by the proposed recurrent circuit, we performed perturbative analysis of the circuit dynamics by considering the instantaneous network state as perturbed from the attactor state, i.e.,
Figure 3: Temporal scaling of the neural sequence in the proposed recurrent circuit by gain modulation on the time-invariant control input. (A) Left: neural sequence generated with different control input gains. Right: The actual temporal scale factor with the control input gain. (B) An example excitatory neuron's temporal responses (top) and responses scaled over time (bottom) in forward (left) and reverse (right) conditions. Color: input gain as suggested by dots in (A). (C) The dimensionality reduction of the neuronal responses at different control input gains. (D) The time interval (absolute value) decreases with control input gain. (E) Weber's law in the recurrent circuit where the standard deviation of the time interval is proportional to the mean interval.
$u(x,t) = \bar{u} (x - z) + \delta u(x,t)$  (ignoring the time dependence on  $z$  for brevity). Substituting this expression into the network dynamics (Eq.13a) yields the perturbation dynamics,
$$
\tau \partial_ {t} \delta u (x, t) = - \delta u (x, t) + \int K (x, x ^ {\prime} | z) \delta u \left(x ^ {\prime}, t\right) d x ^ {\prime}, \tag {17}
$$
where  $K(x,x^{\prime}|z) = \rho \int J(x - x^{\prime \prime})\partial \bar{r} (x^{\prime \prime} - z) / \partial \bar{u} (x^{\prime} - z)dx^{\prime \prime}$ . Treating  $K(x,x^{\prime}|z)$  as an operator acting on  $\delta u(x,t)$ , its eigenvalues  $\lambda_{n}$  can be calculated as (descending order, Fig. 2E) [37, 39],
$$
\lambda_ {1} = 1, \quad \lambda_ {2} = 1 - \sqrt {1 - J _ {c} ^ {2} / J _ {0} ^ {2}}, \quad \lambda_ {n} = 2 ^ {2 - n} (n \geq 3), \tag {18}
$$
and the (unnormized) eigenfunction  $f_{1}(x|z)$  with the largest eigenvalue  $\lambda_{1} = 1$  is calculated as,
$$
f _ {1} (x | z) \propto \partial_ {z} \bar {u} (x - z) \propto - (x - z) \exp [ - (x - z) ^ {2} / 4 a ^ {2} ], \tag {19}
$$
which is the partial derivative of neural response over the  $z$  manifold. The above analysis proves the CAN does satisfy the requirement of the TS generator approximately (Eq. 8), i.e., only preserving the perturbation along the  $z$  manifold  $(\lambda_{1} = 1)$  and removing other perturbations since their eigenvalues are smaller than 1. This is a unique property of the CAN which comes from the translation-invariant recurrent weights along the  $z$  manifold (Eq. 14).
We present simple derivations to identify the TS generator in the recurrent circuit with a control input (rigorous derivations, SI. Sec. 3). The control input will deviate the network state  $u(x,t)$  from the attractor state  $\bar{u}(x - z)$ . Hence, our theory considers the weak limit of control input (small  $I_0$ ), and substitute the attractor states (Eq. 15) into the circuit dynamics (Eq. 13a),
$$
\tau \partial_ {t} \bar {u} (x - z) \approx [ - \bar {u} (x - z) + \rho J * \bar {r} (x - z) ] + I (x | z _ {\infty}) \approx I (x | z _ {\infty}), \tag {20}
$$
where the decaying term  $-\bar{u} (x - z)$  and the recurrent input term  $\rho J*\bar{u} (x - z) = \rho \int J(x,x')\bar{r} (x' - z)dx'$  will cancel each other due to the definition of attractor states (Eq. 15). Then we treat the control input  $I(x|z_{\infty})$  as a "perturbation" of the attractor state, and decompose it using eigenfunctions,
$$
\tau \partial_ {t} \bar {u} (x - z) \approx I (x | z _ {\infty}) = \sum_ {n} a _ {n} f _ {n} (x | z) \approx a _ {1} f _ {1} (x | z) = a _ {1} \partial_ {z} \bar {u} (x - z), \tag {21}
$$
where we ignore perturbations along directions parallel to eigenfunctions  $f_{n}(x|z)$  ( $n \geq 2$ ) because they will eventually vanish (Eq. 18). The coefficient  $a_1$  can be obtained by computing the inner product between the left and right-hand side of the above dynamics with  $f_{1}(x|s)$ , e.g.,  $a_1 = \langle \tau \partial_t\bar{u} (x - z),f_1(x|z)\rangle = \tau \int \partial_t\bar{u} (x - z)f_1(x|z)dx,$
$$
a _ {1} = \tau \partial_ {t} z = \mathrm {I} _ {0} \cdot A _ {u} ^ {- 1} \left(z _ {\infty} - z\right) \exp \left[ - \left(z _ {\infty} - z\right) ^ {2} / 8 a ^ {2} \right]. \tag {22}
$$
Combining the above two equations, the proposed recurrent circuit with the control input approximately achieves the TS equivariance: its attractor state  $\bar{u} (x - z)$  (Eq. 15) travels along the "time"  $z$  manifold in a way consistent with the scaled dynamics derived from TS group (Eq. 10,  $\alpha = 1$ ),
$$
\partial_ {t} \bar {u} (x - z) \approx (\partial_ {t} z) \partial_ {z} \bar {u} (x - z).
$$
Figure 4: Generating a sequence pattern by feedforward circuit mapping. (A-B) A proof of concept example where the feedforward circuit transforms the neural sequence in the recurrent circuit into the  $x$  and  $y$  coordinates of a hand-written sequence of the digit '6'. (C) The hand-written sequence at different control input gains in the recurrent circuit. Color: elapsed time  $t$ . (D) The temporal trajectories of the hand-written sequence at forward (left) and reverser (right) order. (E) The actual temporal scale factor of hand-written sequence is proportional to the control input gain.
In practice, the actual responses  $u(x,t)$  will deviate from the attractor state  $\bar{u}(x - z)$  due to the control input and noises. The recurrent dynamics will quickly remove distortions perpendicular to the  $z$  manifold, corresponding to pull  $u(x,t)$  back to the continuous  $z$  manifold, and then the attractor state  $\bar{u}(x - z)$  will travel along the  $z$  manifold which internally represents " elapsed time".
The emergence of the TS generator. A non-vanishing TS generator  $\hat{g}_u = t(\partial_t z)\partial_z\propto ta_1\partial_z$  (Eq. 8) needs a non-zero  $a_1$ , which can be satisfied with a non-zero offset,  $z_{\infty} - z$ , between the final state  $z_{\infty}$  and the initial state  $z_0$ . Therefore, the spatial offset of control input and the initial network state on the  $z$  manifold emerges the TS generator in the recurrent circuit dynamics.
The representation of the TS factor. Scaling neural sequences over time corresponds to multiplying the internal "time" dynamics  $\partial_t z \propto a_1$  by the scaling factor  $\alpha$  (Eq. 10). Since  $a_1$  is proportional to the control input strength  $\mathrm{I}_0$  (Eq. 22), scaling the neural sequence can be realized by the gain modulation on the control input by the scaling factor  $\alpha$ , i.e.,  $I(x|z_{\infty}) \to \alpha I(x|z_{\infty})$  (Eq. 16). And then the whole TS equivariant recurrent circuit dynamics becomes ( $*$  denotes the convolution over  $x$ ),
$$
\tau \partial_ {t} u (x, t) = - u (x, t) + \rho J * r (x, t) + \alpha \cdot \mathrm {I} _ {0} \exp \left[ - \left(x - z _ {\infty}\right) ^ {2} / 4 a ^ {2} \right]. \tag {23}
$$
Therefore, the TS factor  $\alpha$  determines the control input gain in the recurrent circuit. Generating sequences with different temporal scales can be realized by simple gain modulation, which is a widely observed operation in neural circuits (e.g., [40, 41]). Interestingly, by applying a negative input gain  $(\alpha < 0)$  the circuit dynamics can implement "time reversal", i.e., a flipped sequence over time.
5 Simulation experiments
We simulated the proposed circuit model to test temporal scaling via manipulating the control input's gain (SI. Sec. 4.1, network simulation detail). Changing the input gain  $\alpha$  varies neural sequences' time scales (Fig. 3A, left), and the actual scaling factor is proportional to the input gain  $\alpha$  as predicted by our theory (Fig. 3A, right). It is noteworthy that a negative gain enables the circuit to render a "time reversed" neural sequence, corresponding to flipping the original sequence on the time axis (comparing heatmaps in Fig. 3A, left). At the single neuron level, temporal scaling also changes the temporal speed of single neurons' responses (Fig. 3B). The single neuron's temporal responses under gains with the same polarity, but not different polarities, can be scaled and overlap well over time. Moreover, the dimensionality reduction analysis shows neural sequences under both positive and negative control input gain all evolve along the  $z$  manifold and overlap perfectly (Fig. 3C). These results are consistent with the experimental finding that single neurons responses might not overlap well under different time scales, but the population responses at different time scales all evolve along the same low-dimensional manifold and overlap perfectly [2-7].
A characteristic of temporal scaling at the behavioral level is Weber's law, meaning the standard deviation of the time duration of a sequence linearly increases with the mean duration. To reproduce this phenomenon in the model, we include independent multiplicative noise in the recurrent circuit
mimicking the Poisson spike variability (inserting  $\sqrt{\tau\mathsf{F}[u(x,t)]_+}\xi (x,t)$  to the RHS of Eq. (23) with  $\mathsf{F}$  the Fano factor). The mean time duration (Eq. S34, analytical solution) decreases with the input gain  $\alpha$ , implying the internal "time" representation becomes faster (Fig. 3D). Meanwhile, the standard deviation of the time duration is proportional to the mean duration, consistent with Weber's law.
The recurrent circuit's population responses  $u[z(t)]$  can be mapped to an arbitrary, complex sequence  $\mathbf{y}(t)$  through a feedforward circuit (Eq. 3). As a proof of concept, we use a feedforward circuit (modeled as a three-layer perceptron) to transform  $u[z(t)]$  into a 2D sequence ( $x$  and  $y$  coordinates) of hand-written digits, e.g., digit "6" (Fig. 4A-B, see details in SI. Sec. 4.3). The feedforward circuit was trained via back-propagation by only using the neural response and the hand-written sequence (Fig. 4B) at only one temporal scale. After training, we test whether the whole disentangled circuit can generalize the hand-written "6" sequence at other time scales by manipulating the control input's gain ( $\alpha$  in Eq. 23). Indeed, the feedforward circuit trained at one temporal scale successfully generates hand-written sequences over other time scales (Fig. 4C-D), including both positive (forward sequence) and negative (reversed sequence) temporal scales. The disentangled circuit can also generate hand-written sequences of other digits once we retrain the feedforward circuit (see SI. Figures).
The present study investigates the neural circuit mechanism of temporal scaling group equivariant representation. We propose a disentangled neural circuit that represents the pattern (content) and the time information of temporal sequences separately. The timing information is equivariantly represented by neural sequences generated in a recurrent circuit modeled as a CAN, and the sequence pattern (content) is represented in a feedforward circuit that maps the generic neural sequence to arbitrary temporal sequences. Moreover, TS operators are explicitly represented as a time-invariant control input applied to the CAN: the TS factor determines the control input gain, and the TS generator emerges by the spatial offset between the control input and network state along the continuous attractor manifold representing "time". Eventually, modulating the control input gain enables the recurrent circuit with fixed connectivity to generate sequences with different temporal scales, including "time reversal" when using a negative gain. Our study for the first time formulates the temporal scaling problem from the Lie group, and links the abstract group to a concrete, biologically plausible neural circuit. It gives us insight into the math principle underlying TS representation in neural circuits, and the proposed model may inspire a new building block for equivariant representation in ML tasks.
Comparison to other works. How temporal scaling is achieved in neural circuits has been an active topic (e.g., [2, 3, 6, 9, 13, 14, 42-44]). A large body of earlier neural circuit modeling studies investigated this question via training a recurrent network, e.g., a reservoir network, to represent different time scales (e.g., [6, 9, 11, 14]). Although the trained networks achieve the TS representation, the underlying math principle is still lacking. In contrast, the present paper analytically derives a recurrent circuit implementation of TS equivariant representation, and the explicit representation of TS group operators. Moreover, our hand-crafted circuit model generalizes over temporal scales well, which outperforms recent neural circuit models where the trained network cannot generalize (extrapolate) well to temporal scales beyond the range in the training set [11, 27]. In addition, previous studies considered generating a moving neural sequence by adding an anti-symmetric recurrent weight component in the recurrent network (e.g., [45-48]), whose magnitude increases with temporal speed (or TS factor) [37, 47]. Nonetheless, the TS in the brain happens in short time scales (milliseconds to a second), which is too short to modulate recurrent weights. In contrast, the control input gain in the present study can be quickly modulated in short time scales. Last, previous studies proposed TS covariant networks, inspired by the fact that the width of the temporal field of hippocampal time cells covaries (increases) with the preferred elapsed time (e.g., [22]). In comparison, the TS covariant networks aim to achieve TS invariant representation that is beneficial for recognizing sequences with different temporal scales, while our network model is flexibly generating sequences with different temporal scales. When inverting the information flow and introducing a feedback loop between the control input and the recurrent network, our model also has the potential to recognize (infer) the input sequences and their temporal scaling factors.
Equivariant representation is an active topic in ML research and lots of equivariant neural networks have been proposed (e.g., [49-53]). Nonetheless, to our best knowledge, previous ML studies exclusively investigated the equivariance under group transformations in spatial domain rather than
temporal domain. Moreover, there have not been many ML studies investigating recurrent network mechanisms of equivariance while most of them studied feedforward equivariant networks. Timing or TS representation in ML models typically uses a mechanism called time warping, which relies on a pacemaker (central clock) producing sine waves with different frequencies to represent times. However, the pacemaker mechanism doesn't exist in the nervous system representing time scales ranging from the order of milliseconds to seconds [3]. There are recent ML studies investigated temporal scaling recurrent networks [20], where they directly modulated the time constant of neural dynamics (i.e.,  $\tau$  in Eq. 13a). However, the time constant of real neurons hardly changes due to biophysical properties. And our way of achieving TS by gain modulation of the amplitude of the time-invariant control input is novel.
Limitations of the model. In order to gain a theoretical understanding of the TS equivariant representation, we propose a hand-crafted recurrent circuit model rather than training a neural network model as in previous studies [9-11, 14, 48]. Moreover, we consider a simple model whose recurrent connections do not have random components (Eq. 14). It is interesting to study how a TS equivariant recurrent network can be trained in an unsupervised way. In addition, previously trained network models used reservoir networks containing strong random connections with disordered (chaotic) dynamics [9-11, 14, 48]. Although chaos provides a rich dynamical repertoire to facilitate learning, it impairs the robustness of time representation [9]. Since this study does not consider learning, ignoring the random connections will significantly help us identify how structured recurrent connections achieve TS equivariance and represent operators. Statistically, a characteristic of chaotic spiking networks is the Poisson spiking variability [54, 55], whose statistical effect is equivalent to the injected multiplicative noise in our model. Furthermore, as a proof of concept, the feedforward circuit used in our model is simple and needs to be retrained to generate a new sequence pattern, e.g., other digits' written sequence, which is unlikely in the brain. One way to improve this is by using a compositional model (e.g., [56, 57]) which can be flexibly modulated to change its represented pattern.
Extensions of the model. The proposed neural circuit model has the potential to explain other brain functions. For example, our model may be used to explain the memory formation in the hippocampus, where it is believed that hippocampal theta oscillation compresses (speeds up) sensory input sequences by about 10 times and is beneficial for memory formation [58]. It is possible that the theta oscillation shares a similar computational mechanism with the proposed temporal scaling equivariant recurrent network. At last, experiments found "time" cells in the hippocampus whose temporal firing width (duration) is proportional to their peak timing, i.e., a "time" cell fires later will fire longer, and time cells' preferred peak timing is log-scaled [59]. Such a relation doesn't exist in the proposed model, and neurons in the current model can be "deployed" in the logarithmic space of  $z$  manifold to reproduce this effect. All of these will form our future research.
Y. W. is supported by NSF DMS-2015577. S. W is supported by National Science and Technology Innovation 2030 Major Program (No. 2021ZD0200204, Si Wu).
[58] William E Skaggs. Relations between the theta rhythm and activity patterns of hippocampal neurons. The University of Arizona, 1995.
[59] Rui Cao, John H Bladon, Stephen J Charczynski, Michael E Hasselmo, and Marc W Howard. Internally generated time in the rodent hippocampus is logarithmically compressed. *Elife*, 11:e75353, 2022.