SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we sought to display the converse, namely that our similarity method can identify differences in dynamical structure even when spatial analysis cannot. To do so, we examined a set of three noisy recurrent networks from Galgali et al. [2023]. These systems implement a bistable pair of attractors, a line attractor, and a single stable fixed point at the origin. These systems describe three strategies in a classical perceptual decision-making task from systems neuroscience: unstable evidence integration, stable integration, and leaky integration. Their dynamical equations are included in the Supplementary Information. Here, the inputs to the latter two networks were adversarially optimized so that the condition-averaged trajectories are indistinguishable from the first network (Fig. 3a). These networks have intrinsic noise in their hidden states, which enables DSA to distinguish systems beyond condition averages. We simulated 200 noisy trajectories from 100 networks per class, each with randomly sampled parameters.
Figure 3: Different dynamics, same shape, only distinguished by DSA. a. Sample single-trial (faded) and condition-averaged trajectories of the three systems in two dimensions (x and y axes). Trials start at the center point and move away in a direction depending on the condition (colored). b. Procrustes-MDS on the dissimilarity matrices of the three network types. c. DSA-MDS of the three networks. d. Classification test accuracy of condition-averaged Procrustes and DSA similarity spaces. Dotted line indicates chance.
To confirm that the condition-averaged trajectories are indistinguishable, we visualize the condition-averaged and single-trial trajectories in Fig. 3a for each system. We fit HAVOK models with 100 lags and a rank of 50 for each network and computed DSA and Procrustes pairwise (5,000 total comparisons). As before, we plot a low-dimensional MDS projection of these dissimilarity matrices (3). When the two types of similarities are compared side by side (spatial similarity, Fig. 3b, and DSA, Fig. 3c), it is evident that only our method is capable of easily distinguishing dynamic types from the data alone. This is quantified by almost perfect test accuracy via a linear classifier in Fig. 3d. Thus, DSA can efficiently extract features that describe the underlying dynamical system from data alone, which better reflects the nature of the computation that is being performed in RNNs.
3.4 DSA is invariant under geometric deformations that preserve attractor topology
Next, we examined how DSA responds when smooth geometric deformations are applied to a dynamical system. Here, we chose to study a ring attractor network, a system known to track head direction in the brain, and whose topology is a ring embedded in  $n$  neurons (Zhang [1996], Chaudhuri et al. [2019], Skaggs et al. [1995]). The neural activity on the ring is instantiated as a bump of local activity, which is stable due to the localized and translationally-symmetric connectivity profile. In each trial, we randomly sampled  $n$  uniformly between 100 and 250, and drove the network with constant input and dynamical noise, leading the bump to rotate around the ring at a roughly constant velocity (we used code from Wang and Kang [2022]). The network's dynamics are detailed in the Supplementary Information (16). Importantly, after simulation, the saved activations can be transformed by a continuous deformation that preserves the ring topology while changing the geometry:
$$
r = \frac {1}{1 + \exp (- \beta s)} \tag {7}
$$
The variable  $s$  describes the synaptic activations of the ring network (the dynamical variable), and  $r$  describes the neural activations, which we compare with DSA and Procrustes. Here,  $\beta$  controls the width of the bump. As  $\beta$  increases, the width of the bump becomes progressively tighter, which makes the ring less planar (see Fig. 1 in Kriegeskorte and Wei [2021] for a depiction). We scaled the magnitude of  $\beta$  from 0.1 to 4. In the inset of Fig. 4b, we quantified the ring's dimensionality across  $\beta$ , measured as the participation ratio of the principal components' eigenvalues:  $PR = \frac{(\sum_{i} \lambda_{i})^{2}}{\sum_{i} \lambda_{i}^{2}}$  (Gao et al. [2017]).
After applying these transformations to simulated trajectories from our network, we applied DSA and Procrustes to respectively compare all  $\beta$ s to the lowest  $\beta$  in our dataset. In Fig. 4a we visualize the hidden state trajectory in the first two PCs as the ring is progressively transformed with  $\beta$ . We fit HAVOK models with a lag of 10 and a rank of 1000 to trajectories at each level of deformation. When we compared the network using DSA and Procrustes, we found that only DSA is invariant across these transformations (Fig. 4b). While the value for DSA is not strictly zero, this may be due to approximation or numerical error in HAVOK and the similarity metric, as it remains close to zero
Figure 4: DSA is invariant under geometric deformation of a ring attractor, and sensitive to the transformation of a ring attractor into a line. Top row displays scaling of  $\beta$  in the sigmoid, making the ring progressively less planar. The bottom row displays topological transformation of a ring into a line via  $\alpha$ . The network is a curve line attractor at  $\alpha < 1$ , and is a ring attractor at  $\alpha = 1$ . a, c. Trajectories along the ring plotted in the top two PCs of the network, colored by magnitude of the deformation parameter. In c, trajectories are scaled for visualization purposes. b,d. Distance (D) to  $\beta = 0.1$ , and  $\alpha = 0.0$  in both Procrustes and DSA. Importantly, the topology only changes in the last row, at  $\alpha = 1.0$ . Inset b: Participation Ratio of the ring, a natural continuous measure of dimensionality based on explained variance ratios of PCA eigenvalues (Gao et al. [2017]). Inset d. Persistent Homology Analysis of the ring: top 2 lifespans of Betti number 1 features, indicating that the data becomes more ring-like as  $\alpha$  increases.
throughout (between 0.05 and 0.12). This suggests that DSA identifies similarity at the level of the topology of a dynamical system.
3.5 DSA responds to changes in topology
Finally, we assessed the converse to the previous analysis: does DSA respond to changes in topology? Using the same ring attractor network, we continuously varied the strength of its periodic boundary conditions to convert the ring into a curved line segment attractor by ablating some neurons from the network on one end of the ring. Our ring is specified by a cosine kernel that defines how each neuron connects to its neighbors. This kernel has a length parameter  $l$  which specifies how far apart two neurons can connect. To completely eliminate the boundary conditions,  $l$  neurons must therefore be ablated. Using an ablation length  $c$  as the number of neurons to ablate in the ring, we defined the following parameter to normalize the transformation across different network sizes.
$$
\alpha = 1 - \frac {c}{l} \tag {8}
$$
When  $\alpha = 1$ , there is no ablation and the network is perfectly a ring. When  $\alpha < 1$ , the network breaks into a line segment attractor. As  $\alpha \to 0$  the ring becomes progressively less curved. To ensure that network sizes were consistent across all values of  $\alpha$ , we initialized the network size to be  $n + c$  before ablation, where  $n$  is the desired final number of neurons. We simulated networks of sizes between 200 and 300 neurons. Instead of a constant drive of input as before, we drove the network with a fixed magnitude input  $u$  whose sign flipped stochastically. We chose this input to prevent the bump of activity from getting trapped on one end of the line attractors.
When we compared each value of  $\alpha$  of the network to  $\alpha = 0$  with DSA and Procrustes, we identified that DSA reports values close to 0 until  $\alpha$  becomes close to 1.0, when it jumps almost to  $\pi /2$ . We plotted these results in Fig. 4, where panel c depicts the low-dimensional visualization of the line and the ring, and d displays our metric's response. Note that because the data are noisy, we expected DSA
to jump slightly before 1.0, which we empirically verified to be around  $\alpha = 0.9$  in our simulations. Here we used only 10 delays (for a total embedding size between 2000 and 3000) and rank 100, but found that our results generalized to much larger DMD models as well. This further solidifies the notion that DSA captures topological similarity of two dynamical systems.
Figure 5: DSA unsupervisedly disentangles learning rules. a. Schematic example of different algorithms minimizing  $3x^{2} + 2y^{2}$ . b. DSA MDS plot of hidden activation observables from networks trained on ImageNet c. DSA MDS plot of the dynamics of learning in 3-layer linear networks. d. Test accuracy of a linear decoder on the DSA similarity space for the model classes. Bars indicate standard deviation over a wide range of DSA hyperparameters.
3.6 DSA can disentangle learning rules in an unsupervised fashion
It is still unclear what learning rules the brain uses. Recent progress includes the design of novel biologically-plausible learning rules (Lillicrap et al. [2016], Hinton [2022]), experiments and modeling work that suggests signs of gradient-based learning in the brain (Sadtler et al. [2014], Humphreys et al. [2022], Payeur et al. [2023]), supervised methods to classify learning rules from aggregate statistics in neural data (Nayebi et al. [2020]), and theory describing how representations evolve across training in different learning rules (Bordelon and Pehlevan [2022]). Because artificial networks are initialized with random seeds, their trajectories in the activity space across learning will almost always have different geometry, despite the fact that they may converge to the same minimum. However, because the trajectories arise from the same dynamical system, DSA should be able to capture similarities between learning trajectories from different initializations. We visualize a variety of learning rule trajectories in Fig. 5a to demonstrate how qualitatively different their dynamics can be. We applied DSA on two test cases from the literature to demonstrate that we can empirically disentangle artificial networks trained with different learning rules. To do so, we fit DSA to trajectories of the neural representations (or observables) across training epochs.
In the first case, we drew 21 aggregate statistics from the dataset from Nayebi et al. [2020], which describe the evolution of synaptic weights and activations of layers in various convolutional neural networks trained on multiple tasks with four learning rules. The learning rules utilized in the dataset include Adam (Kingma and Ba [2014]), Stochastic Gradient Descent with Momentum (Sutskever et al. [2013]), Information Alignment (Kunin et al. [2020]), and Feedback Alignment (Lillicrap et al. [2016]). For each analysis, we fixed one task and generated 50 datasets by randomly selecting (without replacement) subsets of trajectories for each learning rule. On each dataset, we fit a HAVOK model with a rank of 45 and lag of 4 to trajectories of these statistics across training. We compared each model pairwise, which results in a dissimilarity matrix. After comparing trajectories of these observables across training with DSA, we visualize the MDS of the similarity space in Fig. 5b. The training dynamics clearly cluster by learning rule under DSA. Quantifying the degree of clustering, we could classify the learning rule with  $88.5\%$  test accuracy using only a linear classifier and 5-fold cross-validation, whereas the linear SVM classifier in Nayebi et al. [2020] only reached  $\sim 55\%$  accuracy (Fig. 2a. in their paper). The test accuracy over a wide range of DSA hyperparameters was  $81\%$  (Fig. 5d).
In the second case, we trained a set of small 3-layer feedforward neural networks (with a hidden size of 100 units) trained to solve a multivariate regression task from Bordelon and Pehlevan [2022]. The learning rules in the second set include Gradient Descent, two forms of Feedback Alignment that vary based on the initialization of the random feedback weights (Boopathy and Fiete [2022]), and Hebbian learning (Hebb [1949]). For each learning rule, we trained 50 networks on 1000 epochs with random initializations and randomly generated data. We fit HAVOK models with a delay of 100 and a rank of 500 onto individual networks' activations across learning. As above, we found that
the MDS visualization of these networks clustered by learning rule in the pairwise similarity space (Fig. 5c). The two types of Feedback Alignment cluster together, suggesting that their dynamics are similar, as might be expected. Once again utilizing a linear classifier, we achieved  $89\%$  test accuracy after assigning both types of FA as the same class, and on average  $80\%$  accuracy over a range of hyperparameters (Fig. 5d). These results suggest the DSA could be used in a hypothesis-driven fashion to assess learning rules in data recorded from biological circuits.
We demonstrated that our novel metric, DSA, can be used to study dynamics in neural networks in numerous scenarios, including the comparison of dynamics between different RNNs, dynamics of a single RNN across training, or dynamics of weight and activity changes in feedforward networks over training. We performed an ablation study in Fig. 1f, which demonstrates that both delay embedding and the DMD were required for DSA to succeed. While we applied our metric only to RNNs, we stress that it can be applied to any dynamical system, including diffusion models or generative transformers. Procrustes Analysis over Vector Fields could also be applied to Koopman Embeddings identified via other algorithms, such as an autoencoder (Lusch et al. [2018]).
As suggested by Figs.2, 3, 4 and our theoretical grounding (9, 10), DSA can capture similarity at the level of topology of the dynamics, also known as conjugacy. This can also be recognized from the methodology of DSA: the similarity transform is a conjugacy between two linear systems. In recent work, Redman et al. [2023] used the 2-Wasserstein Distance over the Koopman eigenspectrum to compare learning dynamics of fully-connected networks trained to classify handwritten digits. This metric is quite analogous to ours in its theoretical motivation. In Supplementary Information 9 we prove that their metric is a specific case of DSA. Thus, DSA quantifies how far two dynamical systems are from being conjugate.
As we demonstrate in Fig. 3, our method could be used to quantitatively compare various hypotheses about the dynamics of a particular biological neural circuit (such as fixed point structure), by calculating similarities of various RNNs or bespoke models to data (Pagan et al. [2022], Schaeffer et al. [2020]). This task is typically computationally intensive, relying on assessing the dynamics of noise perturbations in the neural state space (Chaudhuri et al. [2019], Galgali et al. [2023]) or computing the locations of fixed points in a task-optimized RNN model that has similar spatial trajectories as the neural data (Mante et al. [2013], Chaisangmongkon et al. [2017]). Our method will be useful in the analysis of biological neural systems, where only recorded neural activity is accessible. To our knowledge, DSA is the first that can be used to quantitatively assess different dynamical hypotheses. We suggest that this method should be used in conjunction with shape analyses when assessing fits of computational models to data. In future work, we plan to apply DSA to experimental data.
DSA could potentially be used for rapid alignment of Brain-Computer Interfaces (BCI) across days in a subject. Non-stationary sampling of neurons from intracortical electrode shift, neuron death, and other noise plague longitudinal BCI trials, requiring the subject to relearn BCI control daily in a slow and tedious process (Biran et al. [2005], Perge et al. [2013]). Two recent approaches (Willett et al. [2023], Degenhart et al. [2020]) utilized Procrustes Analysis in their decoding and alignment pipeline. As demonstrated in Figs. 2, 3, and 4, an alignment method that takes dynamics into account will be useful. Karpowicz et al. [2022] utilized LFADS to align dynamics across days, but DSA may provide equal performance with substantially higher data efficiency and less compute due to the method's simplicity.
In Section 3.6 we demonstrated how the DSA metric space could be used in a hypothesis-driven fashion to compare neurophysiological data recorded across training to artificial neural networks trained with different learning rules. To do so, one would train a variety of neural networks with different learning rules to solve the same task as the animal model. By applying DSA pairwise to each network and the neurophysiology data, and identifying which learning rules cluster to the experimental data, we can reject other learning rule hypotheses. DSA has multiple advantages over previous methods for this problem: Unsupervised feature generation can improve classification capabilities, and abstraction from geometrymakes the method more generically applicable across systems of different types and initial conditions.
Limitations It can be challenging to identify the optimal DMD hyperparameters, but it is not computationally intensive to perform a grid sweep. Additionally, we found that the ranks and delays required to achieve our results were relatively small. On a V100 GPU, a single DSA took between 30 seconds and a few minutes, depending on the hyperparameters and size of the dataset.
Acknowledgments The authors are grateful to Antonio Carlos Costa for bringing Koopman operators to our attention, and to Sarthak Chandra, Raymond Wang, Aran Nayebi, Ho Kyung Sung, Will Redman, Tosif Ahamed, and members of the Fiete lab for helpful discussions and advice.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv, 2020. URL https://www.biorxiv.org/content/early/2020/01/02/407007.
Rishidev Chaudhuri, Berk Gerçek, Biraj Pandey, Adrien Peyrache, and Ila Fiete. The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep. Nature Neuroscience, 22(9):1512-1520, 2019.
Alan D. Degenhart, William E. Bishop, Emily R. Oby, Elizabeth C. Tyler-Kabara, Steven M. Chase, Aaron P. Batista, and Byron M. Yu. Stabilization of a brain-computer interface via the alignment of low-dimensional spaces of neural activity. Nature Biomedical Engineering, pages 672-685, 2020.
Marino Pagan, Vincent D Tang, Mikio C. Aoi, Jonathan W. Pillow, Valerio Mante, David Sussillo, and Carlos D. Brody. A new theoretical framework jointly explains behavioral and neural variability across subjects performing flexible decision-making. bioRxiv, 2022. URL https://www.biorxiv.org/content/early/2022/11/28/2022.11.28.518207.
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 2008.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. In Advances in Neural Information Processing Systems, volume 34, pages 4738-4750, 2021.
Lyndon R Duong, Jingyang Zhou, Josue Nassar, Jules Berman, Jeroen Olieslagers, and Alex H Williams. Representational dissimilarity metric spaces for stochastic neural networks. arXiv preprint arXiv:2211.11665, 2022.
Shun-Ichi Amari and Michael A Arbib. Competition and Cooperation in Neural Nets. Systems Neuroscience, Academic Press, J. Metzler (ed.), pages 119-165, 1977.
J J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U. S. A., 81(10):3088-3092, 1984.
J J Hopfield and D W Tank. Computing with neural circuits: a model. Science, 233(4764):625-633, 1986.
H S Seung. How the brain keeps the eyes still. Proc. Natl. Acad. Sci. U. S. A., 93(23):13339-13344, 1996.
K Zhang. Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. J. Neurosci., 15:2112-2126, 1996.