SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Validation using an artificial dataset We generated an artificial dataset with two distinct areas with 250 neurons each to showcase the effect of trial variability. In this dataset A1 (representing a sensory area) always responds to a stimulus while A2 (representing a motor area) responds to the stimulus in only  $80\%$  of the trials, see Figure ② This is a toy representation of the variability that is observed in the real data recorded in mice, so we construct the artificial data so that a recording resembles a hit trial ("hit-like") if the transient activity in A2 is higher than  $30\mathrm{Hz}$  (otherwise it's a "miss-like" trial). From the results of our simulations Figure ②B-C, we can observe that the models that use trial matching (either soft trial matching or hard trial matching) can re-generate faithfully this bimodal response distribution ("hit-like" and "miss-like") in A2. In this dataset we saw little difference between the solutions of soft and hard trial matching, if any, soft trial matching reached its optimal performance with fewer iterations (see Figure A.2C). As expected, when the model is only trained to minimize the neuron loss for trial-averaged statistics, it cannot generate stochastically this bimodal distribution and consistently generates instead a noisy average response.
Figure 3: Large-scale electrophysiology recordings dataset. A. Session stitching: Every neuron from the recordings across sessions is uniquely mapped to a neuron from our model. For example, an excitatory neuron from our model that belongs in the putative area tjM1 is mapped to an excitatory neuron recorded from tjM1. In the pink box are the areas from which we decode the jaw trace. B. Baseline firing rate histogram,  $200\mathrm{ms}$  before the whisker stimulus, from each neuron of our model and the recordings. C. Left: Pearson correlation of the PSTH, the violin plots represent the Pearson correlations across neurons. Right: trial-matched Pearson correlation of  $\mathcal{T}_{\mathrm{trial},k}^{\prime S}$ , the violin plots represent the distribution over 200 generated and recorded trial pairs.
Delayed whisker tactile detection dataset We then apply our modeling approach to the real large-scale electrophysiology recordings from [2]. After optimization, we verify quantitatively that our model generates activity that is similar to the recordings in terms of trial-averaged statistics.
First, we see that the 1,500 neurons in the network exhibit a realistic diversity of averaged firing rates: the distribution of neuron firing rates is log-normal and matches closely the distribution extracted from the data in Figure 3B. Second, the single-neuron PSTHs of our model are a close match to the PSTHs from the recordings. This can be quantified by the Pearson trial-averaged correlation between the generated and held-out test trials which we did not use for parameter fitting. We obtain an averaged Pearson correlation of  $0.30 \pm 0.01$  which is very close to the Pearson correlation obtained when comparing the training and testing sets  $0.31 \pm 0.01$ . Figure 3C shows how the trial-averaged correlation is distributed over neurons. As expected, this trial averaged metric is not affected if we do not use trial matching  $(0.30 \pm 0.01)$ .
To quantify how the models capture the trial-to-trial variability, we then quantify how the distributions of neural activity and jaw movement are consistent between data and model. So we need to define the trial-matched Pearson correlation to compute a Pearson correlation between the distribution of trial statistics  $\mathcal{T}_{\mathrm{trial},k}^{\prime S}$  which are unordered sets of trials. So we compute the optimal assignment  $\pi$  between trial pairs from the data and the recordings, and we report the averaged Pearson correlation overall trial pairs. Between the data and the model, we measure a trial-matched Pearson correlation of  $0.48 \pm 0.01$ , with a performance ceiling at  $0.52 \pm 0.01$  obtained by comparing the training and testing set directly (see Figure 3C for details). For reference, the model without trial matching has a lower trial-matched Pearson correlation  $0.28 \pm 0.003$ .
Successful recovery of trial type distribution While the neuronal activity is recorded, the behavioral response of the animal is also variable. When mice receive a stimulation they perform correctly
Figure 4: Emergent trial types. A. Trial type distribution from the recordings and the models. The whiskers show the  $95\%$  confidence interval of the trial type frequency. In this panel, trial types use the template matching method to avoid disadvantageing models without trial matching which have almost no variability in the jaw movement, see Figure A.3 B. Population and trial-averaged neuronal activity per area and per Hit and Miss trial type from the 400 simulated trials from the model against the averaged recordings from the testing set. C. Two-dimensional UMAP representation of the  $\mathcal{T}_{\mathrm{trial},k}$  of 400 simulated trials. The jaw movement is not used in this representation. D. For the model, we separated the active hit and quiet hit trials based on their location in the UMAP. For the data, we separated the active hit and quiet hit trials based on the jaw movement as in [2].
with a  $66\%$  hit rate, while in the absence of a stimulus, mice still falsely lick with a  $20\%$  false alarm rate. Even in correct trials, the neural activity reflects variability which is correlated to uninstructed jaw and tongue movements [2].
We evaluate the distribution of trial types (hit, miss, correct rejection, and false alarm) from our fitted network model. Indeed, the  $95\%$  confidence intervals of the estimated trial type frequencies are always overlapping between the model and the data (see Figure 4A). In this Figure, we classify the trial type with a nearest-neighbor-like classifier using only the neural activity (see Figure A.3A). In contrast, a model without the trial matching would fail completely because it always produces averaged trajectories instead of capturing the multi-modal variability of the data as seen in Figure 4A. With trial matching it is even possible to classify trial types using jaw movement. To define equivalent trial types in the model, we rely on the presence or absence of the stimulation and a classifier to identify a lick action given the jaw movements. This classifier is a multi-layer perceptron trained to predict a lick action on the water dispenser given the recorded jaw movements (like in the data, the model can "move" the jaw without inducing a lick action). After optimization with trial matching, since the jaw movement  $y_{k}^{t}$  is contained in the fitted statistics  $\mathcal{T}_{\mathrm{trial},k}^{S}$ , the distribution of jaw movement and the trial types are similar in the fitted model and the trial type distribution remains consistent. In Figure 4B we show population-averaged activity traces where the jaw is used to determine the trial type.
Unsupervised discovery of modes of variability So far we have analyzed whether the variability among the main four trial types was expressed in the model, but the existence of these four trial types is not enforced explicitly in the loss function. Rather, the trial matching loss function aims to match the overall statistics of the distributions and it has discovered these four main modes of trial-
variability without explicit supervision. A consequence is that our model has possibly generated other modes of variability which are needed for the model to explain the full distribution of recorded data.
To summarize the full distribution of trials, we project the neural activity in a 2D space in Figure 4C. This representation recapitulates the modes of co-variability in all six areas and is not directly possible to be generated from the dataset because trials from distinct sessions have non-overlapping recorded units. This representation becomes possible with a generative model like ours that can capture the distribution of trial-to-trial variability from incomplete recordings. Technically, we generate 400 trials and we apply UMAP to the concatenation:  $(\mathcal{T}_{wS1,k},\dots \mathcal{T}_{ALM,k})$  where each vector  $\mathcal{T}_A,k$  includes now all neurons simulated in area  $A$  so the generated vectors contain data about all sessions and areas. To confirm that the generated representation of the distribution is consistent with the data we display template vectors for each trial condition  $c$  that are calculated directly on the recorded data trials. These templates are drawn with stars in Figure 4C and are computed as follows: the feature  $\mathcal{T}_{A,c}^{\mathcal{D}}$  of this template vector is computed by averaging the population activity of area  $A$  in all recorded trials from all sessions (see Appendix C for details), the averaged vectors are then concatenated and projected in the 2D UMAP space.
The consistency between the condition-averaged templates and the full trial distribution is confirmed visually in Figure 4C. The emerging distribution in this visualization is grouped in clusters. We observe that the template vectors representing the correct rejection, miss, and false alarm trials are located at the center of the corresponding cluster of generated trials. It also appears that the generated trial distribution provides richer information about the trial-to-trial variability that is hard to study using condition-averaged templates. For instance, we were curious why the generated hit trials are split into two clusters (see the two boxed clusters in Figure 4C). Looking more closely at the difference between those two clusters of generated trials, it turned out that their difference can be explained by a simple feature:  $85\%$  of the generated hit trials on the left-hand cluster of panel 4C have intense jaw movements during the delay period  $\left(\max_{t}\left|y^{t} - y^{t - 1}\right| > 4\delta\right.$  where  $\delta$  is the standard deviation of  $|y^{t} - y^{t - 1}|$  in the  $200~\mathrm{ms}$  before whisker stimulation). This criterion happens to be similar to the one used by experimentalists [2] to separate the hit trials in the recorded data, so we also refer to them as the "active hit" and "quiet hit" trials and show the population activity in Figure 4D. This interesting correspondence highlights the potential for our method to study the structure of the trial-to-trial variability without a priori assumptions of the trial condition partitions. We conclude that our modeling approach can be used for a hypothesis-free identification of modes of trial-to-trial variability, even when they reflect task-irrelevant behavior.
We introduced a generative modeling approach where a data-constrained RSNN is fitted to multi-session electrophysiology data. The two major innovations of this paper are (1) the technical progress towards multi-session RSNN fitted with automatic differentiation, and (2) a trial matching loss function to match the trial-to-trial variability in recorded and generated data.
Interpretable mechanistic model of activity and behavior Our model has a radically different objective in comparison with other deep-learning models: our RSNN is aimed to be biophysically interpretable. In Figure A.5 in the appendix, we illustrate a strong difference between our interpretable RSNN model and the more generic LFADS method which generates neural network activity with deep learning models. The most striking difference is that LFADS [22] relies on Gated recurrent units (GRU) which are abstract recurrent models for which there is no explicit mapping with real neurons. In our case, by construction there is a one-to-one mapping between an RSNN spiking unit and a recorded neuron. We further illustrated a consequence of our model in Figure A.5, we designed an extension of the artificial dataset where two putative areas are activated with a substantial delay of 20, or  $200\mathrm{ms}$ . In real recordings, this would be only possible if unrecorded areas were maintaining some network activity during this long delay, but there is no plausible network hypothesis with two areas that could solve the task. When the delay between the areas is plausibly short  $(20\mathrm{ms})$ , the generated data from our RSNN model and LFADS are similarly accurate. For longer delays, the constraint in the RSNN model cannot allow for an implausible network solution by design so our optimization collapses and does not provide any speculative circuit hypothesis for this artificial dataset unless we allow our network to have implausible long synaptic transmission delays,
see Figure A.5B-C. In contrast, LFADS can generate solutions independently of the delay duration, partly thanks to its bidirectional GRU units with efficient internal memory gating which can handle very large delays. In the long term, we hope that the interpretability of our method will be able to capture biological mechanisms (e.g. predicting network structure, causal interactions between areas, and anatomical connectivity), but in this paper, we have focused on numerical and methodological questions which are getting us one step closer to this long-term objective.
Mechanisms of latent dynamics A long-standing debate is whether the brain computes with low-dimensional latent representations and how that is implemented in a neural circuit. Another reason for the LFADS to fit implausibly long delays, see Figure A.5, is inherent to the deep auto-encoder setting. By construction, LFADS generates trial-to-trial variability from low-dimensional latent representations, so the variability is sourced by the latent variable which contains all the trial-specific information. This is in stark contrast with our approach, where we see the emergence of structured manifolds in the trial-to-trial variability of the RSNN from the noise source of the network dynamics. In the UMAP representation of Figure 4C, we visualize a low-dimensional manifold of trial distribution although we did not enforce explicitly the presence of low-dimensional latent dynamics. Structure in the trial-to-trial variability emerges because the RSNN is capable of transforming the unstructured noise sources (stochastic spikes and Gaussian input current) into a low-dimensional trial-to-trial variability – a typical variational auto-encoder setting would not achieve this.
Note that it is also possible to add a random low-dimensional latent as a source of low-dimensional variability in our RSNN as it is done with LFADS. In the Figure A.4, we reproduce our results on the multi-session dataset from [2] while assuming that all voltages  $v_{i,k}^{t}$  have a trial-specific excitability offset  $\xi_{i,k}$  using a 5-dimensional gaussian noise  $\psi_{k}$  and a one-hidden-layer perceptron  $F_{\theta}$  such that  $\xi_{i,k} = F_{\theta,i}(\psi_k)$ . The latent variable, therefore implements a neuron-specific excitability offset coming from an unknown source. Unsurprisingly this latent noise model accelerates drastically the optimization, probably because  $\xi_{i,k}$  is an efficient noise source for minimizing  $\mathcal{L}_{\mathrm{trial}}$ . More remarkably, the final fitting performance metrics are the same with and without the low-dimensional drive. It suggests that the extra assumption of a low-dimensional input from an unknown source is convenient but not necessary to generate realistic variability. Our first hypothesis that the recorded circuit alone produces the distribution of possible activity and behavior is sufficient. We speculate that providing this low-dimensional input to the model is sometimes counterproductive if the end goal is to identify the mechanism by which the circuit produces the low-dimensional dynamics.
Alternative loss functions to capture variability In our optimization, the trial-matching loss function constrains the network to produce realistic trial-to-trial variability. The main alternative methods to constrain the trial-to-trial variability would be likelihood-based approaches [6, 15] or spike-GANs [27, 33]. These methods are appealing as they do not depend on the choice of trial statistics  $\mathcal{T}_{\mathrm{trial},k}^{S}$ . Since these methods were never applied with a multi-session data-constrained RSNN we explored how to extend them to our setting and compare the results. We tested these alternatives on the Artificial dataset in Figure A.2. The likelihood of the recorded spike trains [6, 15] cannot be defined with multiple sessions because we cannot clamp neurons that are not recorded (see [8] for details). The closest implementation that we could consider was to let the network simulate the data "freely" which requires, therefore, an optimal assignment between recorded and generated data, so it is a form of trial-matched likelihood). With this loss function, we could not retrieve the bi-model hit versus miss trial type distribution unless it is optimized jointly with  $\mathcal{L}_{\mathrm{trial}}$ .
We also tested the implementation of a spike-GAN discriminator. In GANs the min-max optimization is notoriously hard to tune, and we were unable to train our generator with a generic spike-GAN discriminator from scratch (probably because the biological constraints of our generator affect the robustness of the optimization). In our hands, it only worked when the GAN discriminator was fed directly with the trial statistics  $\mathcal{T}_{\mathrm{trial},k}^{S}$  and the network was jointly fitted to the trial-averaged loss  $\mathcal{L}_{\mathrm{neuron}}$ . It shows that a GAN objective and the trial matching loss function hold a similar role. We conclude that both of these clamping-free methods are promising to fit data-constrained RSNNs. What differs between them, however, is that trial matching replaces the discriminator with the optimal assignment  $\pi$  and the statistics  $\mathcal{T}$  which are parameter-free, making them easy to use and numerically robust. It is conceivable for future work the best results are obtained by combining trial matching with other GAN-like generative methods.
This research was supported by the Swiss National Science Foundation (no. 31003A_182010, TMAG-3_209271, 200020_207426), and Sinergia Project CRSII5_198612. Many thanks to Lénaïc Chizat, Vivien Seguy, James Isbister, Shuqi Wang and Vahid Esmaeili for their helpful discussions.
[22] Chethan Pandarinath et al. “Inferring single-trial neural population dynamics using sequential auto-encoders”. In: Nature Methods 15.10 (2018), pp. 805–815.
[23] Ding Zhou and Xue-Xin Wei. "Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE". In: Advances in Neural Information Processing Systems (2020), pp. 7234-7247.
[24] Byron M Yu et al. "Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity". In: Advances in Neural Information Processing Systems 21 (2008).
[25] Shuqi Wang et al. "Mesoscopic modeling of hidden spiking neurons". In: Advances in Neural Information Processing Systems 35 (2022), pp. 23566-23579.
[26] Brian DePasquale et al. "The centrality of population-level factors to network computation is demonstrated by a versatile approach for training spiking networks". In: Neuron (2023).
[27] Manuel Molano-Mazon et al. "Synthesizing realistic neural population activity patterns using Generative Adversarial Networks". In: International Conference on Learning Representations (2018).
[28] Adrian Valente, Jonathan W Pillow, and Srdjan Ostojic. "Extracting computational mechanisms from neural data using low-rank RNNs". In: Advances in Neural Information Processing Systems 35 (2022), pp. 24072-24086.
[29] Valentin Schmutz, Johanni Brea, and Wulfram Gerstner. "Convergence of redundancy-free spiking neural networks to rate networks". In: arXiv preprint arXiv:2303.05174 (2023).
[30] Kanaka Rajan, Christopher D Harvey, and David W Tank. "Recurrent network models of sequence generation and memory". In: Neuron 90.1 (2016), pp. 128-142.
[31] Aaron S Andalman et al. "Neuronal dynamics regulating brain and behavioral state transitions". In: Cell 177.4 (2019), pp. 970-985.
[32] Konstantin-Klemens Lurz et al. "Generalization in data-driven models of primary visual cortex". In: International Conference on Learning Representations (2021).
[33] Poornima Ramesh, Mohamad Atayi, and Jakob H Macke. "Adversarial training of neural encoding models on population spike trains". In: NeurIPS Workshop on Real Neurons & Hidden Units: Future directions at the intersection of neuroscience and artificial intelligence. 2019.
[34] Vahid Esmaeili et al. "Learning-related congruent and incongruent changes of excitation and inhibition in distinct cortical areas". In: PLoS Biology 20.5 (2022), e3001667.
[35] Harold W Kuhn. "The Hungarian method for the assignment problem". In: Naval Research Logistics Quarterly 2.1-2 (1955), pp. 83-97.
[36] Alexandre Défossez et al. “High fidelity neural audio compression”. In: arXiv preprint arXiv:2210.13438 (2022).