Title: High-dimensional neuronal activity from low-dimensional latent dynamics: a solvable model
Venue: NeurIPS 2025
Accepted: yes
Keywords: recurrent neural networks

neuronal recordings

visual cortex

latent variable models

PCA

eigenvalue decay

mean-field limit
Abstract: Computation in recurrent networks of neurons has been hypothesized to occur at the level of low-dimensional latent dynamics, both in artificial systems and in the brain. This hypothesis seems at odds with evidence from large-scale neuronal recordings in mice showing that neuronal population activity is high-dimensional. To demonstrate that low-dimensional latent dynamics and high-dimensional activity can be two sides of the same coin, we present an analytically solvable recurrent neural network (RNN) model whose dynamics can be exactly reduced to a low-dimensional dynamical system, but generates an activity manifold that has a high linear embedding dimension. This raises the question: Do low-dimensional latents explain the high-dimensional activity observed in mouse visual cortex? Spectral theory tells us that the covariance eigenspectrum alone does not allow us to recover the dimensionality of the latents, which can be low or high, when neurons are nonlinear. To address this indeterminacy, we develop Neural Cross-Encoder (NCE), an interpretable, nonlinear latent variable modeling method for neuronal recordings, and find that high-dimensional neuronal responses to drifting gratings and spontaneous activity in visual cortex can be reduced to low-dimensional latents, while the responses to natural images cannot. We conclude that the high-dimensional activity measured in certain conditions, such as in the absence of a stimulus, is explained by low-dimensional latents that are nonlinearly processed by individual neurons.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces theoretical models and empirical results supporting the hypothesis that high-dimensional neural activity can arise from a low-dimensional latent space combined with nonlinear activations. The paper begins by first presenting a RNN model whose dynamics can be explicitly solved. This allows for direct analysis of the covariance eigenspectrum of the model’s post-activation activity, which the paper shows to be high-dimensional. The paper then relaxes assumptions on this solvable model and proposes a conjecture about the decay exponent of the covariance eigenspectrum in this relaxed model. Finally, the paper develops the Neural Cross-Encoder (NCE), a nonlinear autoencoder-inspired architecture, which they apply to neural recordings on mice during a visual task. They find that the NCE distinguishes between response types with low and high-dimensional pre-activations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper tackles an interesting and fundamental question about how we should interpret and test for latent dimensionality in neural population activity. Overall, the paper is easy to read and the technical steps are clearly explained. The solvable RNN model presented in Section 2 is relatively simple yet illustrative of their hypothesis that post-activation activity may still be generated from a low-d pre-activation latent space. Their overarching question and empirical results in Section 4 are well-motivated by real findings from visual cortical population activity. \n\nWeaknesses: My main concern with this paper is about the cohesiveness between the theoretical results presented in Section 2-3 and the empirical results presented in Section 4. Right now, the paper reads as three separate contributions across these sections, and it is not clear to me how the theoretical models allow us to reinterpret the notion of dimensionality in neural recordings. I would have liked to see some experiments about how well Conjecture 1 holds for fitted NCE models of different dimensionalities $d$ in Section 4. Otherwise, I am not sure about the role of Section 3 in this paper: the conclusion that high-d activity might arise from a nonlinear transformation of low-d states is already apparent from the solvable RNN model. \n\nI also have some concerns about the Neural Cross-Encoder methodology. In 4.2, the authors state that the NCE has a 3-layer encoder to allow for flexible estimation of latent variables; however, the choice of 3 layers is a bit arbitrary and might also lead to overfitting in some cases. I think the authors should address how one would decide the number of encoder layers / flexibility of the encoder. It is also not clear to me why the NCE is better at discarding variability that is not shared across neurons compared to a standard autoencoder fit on the entire set of neurons, since that should also discard such variability. I would have liked to see a more detailed explanation to motivate NCE as well as empirical comparisons with the standard autoencoder. \n\nFinally, some suggestions on clarity:\n- On line 135 the authors introduce the notation $\\mathbf{z}$, and it would help to draw the connection to $\\mathbf{\\kappa}$ from the previous section.\n- Line 200 typo in “variability”\n\nQUESTIONS:\n- In Fig 3I, it seems that the linear model is overfitting for large $d$ while the NCE does not. Why do you think this is the case, especially since the NCE is more flexible than the linear model?\n- How does the NCE deal with noise in the observed neural recordings and still extract seemingly meaningful latent variables (at least for the drifting gratings and spontaneous response types)?\n\nLIMITATIONS:\nThe authors address limitations about potential difficulties when fitting NCE, and in particular they point out that even if NCE is unable to find a well-fitting model, this does not mean such a model doesn't exist. I agree with this limitation and I think it could be better incorporated into the interpretation of results (section 4.3) as well. For example, on line 217 the authors write that the results on natural images suggest that natural images produce high-d representations in the space of pre-activations. However, according to the limitation, this is not necessarily the case -- it could also be due to getting stuck in local optima during training.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide theoretical proof using an analytically solvable recurrent neural network model that high-dimensional observations can be generated from low-dimensional latent variables. Further, they demonstrate that the generated high-dimensional observations can have a long-tailed covariance eigenspectrum, as is commonly seen in real neural data. Finally, the authors present a novel nonlinear latent variable model for calcium recordings, called Neural Cross-Encoder, and use it to demonstrate that high-dimensional neural activity in some regimes (i.e., spontaneous activity; response to natural images) can be described using low-dimensional latents while in other regimes they cannot (i.e., response to natural images).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The theoretical/analytical work presented in sections 2 and 3 are the strongest part of the manuscript. Proofs and derivations (not closely checked) seem complete and well-supported by the cited prior work.\n\n- Proofs provided in the appendix were helpful for understanding the main results / claims in the manuscript.\n\n**Weaknesses**\n- The relevance of the experimental analyses to the theoretical work in sections 2 and 3 is not particularly clear. Moreover, the results/claims in section 4 (i) are not particularly novel / surprising, (ii) can be improved with additional experimental details and a few more/modified analyses, and (iii) do not necessarily help prove / support the theoretical claims made previously – as currently presented. Please see questions below for specific recommendations and points of confusion where additional clarification is needed.\n\nQUESTIONS:\n1. It would be helpful if the authors could more clearly communicate that the analyses provided in section 4 are not related to the theoretical work in section 2 (i.e., solvable RNN model), but rather only for supporting / validating the claims made in section 3 (i.e., the relationship between pre-activation dimensionality, activation function, and post-activation eigenspectrum). Also, it would be informative/interesting if the authors visualized the covariance eigenspectrum of the true vs. model-predicted neural activity. As the authors pointed out, the goal here isn’t to estimate (or predict/recreate) the tail of the eigenspectrum, but it would be interesting to see how the decay changes as a function of dimensionality + activation function in real data.\n2. How are the pre-activation dimensionality results in section 4 novel / surprising / noteworthy? First, natural images are statistically complex stimuli and so it is not entirely surprising that their latent representations are of higher dimensionality than the drifting gratings. Second, how do we know that the results are not a byproduct of the model class used in NCE? It might be that for more complex stimuli, which require more complex neural encoding, the current encoder architecture choice is not sufficiently descriptive to learn the complex neural encoding [1]. Third, comparing against only a linear activation function seems insufficient because calcium recordings are known to be nonlinear. It would be interesting / more compelling if the authors explored using different activation functions and looked at the impact on explained variance and the covariance eigenspectrum. The analyses provided in section 3 only considered general rectified power activation functions so it would be more compelling / demonstrative if the authors compared the performance of NCE within this family of functions (not just against linear, i.e., reduced rank regression).\n3. It would be helpful if the authors could provide additional information in the Appendix about which specific regions were recorded and roughly how many neurons in each of the regions were determined “responsive”. Were neuron splits between source and target region agnostic (the authors stated that they collected data from primary and higher visual cortices)? It would be helpful if authors could also provide information on how many neurons and from which regions were assigned to source and target. This information may be important towards interpreting the results for each of the three task conditions, especially gratings vs natural images.\n4. What was the encoder architecture used in NCE? Please add this to the Appendix.\n5. Is the analysis presented in section 2 only limited to neurons on the unit circle/sphere manifold? Would the authors expect that this analysis could generalize to other manifolds?\n\n\n**Minor**\n- Could the authors provide an explanation as to why specifically for the spontaneous condition multiple timepoints were used as input for estimating the latent variable as a single timepoint? Was something similar attempted for the stimulus conditions as well? Why/why not? If yes, what were the results?\n- Was there a particular reason why the authors first pretrained the NCE model using knowledge distillation rather than directly training on the target neurons?\n- Equation (2), how did the authors choose the values for $J$ and $\\Delta$?\n- Could the authors clarify the following: “...per experiment, across 7 experiments (3 spontaneous activity, 3 natural images, 1 drifting gratings, across three mice)...” Does this mean 7 experiments were conducted for each of the 3 mice or across the 3 mice aggregated (i.e., only 1 mouse had the drifting gratings stimulus or all 3).\n- Fig. 3F would benefit from a few more y-axis labels for the points not close to the extrema.\n- In line 101, minor typo that it should be a pair of neurons $i$ and $j$. Also minor typo line 905 in Appendix: “standard deviation [of]”. Also, the equations following lines 980 and 981 in Appendix E have a reversed bracket in the Appendix, e.g., $\\theta \\sim \\mathcal{U}(\\left[0, 2\\pi\\right])$\n- In Appendix line 1044 (last line of page 27), it would be useful to remind readers of what $\\mathcal{F}$, $a$, and $b$ denote, especially as $ \\mathcal{F} $ and $ a $ were not defined.\n- Clarification for the equations after line 897 of Appendix A: is that an inner product or elementwise product in the integral: $ \\begin{pmatrix}\\cos(\\theta) \\sin(\\theta)   \\end{pmatrix} \\cdot  \\begin{pmatrix}\\cos(\\mathcal{v}) \\sin(\\mathcal{v}) \\end{pmatrix} $?\n\n\n**References**\n\n[1] D.L.K. Yamins, H. Hong, C.F. Cadieu, E.A. Solomon, D. Seibert, & J.J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex, Proc. Natl. Acad. Sci. U.S.A. 111 (23) 8619-8624, https://doi.org/10.1073/pnas.1403112111 (2014).\n\nLIMITATIONS:\nLimitations with respect to the experimental results in section 4 and the NCE model architecture are mostly sufficient. I recommend the following revision: “The fact that we were not able to accurately predict neuronal responses to natural images with a low-dimensional NCE model does not necessarily exclude the possibility for such a model to exist, and one could possibly find it with a larger training set **or a different model class/hyperparameter configuration**.” This is because the proposed NCE model may not have sufficient descriptive power/modeling capacity to robustly identify the shared variance in populations of neurons, especially in the context of more semantically complex stimuli wherein the encoding is less clear / more distributed [1,2].\n\nAlso, depending on the answer to my 5th question above, it may also be worth explicitly mentioning that the theoretical analysis provided in section 2 is limited to only specific manifold structures (e.g., unit ring/sphere).\n\n**References:**\n\n[1] Bolaños, F., Orlandi, J.G., Aoki, R. et al. Efficient coding of natural images in the mouse visual cortex. Nat Commun 15, 2466 (2024). https://doi.org/10.1038/s41467-024-45919-3\n\n[2] Michael Weliky, József Fiser, Ruskin H Hunt, David N Wagner. Coding of Natural Scenes in Primary Visual Cortex. Neuron 37(4), 703-718, 2003.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe contributions of the paper are twofold: First, it presents an analytically solvable recurrent neural network model that combines an efficiently low-dimensional dynamics of the neuron activations in the pre-activation space (activations inside the output function) and a very high-dimensional dynamics with a power-law eigenspectrum for the post activations (post the output threshold). The eigenspectrum of the post activations is derived for different types of low-dimensional solutions for the pre-activations, and RKHS theory a relationship beween the form of the activation function (rectified power function) and the decay exponent of the spectrum is derived and numerically verified. \n\nThe second contribution is an analysis the the estimated pre-activation dimensionality of mouse visual cortex responses using a neural cross-encoder architecture. This analysis shows low dimensionality for grating and high dimensionality for natural image responses.\n\nSTRENGTHS AND WEAKNESSES:\nThe mathematical analysis part of the paper is very strong and combines methods from probabilistics, neural field theory, and functional analysis / RKHS theory to derive and analyze the solutions of recurrent networks with very specific assumptions for the lateral coupling structure. \n\nIt is not entirely clear in how far the results break down for more general forms of lateral connectivity (different width of the interaction kernel, deviation form cos shape). Generality in presence of such changes would be required for a validity of the model for larger classes of real neurons.\n\nQUESTIONS:\nWhat exactly of the model in (13) is fitted to the neural activities?\n\nApparently sloppy notation in L. 117 ff: For even m the eigenvalues should be always 0. Should not for the uneven ones sin (m \\theta) be the eigenfunction?\n\nL. 957 (appendix).  should second z_s be z_t ?\n\nIs the factor (2/ \\pi) in the equation below L. 921 correct?\n\n(17) =>  C_{ij} = 4 * [term in (18)] - 4 * (1/4)\n\nLIMITATIONS:\nLimitations are thoroughly discussed,", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a theoretical framework to resolve the apparent contradiction between experimentally observed high-dimensional neuronal activity and commonly hypothesized low-dimensional neuronal latents. Following their theoretical model of neuronal computations as a linear-nonlinear system, the authors propose a latent variable identification method, namely Neural Cross-Encoder, to identify the low-dimensional latents underlying high-dimensional neuronal activity. They demonstrate that this method reveals low-dimensional latent variable space underlying spontaneous neuronal responses from the visual cortex and responses to drifting gratings but not responses to naturalistic stimuli. Taken together, this work advances our understanding of neuronal computations in response to stimuli as well as build on methods to identify latent variables underlying these computations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written, reasonably easy to follow and deals with a very pertinent question in the field. Resolving this apparent contradiction in theoretical/modelling approaches and experimental observations will have a significant impact on unifying and advancing the field further. \n2. The theoretical connections between the covariance matrix eigenspectrum and the eigenspectrum of the integral operator is very interesting and well-described. Section 3, overall, is a nice read and sets up the reader for the NCE method. \n3. The NCE method description is very well written and shows promising results, especially on the neuronal recordings. The semantic relevance of some of the identified latent variables is also a very interesting result, and potentially highlights the utility of the method. \n\nWeaknesses:\n1. I think a core contribution of the paper is the NCE method, and it falls short on highlighting the correctness and utility of the method. Firstly, the method has certain hyperparameters (e.g. size of the source and target neuronal populations) that seemed to have been chosen without proper justification. Moreover, the authors do not comment (in the main text) on the robustness of the method to the choice of these hyperparameter values. Finally, the linear model baseline seems like a weak comparison and underscores the utility of the method. Ideally, the authors would also show the method's abilities to uncover latent variables in some simulation setting before demonstrating their findings in recorded neuronal data.\n2. While the theoretical contributions are very interesting, they rely on the assumption of infinite neurons. Real systems are finite dimensional, and it would be interesting to see how these correspondences hold in (high-dimensional) finite dimensional systems. At the least, I think the authors should add a note or comment on these assumptions. \n3. Conjecture 1 is interesting (and a very cool result), but it warrants further justification. In the current version, the reader lacks context as to how the authors came up with this conjecture. The authors could probably add a few statements in this section to improve the readability of this section.\n\nQUESTIONS:\n1. While I enjoyed reading Section 2 (and the appendix), I am not quite sure what it adds to the paper. Section 3 and further sections do not require an RNN model assumption, and can be understood even without the results in Section 2. Could the authors clarify why they feel Section 2 is important in the context of the paper?\n2. The choice of the size of source and target neuronal populations is unclear. Do the results change if these choices are changed? Also, can you add an experiment to demonstrate the utility of NCE in simulated settings?\n3. Can you compare NCE to CEBRA or Rastermap or other latent variable methods? Given this is one of the core contributions of this work, it is imperative to show that NCE is indeed a performant latent variable method, at least comparable to existing methods.\n4. In the theory section, the latent variables were assumed to be drawn i.i.d. from a uniform distribution. Can you comment on this assumption, how reasonable is it and whether this can be relaxed? Does this affect NCE performance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}