Beyond single neurons: population response geometry in digital twins of mouse visual cortex

Authors: Dario Liscai, Emanuele Luconi, Alessandro Marin Vargas, Alessandro Sanzeni
Primary area: applications to neuroscience & cognitive science
Keywords: neuroscience, representation learning, population responses, cortical hierarchy, computational biology, visual perception

Abstract:
Hierarchical visual processing  is essential for cognitive functions like object recognition and spatial localization. Traditional studies of the neural basis of these computations have focused on single-neuron activity, but recent advances in large-scale neural recordings emphasize the growing need to understand computations at the population level. Digital twins-computational models trained on neural data-have successfully replicated single-neuron behavior, but their effectiveness in capturing the joint activity of neurons remains unclear. In this study, we investigate how well digital twins describe population responses in  mouse visual cortex. We show that these models fail to accurately represent the geometry of  population activity, particularly its differentiability and how this geometry evolves across the visual hierarchy. To address this, we explore how dataset, network architecture, loss function, and training method affect the ability of digital twins to recapitulate population properties. We demonstrate that improving model alignment with experiments requires training strategies that enhance robustness and generalization, reflecting principles observed in biological systems. These findings underscore the need to evaluate digital twins from multiple perspectives, identify key areas for refinement, and establish a foundation for using these models to explore neural computations at the population level.

Decision: accept (poster)
Reviewer scores: 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730613013887

thread_type:
official_review

<thread_object>
creator: reviewer_a1nt

<content>
summary:
This paper argues that some digital twins (i.e., ANN models that predict single neuron activities well) fail to capture population level properties (specifically having to do with the representational geometry) of neural responses that have been previously observed in the mouse visual cortex. The primary issue is the fact that the population representations in the digital twins have a geometry that is non-differentiable, which suggests that small changes in inputs can lead to very large changes in the network's representation. The paper then shows that dropout regularization can go some way towards mitigating these kinds of discrepancies between the ANN's responses and those of the actual mouse brain.

strengths:
Understanding how highly predictive ANN models of the brain can fail to capture certain properties of neural population responses that are functionally important is an important issue to address. This paper exposes some shortcomings of existing models, and gives a fairly straightforward way of mitigating some of those shortcomings, using dropout in the network. This is useful because it may lead to models that do a better job of accurately simulating brain responses.

weaknesses:
The paper could be clearer in a lot of places, especially for readers who are not familiar with the model being used, as well as the background literature which the paper heavily draws upon. 
- Although the model is described in the Appendix, it wasn't very clear (at least, not to me). There are a lot of different parts, and it was hard to see how they fit together. It might help to have a diagram illustrating the architecture and how everything fits together.

Although a lot of the background theory relating to the differentiability of neural representational geometry is presented in Stringer 2019, it would help readers if this paper were a bit more self contained, e.g. some intuitive explanation of why the cutoff for differentiability is 1 + 2/d would have been helpful to the uninitiated reader. 

Training the model only on correlations between neurons resulted in a loss of single neuron predictivity, presumably because single neuron predictivity is no longer part of the optimization objective. I would have thought a natural thing to try would be to train the model on a combined objective that optimizes for both objectives at once, and see if that changes the results in any way.

The claim that the digital twins fail to capture the hierarchical relationships between areas seemed like an overstatement, since what is really shown here is that one aspect of hierarchical relationships was not captured, not that it doesn't capture any relevant aspects of the visual hierarchy.

questions:
I was a bit confused at a high level as to how it is even possible for a digital twin to do a good job at predicting single neuron responses and yet still not capture population structure. Doesn't the population structure depend on the single neuron responses? How can you capture each individual neuron's activity with a high degree of accuracy and not get the population structure right?

Questions about the model:
- I was very surprised to read in the paper that the model does not have increasing receptive field sizes. Isn't that just a standard feature of a CNN?
- What is the intuition behind the formula for CC_max (where does this formula come from)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615161131

thread_type:
official_review

<thread_object>
creator: reviewer_sa6o

<content>
summary:
The study examines how well digital twins, computational models trained on neural data, capture both single-neuron and population-level responses in the mouse visual cortex. While these models predict single-neuron activity accurately, they fail to replicate the complex geometry and hierarchical organization of population responses seen in biological data. Factors like dataset composition and network architecture didn't resolve these issues. Introducing regularization techniques like dropout improved the models' ability to create more accurate population representations but reduced single-neuron prediction accuracy. The study highlights the need to refine digital twins to better understand neural computations at a population level.

strengths:
The paper tackles an important gap in computational neuroscience by focusing on the geometry of population-level neural responses in digital twins, not just single-neuron predictions. The study meticulously replicates key experimental findings and explores how factors like dataset variation, architectural changes, and training objectives impact model performance. Regularization techniques, especially dropout, are shown to improve the models' alignment with biological data. The paper is well-explained and accessible, providing valuable insights for both neuroscience and machine learning. By identifying the limits of current models, the research sets the stage for developing more accurate and biologically realistic models, emphasizing the need for principles like robustness and generalization in neural computation replication.

weaknesses:
Despite improvements with regularization, the models still show quantitative differences from experimental data, especially in discriminability measures and visual area hierarchy, warranting a deeper analysis to uncover the causes. There's a trade-off where enhancing population-level representation reduces single-neuron prediction accuracy, raising practical concerns. The paper's focus on a shared-core architecture limits insights; exploring more varied architectures could better reflect the visual cortex's hierarchical nature. Additionally, the study relies on computational simulations, and direct physiological validation of these models would further substantiate the findings.

questions:
Have the authors considered multi-objective optimization to balance single-neuron accuracy and population-level representation? Could adding biological features like spiking noise, synaptic dynamics, or neural variability improve the models' population properties? Have they explored area-specific modules or varied receptive field sizes to better capture hierarchical processing? Can they comment on the models' generalizability to different visual stimuli, given the issues with non-differentiable representations for gratings? Would using spiking neural network models help capture population response geometry and address trial-to-trial variability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731109836745

thread_type:
official_review

<thread_object>
creator: reviewer_ydfr

<content>
summary:
The authors demonstrate that a reasonably successful model of visual cortical neurons does not capture population metrics.  They show that measures of the population geometry and visual hierarchy fail on this model.  The eigenspectrum of the covariance matrix does not show scaling consistent with differentiability and the relative discriminability across areas does not match what is observed empirically.  The authors investigated various aspects of the model and of training to see what effect they had on these measurements.  These include selecting for reliability, choices of dataset, dataset augmentation, model architecture, and using a covariance-based loss function.  All of those failed to show different results for the population metrics.  The approach that did produce differentiable representations was adding dropout during training, which showed an exponent consistent with differentiability for sufficiently high dropout rates.  They claim that the models that show differentiability also show the appropriate hierarchical relationships for discriminability.

strengths:
This paper is reasonably clear and well-written.  The question is a very important one as population level characteristics of neural activity are undoubtedly important for neural computation and models need to be able to capture it.  It is extremely important to point out that our current state-of-the-art models are not meeting this criteria and likewise important to address what aspects of training will correct this.  The authors tested a broad set of training procedures.

weaknesses:
The authors did an excellent job of testing an array of procedures, but the results are still unfortunately limited by the number of models and data sets available for testing.  This is not terribly actionable, sadly, in a practical sense.

The authors wish to make a general claim about regularization, but only test dropout.

questions:
Does this result hold for task-trained models as well?

Can you test other forms of regularization?  Do they have the same effect, or is dropout special?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732399158665

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment by Authors

comment:
Thanks to all reviewers for their helpful comments. We have incorporated all suggestions, which have strengthened the paper substantially. We have submitted a revised version of the manuscript (with the revision shown in blue). Here are the main changes:

**Added experiments:**
- **Robustness of results.** During revision, we have significantly increased the number of runs used to probe object discriminability along the visual hierarchy (Fig.5). 
- **Increase transformation factor data augmentation.** We added more runs with a transformation factor from 0.8 to 1 (Fig. 4). 
- **Multi-objective optimization.** Included models jointly trained to predict single-neuron responses and correlation of the population (Supp. Fig. 8).
-  **Task-driven models.** We trained a linear readout to predict neural responses from the first three layer blocks of a frozen ResNet 50 trained on object recognition (Supp. Fig. 6). 
- **Models trained independently for each area.** We trained the model to predict neurons only from a specific area (Supp. Fig. 7).
- **Trade-off between single-neuron responses and differentiability.** We now show the relationship between the power-law and the single-neuron performance (Supp. Fig. 9)

**Modified text:**
- Clarified the experimental results for Stringer and Froudarakis in the introduction (Sec. 1)
- Improved the description of the model architecture and CC_norm metric (Appendix A1)
- Clarified hierarchy in discriminability (Sec. 4)
- Expanded the results with the new experiments (Sec. 5 and Sec. 6)
- Improved limitations and discussion: limitations of the architecture, trade-off between single neuron and population-level properties (Sec. 7)

Additionally, we reply to each reviewer individually to address their questions and concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732551627456

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message to reviewers

comment:
As the review process approaches its final stages, we would like to thank all reviewers once again for their thoughtful feedback, which has greatly improved our work. We have carefully addressed all comments and made the necessary revisions. We kindly invite you to review our responses and updated manuscript to ensure we have addressed your concerns. If any additional clarifications or reasonable changes are required, please let us know, and we will do our best to accommodate them. We hope our revisions meet your expectations and sincerely appreciate your consideration in recommending our paper for acceptance.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper examines the performance of “digital twin” models for neuroscience data, i.e. deep networks trained to predict the responses of neurons to different stimuli (here visual stimuli). The authors claim that digital twin models can capture single-neuron responses reasonably well while still failing to capture population-level properties of the representations (e.g. the geometry and differentiability of the representational manifolds). To support this claim, the authors examine the properties of a few different digital twin models tested on visual response datasets from mice, and show that they fail to capture population-level properties well. The authors then explore what design choices can be used to improve population-level matches, and find that choices that improve robustness and generalization (e.g. drop-out) can help in this regard.

The strengths of this paper are that it addresses an important topic for computational neuroscience, is clearly written, and uses appropriate metrics and data to support the claims. The weaknesses are that there were relatively few model architectures and datasets explored in the initial submission, and a lack of exploration of other potential solutions (e.g. multi-objective models).

Overall, the authors did a good job of addressing these weaknesses in the rebuttal, and the final average score was over 6. Based on this, a decision of accept (poster) was reached.