Title: Reproducibility of predictive networks for mouse visual cortex
Venue: NeurIPS 2024
Accepted: yes
Keywords: reproducibility

predictive models for visual cortex

neuroscience
Abstract: Deep predictive models of neuronal activity have recently enabled several new discoveries about the selectivity and invariance of neurons in the visual cortex.
These models learn a shared set of nonlinear basis functions, which are linearly combined via a learned weight vector to represent a neuron's function.
Such weight vectors, which can be thought as embeddings of neuronal function, have been proposed to define functional cell types via unsupervised clustering.
However, as deep models are usually highly overparameterized, the learning problem is unlikely to have a unique solution, which raises the question if such embeddings can be used in a meaningful way for downstream analysis.
In this paper, we investigate how stable neuronal embeddings are with respect to changes in model architecture and initialization. 
We find that $L_1$ regularization to be an important ingredient for structured embeddings and develop an adaptive regularization that adjusts the strength of regularization per neuron.  
This regularization improves both predictive performance and how consistently neuronal embeddings cluster across model fits  compared to uniform regularization.
To overcome overparametrization, we propose an iterative feature pruning strategy which reduces the dimensionality of performance-optimized models by half without loss of performance and improves the consistency of neuronal embeddings with respect to clustering neurons.
Our results suggest that to achieve an objective taxonomy of cell types or a compact representation of the functional landscape, we need novel architectures or learning techniques that improve identifiability. 
The code is available https://github.com/pollytur/readout_reproducibility.

Reproducibility of predictive networks for mouse visual cortex

Abstract

Deep predictive models of neuronal activity have recently enabled several new discoveries about the selectivity and invariance of neurons in the visual cortex. These models learn a shared set of nonlinear basis functions, which are linearly combined via a learned weight vector to represent a neuron's function. Such weight vectors, which can be thought as embeddings of neuronal function, have been proposed to define functional cell types via unsupervised clustering. However, as deep models are usually highly overparameterized, the learning problem is unlikely to have a unique solution, which raises the question if such embeddings can be used in a meaningful way for downstream analysis. In this paper, we investigate how stable neuronal embeddings are with respect to changes in model architecture and initialization. We find that L 1 regularization to be an important ingredient for structured embeddings and develop an adaptive regularization that adjusts the strength of regularization per neuron. This regularization improves both predictive performance and how consistently neuronal embeddings cluster across model fits compared to uniform regularization. To overcome overparametrization, we propose an iterative feature pruning strategy which reduces the dimensionality of performance-optimized models by half without loss of performance and improves the consistency of neuronal embeddings with respect to clustering neurons. Our results suggest that to achieve an objective taxonomy of cell types or a compact representation of the functional landscape, we need novel architectures or learning techniques that improve identifiability. The code is available https://github. com/pollytur/readout_reproducibility .

1 Introduction

One central idea in neuroscience is that neurons cluster into distinct cell types defined by anatomical, genetic, electrophysiological or functional properties of single cells [23, 49, 54, 59, 69, 70]. Defining a unified taxonomy of cell types across these different descriptive properties is an active area of research. The functional properties of neurons, i. e. which computations they implement on the sensory input, constitute an important dimension along which cell types should manifest. However, each neuron's function is a complex object, mapping from a high-dimensional sensory input to the output of the neuron. Classical work in neuroscience has described neuronal function by a few parameters such as tuning to orientation, spatial frequency, or phase invariance [1, 2, 5, 8, 9, 13, 25, 40]. However, when considering neurons from higher visual areas, deeper into the brain's processing network, neuronal functions become more complex and not easily described by few

manually picked parameters. In addition, recent work in the mouse visual cortex demonstrated that even in early areas, neuronal functional properties are not necessarily well-described by simple properties such as orientation [47, 71]. One possible avenue to address these challenges is to use functional representations learned by data-driven deep networks [36, 37, 39, 44] trained to predict neuronal responses from sensory inputs. The central design element of these networks is a split into a core - a common feature representation shared across neurons, and a neuron specific readout -typically a linear layer with a final nonlinearity (see Fig. 1). These networks are state-of-the-art in predicting neuronal responses on arbitrary visual input, and several studies have used these networks as 'digital twins' to identify novel functional properties and experimentally verify them in vivo [37, 47, 57, 58, 67, 68]. Because readout weights determine a compact representation of a neuron's inputoutput mapping, they can serve as an embedding of a neuron's function. Such embeddings could be used to describe the functional landscape of neurons or obtain cell types via unsupervised clustering [59]. However, because the feature representations provided by the core are likely overcomplete, there will be many readout vectors that represent approximately the same neuronal function. Thus, it is not clear how identifiable functional cell types are based on clustering these embeddings. Moreover, since early data-driven networks [22, 28] there have been several advances in readout [48, 51, 65] and network architecture [33, 36, 56, 67] to decrease the number of per-neuron parameters, incorporate additional signals such as behavior and eye movements, or add specific inductive biases, such as rotation equivariance. Currently, there is no study systematically investigating the impact of these architectural choices on the robustness and the consistency of embedding-based clustering.

Here we address this question by quantifying how consistent the models are across several fits with different seeds. We measure consistency by (1) adjusted rand index (ARI) of clustering partitions across models, (2) correlations of predicted responses across models, and (3) consistency of tuning indexes describing known nonlinear functional properties of visual neurons. Our contributions are:

We show that L 1 regularization, used in early models [28, 59], is instrumental in obtaining a structured clustering when using embeddings of newer readout mechanisms [51].

We introduce a new adaptive regularization scheme, which improves consistency of learned neuron embeddings while maintaining close to the state-of-the-art predictive performance.

We address the identifiability problem by proposing an iterative feature pruning strategy which reduces the dimensionality of performance-optimized models by half without loss of performance and improves the consistency of neuronal embeddings.

We show that even though our innovations improve the consistency of clustering neurons, older readout mechanisms [28] reproduce neuronal tuning properties more consistently across models.

Our results suggest that to achieve an objective taxonomy of cell types or a compact representation of the functional landscape, we need novel architectures or learning techniques that improve identifiability while preserving neuronal functional properties.

2 Background and related work

Predictive models for visual cortex. Starting with the work of Antolík et al. [22], a number of deep predictive models for neuronal population responses to natural images have been proposed. These models capture the stimulus-response function of a population of neurons by learning a nonlinear feature space shared by all neurons (Fig. 1A; the 'core'), typically implemented by a convolutional neural network [26, 28, 51], sometimes including recurrence [36, 55]. The core can be pretrained and shared across datasets [67]; it forms a set of basis functions spanning the space of all neuron's input-output function. The second part of these model architectures is the 'readout': it linearly combines the core's features using a set of neuron-specific weights.

As the dimensionality of the core's features is quite high (height × width × feature channels), several different ways have been proposed to constrain and regularize the readout, accounting to the fact that neurons do not see the whole stimuli but rather focus on a small receptive field (RF). Initially, Klindt et al. [28] proposed a factorization across space (RF location) and features (function) (Fig. 1A; 'Factorized readout'). This approach required relatively strong L 1 regularization to constrain the RF. More recently, Lurz et al. [51] further simplified the problem by learning only an ( x, y ) coordinate, representing the mean of a Gaussian for the receptive field location, as well as a vector of feature

x'

y'

Factorised readout

neuron 1

neuron 2

neuron n

…

x'

y'

∑

)

neuron 1

neuron 2

neuron n

…

xn,yn ~ N(µ,

∑

)

f

f

Core

x'

f

Feature map

y'

One core layer

Input

[x,y,c]

f 2d filters

…

n rotations per filter

Batch

norm

[x',y',fn]

A

Train model

C

Align readouts

Cluster readouts

Compute ARI

B

Row

readout

Aligned

readout

r2

r1

Orbit

under

cyclic

rotations

weights (Fig. 1A; 'Gaussian readout'). This removed the necessity for strong L 1 regularization. Current state-of-the-art models use Gaussian readout with weak or no regularization [60, 64, 67].

Functional properties. Previous work verified these model in vivo [37, 47, 58, 67, 68]. There is a long history of work investigating neuronal functional properties experimentally, beginning with Hubel and Wiesel (1962) [1] who showed bar stimuli to cats and found that many neurons in V1 are orientation selective. Since then, a number of studies showed that many V1 neurons are - in a linear model approximation - optimally driven by a Gabor stimulus of specific orientation, size, spatial frequency, and phase [6, 20, 32, 40]. Later, non-linear phenomena were investigated, such as surround suppression and cross-orientation inhibition. Neurons that are surround suppressed can be driven by stimulating their receptive field, while an additional stimulation of their surround reduces neural activity. Interestingly, only stimulating the surround would not elicit a response. Cross-orientation inhibition reduces neural activity of some neurons if a neuron's optimal Gabor stimulus is linearly combined with a 90 degrees rotated version of that Gabor stimulus, forming a plaid [3, 5, 7, 8, 10, 14, 17]. Recently, such experiments were performed in silico with neural predictive models, allowing for large-scale analysis of neuron's tuning properties without experimental limitations [53, 59].

Network pruning. While modern deep neural networks are typically highly overparameterized, as this simplifies optimization [38, 50, 52], overparametrization poses a problem for reproducibility of neural embeddings: model training might end in various local optima that leads to different neuronal representations even if model outputs and predictive performance are similar. Prior work has shown that learned deep neural networks can be pruned substantially: Frankle and Carbin [34] showed

that up to 80% of weights can be removed without sacrificing model performance. Further work investigated various pruning strategies. For instance Li et al. [29] pruned CNN channels based on their L 1 norm, Luo and Wu [30] used entropy as the pruning criteria, and various other criteria were studied [24, 27, 31]. Interestingly, a later study showed that model performance is relatively robust against the specific pruning strategy used [35].

3 Data and model architecture

Data. We used the data from the NeurIPS 2022 Sensorium Competition for our study [60]. This dataset contains responses of primary visual cortex (V1) neurons to gray-scale natural images of seven mice, recorded using two-photon calcium imaging. In addition the mice's running speed, pupil center positions, pupil dilation and its time-derivative were recorded.

Model architecture. Our model (Fig. 1) builds upon the baseline model from the NeurIPS 2022 Sensorium competition [60]. Similar as the baseline model, our model includes the shifter subnetwork [36] to account for eye movements and correct the resulting receptive field displacement. We also concatenate the remaining behavioral variables to the stimuli as the input to the model [56, 60]. For the model core, we use a rotation-equivariant convolutional neural network, roughly following earlier work [33, 46, 59]. Compared to this prior line of work, our model includes an additional convolutional layer (as the dataset is larger). It consists of four layers with filter sizes 13, 5, 5, and 5 pixels. In all layers, we use sixteen channels and eight rotations for each of the channels, which results in 128-dimensional neuronal embeddings. The details of model and training config are provided in Appendix A.1. For the readout, we use both the factorized readout and the Gaussian readout (Fig. 1A). The factorized readout has been used in prior work on functional cell types [46, 59]. As it does not support the shifter network, this version of the model cannot account for eye movements.

4 Methods

Training. Following prior work [28, 33, 36, 47, 53, 58, 60, 64, 66], we trained the model minimizing the Poisson loss L p = N -1 ∑ N i =1 (ˆ r ( i ) -r ( i ) log ˆ r ( i ) ) between predicted ˆ r ( i ) and observed r ( i ) spike counts for all i = 1 , ..., N neurons, as it is oftentimes assumed that neuron's firing rates follow a Poisson process [15]. For factorized readouts we add a L 1 regularization of the readout mask and embeddings to the loss, i.e. L 1 = ∑ N i =0 γ ( ∑ mask i,j | m ij | + ∑ embedding k | w k | ) for spatial mask m , embedding w , and regularization coefficient γ , in line with previous work [28, 33, 39, 46, 53, 59]. As the Gaussian readout selects one spatial position, the L 1 penalty term reduces to L 1 = ∑ N i =0 γ ∑ embedding k | w k | . Thus, for both models the total loss becomes Loss = L p + L 1 . For the factorized readout γ is typically chosen to maximize performance (Fig. 6), while the choice of γ for Gaussian readouts is the subject of this paper.

Evaluation of model performance. Following earlier work [21, 33, 36, 47, 53, 60, 64, 66], we report model's predictive performance using Pearson correlation between each measured and predicted neural activity pair across images on the test set.

Evaluation of embedding consistency. Assessing how 'consistent' neuronal embeddings are across model architectures and initializations is a non-trivial question. They will naturally differ across runs in absolute terms, because core and readout are learned jointly. In the context of identifying cell types, we are interested in the relative organization of the embedding space: Will the same sets of neurons consistently cluster in embedding space? To address this question, we use clustering and evaluate how frequently pairs of neurons end up in the same group by computing the adjusted rand index (ARI, [4]). The ARI quantifies the similarity of two cluster assignments X and Y :

Here, a i is the number of data points in cluster i of partition X , b j is the number of data points in cluster j of partition Y , n ij is the number of data points in clusters i and j of partition X and Y ,

respectively, and n is the total number of data points. The ARI is invariant under permutations of the cluster labels. It is one if and only if the two partitions match exactly. It is zero when the agreement between the two partitions is as good as expected by chance and, hence, can be negative [61].

For clustering, we follow the rotation-invariant clustering procedure (Fig. 1B) introduced by Ustyuzhaninov et al. [46] and then cluster the aligned embeddings with k means. As we do not know the true underlying number of clusters, we explore a range of k = 5 up to k = 100 clusters, which covers the typical amount of cell types suggested in various works and modalities [41, 42, 45, 59].

To visualize the neuronal embeddings, we use t-SNE [16] using the recommendations of Kobak and Berens [43]. We use a perplexity given by 1% of the size of the dataset and set early exaggeration to 15. To avoid visual clutter, we plot a randomly sampled subset of 2,000 neurons from each of the seven animals in the dataset. We use the same neurons across all ARI experiments and t-SNE visualizations. The results do not depend on the specific cluster consistency metric used (see Appendix A.9). We assessed the credibility of t-SNE visualizations in Appendix A.10.

Neuronal tuning properties. Unlike the embeddings, which were not yet biologically validated, the predicted activity from our models has been consistently verified in vivo across several studies [47, 58, 59, 68]. By focusing on predicted activity, we leverage the stability and biological relevance of these outputs and relate our findings to real-world scenarios. Specifically, we investigated the predicted neural activity to parametric, artificial stimuli that are known to yield interesting phenomena and were used extensively in biological experiments (see Section 2).

To quantify neuron's tuning properties, for phase or orientation tuning , we use the parameters of the optimal Gabor and change only the phase or orientation to create a set of new Gabors. We feed this set of stimuli through the model and fit sinusoidal functions to the stimulated response curve, namely f ( α ) = A sin( α + ϕ ) + B for the phase sweep. For the orientation sweep, rotating the Gabor-based stimuli by 180-degrees will lead the same stimulus, accounted for by the additional factor of two when fitting f ( α ) = A sin(2 α + ϕ ) + B to the stimulated responses. We define the corresponding tuning indices as the ratio of the fitted amplitude A to the mean B of the curve.

For cross-orientation inhibition we added an optimal Gabor and its orthogonal copy and stimulated the tuning curve by varying the contrast of the orthogonal Gabor. Analogically, for surround suppression we varied the size and the contrast of a grating based on the optimal Gabor's parameters. To quantify both indices, we followed [11, 53] and saved the highest model prediction ˆ r max . Next, we computed the according tuning indices as 1 -ˆ r supp / ˆ r max , where ˆ r supp corresponds to the highest suppression/inhibition observed. See Appendix Section A.2 for parameter details.

We compute tuning index consistency across three trained models only differing in their parameter initialisation before fitting. Specifically, we compare their similarity by the normalized mean absolute error NMAE ( x, y ) = ∑ neurons i (( | x i -y i | ) / ( | x i -¯ x | )) where x and y are tuning indices for different model fits and ¯ x denotes the mean across neurons.

Optimal Gabor search. All tuning properties investigated in our paper are based on optimal Gabor parameters. Previous work [33, 53, 59] brute-force searched the optimal stimulus. Since their models relied on the factorized readout which could integrate over multiple spatial input positions, they had to show Gabor stimuli of various properties at all spatial input positions. Thus, they needed to iterate through millions of artificially generated Gabor stimuli for each neuron. Here, we introduce a technical improvement upon their work significantly speeding up the optimal Gabor search for Gaussian readouts: as this readout type selects only one spatial position per neuron it is sufficient to limit the presented stimulus canvas size to the model's receptive field size. This significantly decreased the canvas size from 36 × 64 = 2 , 304 to 25 × 25 = 625 input pixels, leading to an approximately 3.7-fold speedup. Additionally, our improved approach reduces the number of floating point operations in the model's forward pass, further benefiting the computation time. Another important detail is that for the optimal Gabor search and tuning curves experiments we completely ignored the learnt neuronal position as there was only one position to choose. For all in silico experiments, we removed models' shifter pupil position prediction network [36], as pupil position only impacted the spacial position selection, which is now limited to a single pixel. All other behavior variables we set to their training set median values, following [56, 67].

Validation

Score

0.4

=50

=100

Yadaptive = 10

B

D

E

F

G

H

25

0.45

y=0

weightsdensity

20-

0.40

0.35

y=100

y=50

factorised

15

0.30

Yadaptive = 10

VS

0.25

10-

0.20

5

X

0.15

0.10

0

0.3

-0.2-0.1

0.0

0.20.3

0.05

weight value

20

40

60

80

100

Number of clusters

5 Results

From factorized to Gaussian readouts. We start with our observation that the structure of the neuronal embeddings depends quite strongly on the type of readout mechanism employed by the model (Fig. 2). Earlier studies employing the factorized readout reported clusters in the neuronal embeddings space [46, 59] and suggested that those could represent functional cell types (Fig. 2B). However, for more recent models with a Gaussian readout [51, 60] this structure is much less clear (Fig. 2C), although these models exhibit substantially better predictive performance (Fig. 2A). This vizualization difference was also evident in quantitative terms: Clustering the neuronal embeddings led to significantly more consistent clusters for the factorized readout than for the Gaussian readout (Fig. 2I; red vs. blue), independent of the number of clusters used.

We hypothesized that this difference might be caused by the L 1 regularization in earlier models, which could reveal the structure in neuronal function better and lead to more consistent embeddings. Indeed, the weight distribution for the factorized model was much sparser compared to the Gaussian readout (Fig. 2G). Increasing the L 1 regularization indeed resulted in more structured embeddings at least qualitatively (Fig. 2D,E) and in terms of the distribution of embeddings (Fig. 2G; pink and green). It also came with a drop in predictive performance comparable to that of the factorized readout (Fig. 2A). However, somewhat surprisingly, it did not lead to more consistent cluster assignments (Fig. 2I; pink and green). This result suggests that the structure that emerges for the Gaussian readout, arises somewhat trivially by very aggressively forcing weights to be zero due to the strong L1 penalty, but this does not happen in a consistent way across models, therefore not improving the consistency

0

20

40

60

80

100

adaptive

0.32

0.34

0.36

0.38

Validation score

=0.5

=0.1

0.0

0.5

1.0

1.5

2.0

Learnt

values

10 0

10 1

10 2

10 3

10 4

Weights distribution

20

40

60

80

Number of clusters

0.1

0.2

0.3

0.4

ARI

=0.1=50

=0.1=30

=0.1=10

=0.1=1

=0.5=50

=0.5=30

=0.5=10

A

B

C

of embeddings across runs. Note that the factorized readout is always L 1 -regularized - it does not work without it [28] and is very sensitive to the choice of regularization strength (Appendix A.3). This is also why we show only a single gamma value in the table.

Adaptive regularization improves consistency of embeddings while maintaining predictive performance. We next asked what could explain the difference in embedding consistency between factorized and Gaussian readout. One important difference is that the factorized readout jointly regularizes the mask and the neuronal weight. As a consequence, the factorized readout could reduce the L1 penalty on the neuron weights by increasing the size of its receptive field mask (Fig. 2H). As a consequence, the neuronal weights may not be regularized equally.

Motivated by this observation, we devised a new adaptive regularization strategy for the Gaussian readout, where the strength of the L 1 regularizer is adjustable for each neuron:

Here, γ controls the overall strength of regularization as before. The variables β n are learned coefficients for neuron n , onto which we imposed a log-normal hyperprior p ( β n ) ∼ LogNormal (1 , σ ) , achieved by adding the according loss term L β = 1 /σ 2 ∑ N n =0 log( β n ) .

This adaptive regularization procedure introduces an additional hyperparameter, σ , that controls how far each neuron's regularization strength is allowed to deviate from the overall average controlled by γ . We found that for relatively large σ = 0 . 5 , the model essentially learns to down-regulate all neurons' weights (Fig. 3B) and falls back to a very mildly regularized mode countering the effect of increasing γ (Fig. 3A; blue). When constraining the relative weights more closely around 1 ( σ = 0 . 1 ), the model indeed learns to redistribute the regularization across neurons. The average weights remain closer to 1 (Fig. 3B) and changing the overall regularization strength γ does have an effect (Fig. 3A).

The optimal model chosen by this adaptive regularization strategy ( γ = 10 , σ = 0 . 1 ) indeed performs well on all dimensions we explored: First, its predictive performance matches the state-of-the-art model with the Gaussian readout (Fig. 2A). Second, its learned neuronal embeddings are more consistent that those of both factorized and uniformly regularized Gaussian readout (Fig. 2I; purple).

Pruning models post-hoc improves consistency of neuronal clusters. Wesaw that the consistency of neuronal clusters could be improved by adaptive regularization of the readout. However, in absolute terms the consistency is still not very high. One potential reason for this could be that deep models are typically highly overparameterized [34]. As a result, several of the feature maps of the core

2.5

5.0

7.5

10.0

12.5

15.0

Channels left

0.225

0.250

0.275

0.300

0.325

0.350

0.375

0.400

Validation Score

20

40

60

80

100

Number of clusters

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

ARI

=0

=50

=100

=0, 7ch

=50, 11ch

=100, 12ch

factorised

adaptive

=10

0.30

0.32

0.34

0.36

0.38

Validation Score

0.10

0.15

0.20

0.25

0.30

ARI

non-pruned

pruned

adaptive

=10

factorised

A

B

C

could be redundant, resulting in multiple ways to represent the same neuronal function. In this case, two neurons with the same function could have highly different embeddings. We therefore now investigate whether pruning trained networks could lead to more consistent neuronal embeddings across model fits. We took the following approach to pruning models:

Train a model as usual.

In the trained model, mask one core output convolutional channel and compute performance.

Remove the channel for which masking decreases model performance the least.

Finetune the model on the reduced core.

Repeat steps 2-4 until only one channel is left.

In agreement with prior work on pruning neural nets [34], we observe a slight performance improvement during first pruning iterations and find that 1/3 to 1/2 of the convolutional channels can be removed without affecting performance (Fig. 4A). We selected the smallest number of channels before the performance starts to decrease. We also observed that for the more regularized models more channels should be kept after pruning (Fig. 4A), which makes sense as severe L 1 regularization is also a feature selection mechanism. Pruning noticeably improved the consistency of clustering neurons for models with the Gaussian readout (Fig. 4B), but interestingly only for the regularized models. However, this improvement in consistency still comes at the cost of predictive performance. Overall, the (non-pruned) model with the adaptive readout achieves the best consistency-performance trade-off (Fig. 4C, purple).

Neuronal tuning properties. To explore to how improving reproducibility of neural embeddings impact neurons' actual behavior, we examined the predicted neurons' tuning with respect to four known properties of V1 neurons: orientation tuning, phase invariance, surround suppression and cross-orientation inhibition (see Section 4 for details). We chose these properties because they are well-established and widely studied nonlinear phenomena of V1 neurons [12, 17]. We performed in-silico experiments showing classical grating-based stimuli and computed neuronal tuning curves from the predicted model responses. Note that the neurons have not seen any of these stimuli during the in-vivo experiment. Hence, our models have not been explicitly trained on these stimuli. Thus, we cannot compare these predictions against ground truth. However, we can evaluate qualitatively whether the predicted tuning curves reproduce what is known from biology and we can quantify how consistently these phenomena are expressed across predictive models that just differ in their initialization (in an ideal world tuning curves would be identical across seeds).

We found that different readouts gave rise to substantially different tuning properties for the same neurons (Fig. 5, top row; one example neuron's tuning curves). Strong uniform regularization of Gaussian readouts ( γ = 50 and γ = 100 ) made neuron's responses highly biased towards their mean response, while the non-regularized Gaussian model exhibited the biggest responses amplitude,

0

2

3

2

2

Phase

0

2

4

6

8

10

Tuning Curves

for neuron 5968

Phase Invariance

0

2

3

2

2

Orientation

Orientation Tuning

0

5

10

15

Gabor diameter

Surround Suppression

0

2

4

6

8

Orthogonal mask contrast

Cross-Orientation Inhibition

0.00

0.25

0.50

0.75

1.00

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Indexes distribution

for six sessions, %

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

factorised

=0

=50

=100

adaptive

=10

Index Value

followed by the adaptively regularized Gaussian and the factorized model, which both exhibit moderate shrinkage effects.

For each of the four response properties, we computed a tuning index that measures the degree of modulation. As tuning indices measure the effect size, tuning curves with small amplitude and high baseline lead to high phase invariance, and low orientation tuning, surround suppression, and crossorientation inhibition tuning indices (distributions in Fig. 5 bottom). Non-regularized Gaussian and factorized models exhibited similar distributions. This result suggests that even though regularization leads to sparser and less overfitted embeddings, it also removes important functional properties due to reduced expressive power. The adaptively regularized models stayed in between of the nonregularized and highly regularized models, as it puts different regularization weights across various neurons, effectively combining both modes. Thus, it keeps the overall regularization coefficient γ low, offering a good combination of both viable tuning properties and preserved consistent clustering.

6 Discussion

We performed several important reproducibility checks for the commonly used readout mechanisms of predictive models of the visual cortex. We found that the older factorized readout led to a more structured embedding space, more consistent neuronal clusters and more reproducible in-silico tuning

curves than the more recent, performance-optimized Gaussian readout. The structure of the embedding space was primarily related to the L 1 regularization of the weights. By equipping the more recent Gaussian readout with a novel, adaptive L 1 regularization, we could recover a similarly structured embedding space and achieve more consistent neuronal clusters without sacrificing predictive performance. However, all of the models with a regularized Gaussian readout exhibited strongly biased neuronal tuning curves, calling into question how faithfully they can represent neuronal tuning beyond experimentally verified properties such as maximally exciting stimuli [37, 47, 62, 63, 68]. Rigorous, biologically meaningful evaluation remains an open question, but the presence of non-linear phenomena (tuning curves) is crucial. While current methods focus on explained variance/correlation, models with similar scores can differ in explainability [53]. To our knowledge, we conducted state-of-the-art evaluations for biological meaning by testing the tuning curves.

An important goal of models is that they should be resistant to small hyperparameters changes and different initial conditions in order to lead to meaningful biological conclusions. The comparably low consistency of state-of-the-art models across seeds suggests that future research is required to improve the consistency of models, potentially by improving identifiability or improved training paradigms that find robust solutions.

One limitation of our work is that we focused on one core architecture (rotation-equivariant CNN). We do not have a reason to believe that other core architectures would behave fundamentally differently. One sanity check with a regular CNN core (Appendix A.6) yielded similar results. Whether the same holds for Vision Transformer cores [64] remains to be checked.

In conclusion, our work (1) raises the important question how reproducible current neuronal predictive models are; (2) suggests a framework to assess model consistency along orthogonal dimensions (clustering of neuronal embeddings, predicted responses and tuning curves); (3) provides a technique to improve the consistency of the clusters with keeping the predictive performance high.

References

David H Hubel and Torsten N Wiesel. 'Receptive fields, binocular interaction and functional architecture in the cat's visual cortex'. In: The Journal of physiology 160.1 (1962), p. 106.

Colin Blakemore and Elisabeth A. Tobin. 'Lateral Inhibition between Orientation Detectors in the Cat's Visual Cortex'. en. In: Experimental Brain Research 15 (1972), pp. 439-440. ISSN: 0014-4819, 1432-1106. DOI: 10.1007/BF00234129 .

M. Concetta Morrone, D. C. Burr, and Lamberto Maffei. 'Functional Implications of CrossOrientation Inhibition of Cortical Visual Cells. I. Neurophysiological Evidence'. en. In: Proceedings of the Royal Society of London. Series B. Biological Sciences 216 (1982), pp. 335354. ISSN: 0080-4649, 2053-9193. DOI: 10.1098/rspb.1982.0078 .

Lawrence Hubert and Phipps Arabie. 'Comparing partitions'. In: Journal of classification 2 (1985), pp. 193-218.

A. B. Bonds. 'Role of Inhibition in the Specification of Orientation Selectivity of Cells in the Cat Striate Cortex'. en. In: Visual Neuroscience 2 (1989), pp. 41-55. ISSN: 1469-8714, 0952-5238. DOI: 10.1017/S0952523800004314 .

Richard T Born and RB Tootell. 'Single-unit and 2-deoxyglucose studies of side inhibition in macaque striate cortex.' In: Proceedings of the National Academy of Sciences 88.16 (1991), pp. 7071-7075.

G. C. DeAngelis et al. 'Organization of Suppression in Receptive Fields of Neurons in Cat Visual Cortex'. In: Journal of Neurophysiology 68 (1992), pp. 144-163. ISSN: 0022-3077. DOI: 10.1152/jn.1992.68.1.144 .

David J. Heeger. 'Normalization of Cell Responses in Cat Striate Cortex'. en. In: Visual Neuroscience 9 (1992), pp. 181-197. ISSN: 1469-8714, 0952-5238. DOI: 10.1017/ S0952523800009640 .

G. C. DeAngelis, R. D. Freeman, and I. Ohzawa. 'Length and Width Tuning of Neurons in the Cat's Primary Visual Cortex'. In: Journal of Neurophysiology 71 (1994), pp. 347-374. ISSN: 0022-3077. DOI: 10.1152/jn.1994.71.1.347 .

Matteo Carandini, David J. Heeger, and J. Anthony Movshon. 'Linearity and Normalization in Simple Cells of the Macaque Primary Visual Cortex'. en. In: Journal of Neuroscience 17 (1997), pp. 8621-8644. ISSN: 0270-6474, 1529-2401. DOI: 10.1523/JNEUROSCI.17-2108621.1997 .

James R Cavanaugh, Wyeth Bair, and J Anthony Movshon. 'Nature and interaction of signals from the receptive field center and surround in macaque V1 neurons'. In: Journal of neurophysiology 88.5 (2002), pp. 2530-2546.

James R. Cavanaugh, Wyeth Bair, and J. Anthony Movshon. 'Nature and Interaction of Signals From the Receptive Field Center and Surround in Macaque V1 Neurons'. In: Journal of Neurophysiology 88 (2002), pp. 2530-2546.

James R. Cavanaugh, Wyeth Bair, and J. Anthony Movshon. 'Selectivity and Spatial Distribution of Signals From the Receptive Field Surround in Macaque V1 Neurons'. In: Journal of Neurophysiology 88 (2002), pp. 2547-2556. ISSN: 0022-3077. DOI: 10.1152/jn.00693. 2001 .

Tobe C.B Freeman et al. 'Suppression without Inhibition in Visual Cortex'. en. In: Neuron 35 (2002), pp. 759-771. ISSN: 08966273. DOI: 10.1016/S0896-6273(02)00819-X .

Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems . MIT press, 2005.

L Van Der Maaten and G Hinton. 'Visualizing high-dimensional data using t-629 sne'. In: Journal of Machine Learning Research 9.2579-2605 (2008), p. 630.

Laura Busse, Alex R. Wade, and Matteo Carandini. 'Representation of Concurrent Stimuli by Population Activity in Visual Cortex'. In: Neuron 64 (2009), pp. 931-942. ISSN: 0896-6273. DOI: 10.1016/j.neuron.2009.11.004 .

Laurens Van Der Maaten. 'Learning a parametric embedding by preserving local structure'. In: Artificial intelligence and statistics . PMLR. 2009, pp. 384-391.

NX Vinh, J Epps, and J Bailey. 'Information Theoretic Measures for Clusterings Comparison: Variants'. In: Properties, Normalization and Correction for Chance 18 (2009).

Anne K Kreile, Tobias Bonhoeffer, and Mark Hübener. 'Altered visual experience induces instructive changes of orientation preference in mouse visual cortex'. In: Journal of Neuroscience 31.39 (2011), pp. 13911-13920.

Brett Vintch, J Anthony Movshon, and Eero P Simoncelli. 'A convolutional subunit model for neuronal responses in macaque V1'. In: Journal of Neuroscience 35.44 (2015), pp. 1482914841.

Ján Antolík et al. 'Model constrained by visual hierarchy improves prediction of neural responses to natural scenes'. In: PLoS computational biology 12.6 (2016), e1004927.

Tom Baden et al. 'The functional diversity of retinal ganglion cells in the mouse'. en. In: Nature 529.7586 (Jan. 2016), pp. 345-350. ISSN: 0028-0836, 1476-4687. DOI: 10.1038/ nature16468 . URL: http://www.nature.com/articles/nature16468 (visited on 10/19/2021).

Hengyuan Hu et al. 'Network trimming: A data-driven neuron pruning approach towards efficient deep architectures'. In: arXiv preprint arXiv:1607.03250 (2016).

Satoru Kondo and Kenichi Ohki. 'Laminar differences in the orientation selectivity of geniculate afferents in mouse primary visual cortex'. In: Nature Neuroscience 19.2 (2016), pp. 316319.

Lane McIntosh et al. 'Deep learning models of the retinal response to natural scenes'. In: Advances in neural information processing systems 29 (2016).

Pavlo Molchanov et al. 'Pruning convolutional neural networks for resource efficient inference'. In: arXiv preprint arXiv:1611.06440 (2016).

David Klindt et al. 'Neural system identification for large populations separating 'what' and 'where''. In: Advances in neural information processing systems 30 (2017).

Hao Li et al. Pruning Filters for Efficient ConvNets . 2017. arXiv: 1608.08710 [cs.CV] .

Jian-Hao Luo and Jianxin Wu. 'An entropy-based pruning method for cnn compression'. In: arXiv preprint arXiv:1706.05791 (2017).

Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 'Thinet: A filter level pruning method for deep neural network compression'. In: Proceedings of the IEEE international conference on computer vision . 2017, pp. 5058-5066.

Kirstie J Salinas et al. 'Contralateral bias of high spatial frequency tuning and cardinal direction selectivity in mouse visual cortex'. In: Journal of Neuroscience 37.42 (2017), pp. 1012510138.

Alexander S Ecker et al. 'A rotation-equivariant convolutional neural network model of primary visual cortex'. In: arXiv preprint arXiv:1809.10504 (2018).

Jonathan Frankle and Michael Carbin. 'The lottery ticket hypothesis: Finding sparse, trainable neural networks'. In: arXiv preprint arXiv:1803.03635 (2018).

Deepak Mittal et al. 'Recovering from random pruning: On the plasticity of deep convolutional neural networks'. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) . IEEE. 2018, pp. 848-857.

Fabian Sinz et al. 'Stimulus domain transfer in recurrent models for large scale cortical population prediction on video'. In: Advances in neural information processing systems 31 (2018).

Pouya Bashivan, Kohitij Kar, and James J DiCarlo. 'Neural population control via deep image synthesis'. In: Science 364.6439 (2019), eaav9436.

Mikhail Belkin et al. 'Reconciling modern machine-learning practice and the classical biasvariance trade-off'. In: Proceedings of the National Academy of Sciences 116.32 (2019), pp. 15849-15854.

Santiago A Cadena et al. 'Deep convolutional models improve predictions of macaque V1 responses to natural images'. In: PLoS computational biology 15.4 (2019), e1006897.

Paul G Fahey et al. 'A global map of orientation tuning in mouse visual cortex'. In: BioRXiv (2019), p. 745323.

Nathan W Gouwens et al. 'Classification of electrophysiological and morphological neuron types in the mouse visual cortex'. In: Nature neuroscience 22.7 (2019), pp. 1182-1195.

Lida Kanari et al. 'Objective morphological classification of neocortical pyramidal cells'. In: Cerebral Cortex 29.4 (2019), pp. 1719-1735.

Dmitry Kobak and Philipp Berens. 'The art of using t-SNE for single-cell transcriptomics'. In: Nature communications 10.1 (2019), p. 5416.

Carlos R Ponce et al. 'Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences'. In: Cell 177.4 (2019), pp. 999-1009.

Federico Scala et al. 'Layer 4 of mouse neocortex differs in cell types and circuit organization between sensory areas'. In: Nature communications 10.1 (2019), p. 4174.

Ivan Ustyuzhaninov et al. 'Rotation-invariant clustering of neuronal responses in primary visual cortex'. In: International Conference on Learning Representations . 2019.

Edgar Y Walker et al. 'Inception loops discover what excites neurons most using deep predictive models'. In: Nature neuroscience 22.12 (2019), pp. 2060-2065.

Ronald (James) Cotton, Fabian Sinz, and Andreas Tolias. 'Factorized Neural Processes for Neural Processes: K-Shot Prediction of Neural Responses'. In: Advances in Neural Information Processing Systems . Ed. by H. Larochelle et al. Vol. 33. Curran Associates, Inc., 2020, pp. 11368-11379. URL: https://proceedings.neurips.cc/paper_files/paper/ 2020/file/82e9e7a12665240d13d0b928be28f230-Paper.pdf .

Nathan W Gouwens et al. 'Integrated morphoelectric and transcriptomic classification of cortical GABAergic cells'. In: Cell 183.4 (2020), pp. 935-953.

Yiping Lu et al. 'A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterization from depth'. In: International Conference on Machine Learning . PMLR. 2020, pp. 6426-6436.

Konstantin-Klemens Lurz et al. 'Generalization in data-driven models of primary visual cortex'. In: BioRxiv (2020), pp. 2020-10.

Samet Oymak and Mahdi Soltanolkotabi. 'Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks'. In: IEEE Journal on Selected Areas in Information Theory 1.1 (2020), pp. 84-105.

Max F Burg et al. 'Learning divisive normalization in primary visual cortex'. In: PLOS Computational Biology 17.6 (2021), e1009028.

Federico Scala et al. 'Phenotypic variation of transcriptomic cell types in mouse motor cortex'. In: Nature 598.7879 (2021), pp. 144-150.

Eleanor Batty et al. 'Multilayer recurrent network models of primate retinal ganglion cell responses'. In: International Conference on Learning Representations . 2022.

Katrin Franke et al. 'State-dependent pupil dilation rapidly shifts visual feature selectivity'. In: Nature 610.7930 (2022), pp. 128-134.

Matías A Goldin et al. 'Context-dependent selectivity to natural images in the retina'. In: Nature Communications 13.1 (2022), p. 5556.

L Hoefling et al. 'A chromatic feature detector in the retina signals visual context changes. bioRxiv (2022)'. In: Google Scholar (2022).

Ivan Ustyuzhaninov et al. 'Digital twin reveals combinatorial code of non-linear computations in the mouse primary visual cortex'. In: bioRxiv (2022), pp. 2022-02.

Konstantin F Willeke et al. 'The Sensorium competition on predicting large-scale mouse primary visual cortex activity'. In: arXiv preprint arXiv:2206.08666 (2022).

José E Chacón and Ana I Rastrojo. 'Minimum adjusted Rand index for two clusterings of a given size'. In: Advances in Data Analysis and Classification 17.1 (2023), pp. 125-133.

Zhiwei Ding et al. 'Bipartite invariance in mouse primary visual cortex'. In: bioRxiv (2023).

Jiakun Fu et al. 'Pattern completion and disruption characterize contextual modulation in mouse visual cortex'. In: bioRxiv (2023).

Bryan M Li et al. 'V1T: large-scale mouse V1 response prediction using a Vision Transformer'. In: arXiv preprint arXiv:2302.03023 (2023).

Pawel Pierzchlewicz et al. 'Energy Guided Diffusion for Generating Neurally Exciting Images'. In: Advances in Neural Information Processing Systems . Ed. by A. Oh et al. Vol. 36. Curran Associates, Inc., 2023, pp. 32574-32601. URL: https://proceedings.neurips.cc/ paper_files/paper/2023/file/67226725b09ca9363637f63f85ed4bba- PaperConference.pdf .

Polina Turishcheva et al. 'The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos'. In: arXiv preprint arXiv:2305.19654 (2023).

Eric Y Wang et al. 'Towards a foundation model of the mouse visual cortex'. In: bioRxiv (2023).

Konstantin F Willeke et al. 'Deep learning-driven characterization of single cell tuning in primate visual area V4 unveils topological organization'. In: bioRxiv (2023), pp. 2023-05.

Zizhen Yao et al. 'A high-resolution transcriptomic and spatial atlas of cell types in the whole mouse brain'. In: Nature 624.7991 (2023), pp. 317-332.

Max F Burg et al. 'Most discriminative stimuli for functional cell type clustering'. In: The Twelfth International Conference on Learning Representations . 2024. URL: https: //openreview.net/forum?id=9W6KaAcYlr .

Jiakun Fu et al. 'Heterogeneous orientation tuning in primary visual cortex of mice diverges from Gabor-like receptive fields in primates'. In: Cell Reports (2024). Accepted in principle.

Jan Lause, Philipp Berens, and Dmitry Kobak. 'The art of seeing the elephant in the room: 2D embeddings of single-cell data do make sense'. In: PLOS Computational Biology 20.10 (2024), e1012403.

A Appendix

A.1 Model config and Training pipeline

We stayed close to the Willeke et al. [60] benchmark with list of the following changes. For the model , we used rotation equivariant model with 16 hidden_channels and 8 rotations per channel, which resulted in the total readout dimensionality of 128. grid_mean_predictor was turned-off, input_kern was 13 and hidden_kern was 5, to make models closer to Ecker et al. [33] and Ustyuzhaninov et al. [59]. gamma_input , corresponding to the input Laplace smoothing was set to 500 and gamma_hidden , applying Laplace smoothing over the first layer of convolutuonal kernels was set to 500,000, depth_separable=False For the training , we also used early stopping and learning rate scheduling with 4 steps. The only differences were that we used batch_size =256 instead of 128 and initial learning rate of 0.005 instead of 0.009.

A.2 Parameters of optimal gabor and tuning curves sweep

For the optimal Gabor search, we followed Burg et al. [53] and generated the Gabors with 6 contrasts steps and 8 sizes steps. The maximum input value for the stimuli in the dataset was 1.75, which we used for the maximum contrast and the minimal contrast we set to 1% of the maximum. The stimuli diameter sizes were between 4 and 25. Minimum spacial frequency was 1 . 3 -1 · 1 . 3 i , i = 0 , 1 , ..., 10 .

For orientation tuning 24 orientations equidistantly partitioning the interval between 0 and π were used with 8 phases and maximum response acrross 8 phases was taken for each orientation.

For phase invariance we created stimuli for twelve equidistant phases between 0 and 2 π .

For surround suppression we used the following stimulus creation parameters

For cross-orientation inhibition 9 contrasts and 8 phases were used to create the combinations of the optimal Gabor and its orthogonal version.

A.3 Choice of regularization for the factorized readout

For factorized readouts severe L 1 regularization is design necessity because mask and readout are regularized jointly. The idea of the mask is to learn the receptive field, ideally 1 'pixel' only, so the strong regularization is needed and tightly connected with the performance. We select the regularization based on the best performance γ = 0 . 003 (see Fig. 6).

0.002

0.004

0.006

0.008

0.010

Regularisation coefficient (

)

0.26

0.28

0.30

0.32

0.34

Single-trial validation correlation

A.4 ARI stability - 1 model and 3 seeds for k-means

20

40

60

80

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

=0

0ch

9ch

20

40

60

80

=50

0ch

5ch

20

40

60

80

=100

0ch

4ch

number of cluster

ARI value

A.5 Selection of channels during pruning

To select the amount of channels, we train same models with 3 different starting seeds and estimate the mean barρ and σ ρ of performance on each pruning stage. accounting for std using the following logic: ¯ ρ i -σ ρ i < = ¯ ρ i +1 -σ ρ i +1 where i is the amount of channels pruned.

A.6 Motivation for rotation equivariant core and non-equivariant control

The motivation behind the rotation-equivariant core is that V1 neurons are orientation selective and neurons with all preferred orientations exist. Thus, any feature is likely to be extracted in multiple rotated versions [33, 46, 59]. To control that our results are not specific to the rotation-equivariant core, we performed a few control experiments with a non-rotation-equivariant CNN core (Fig. A.6): The general pattern of results is the same: the adaptive regularization improves consistency of clustering and so does pruning feature maps.

0

20

40

60

80

100

120

140

Number of clusters

0.050

0.075

0.100

0.125

0.150

0.175

0.200

0.225

ARI

=0

=50

lognorm

=10

=0.003

0

20

40

60

80

Number of clusters

0.06

0.08

0.10

0.12

0.14

0.16

0.18

0.20

ARI

=0, 16ch

=0, 13ch

=50, 16ch

=50, 13ch

lognorm

=10

=0.002

A.7 Table with correlations for shifter turned on

A.8 Optimal Gabors

Gamma=0

seed=42

Neuron

1770

Neuron

2566

Neuron

5476

Neuron

5968

Neuron

6114

Neuron

6886

Neuron

6890

Neuron

7736

Neuron

7740

Neuron

7801

Gamma=0

seed=101

Gamma=0

seed=7607

Gamma=50

seed=42

Gamma=50

seed=101

Gamma=50

seed=7607

Gamma=100

seed=42

Gamma=100

seed=101

Gamma=100

seed=7607

Optimal Gabors of non-pruned models

Optimal Gabors of pruned models

Gamma=0

seed=42

Neuron

1770

Neuron

2566

Neuron

5476

Neuron

5968

Neuron

6114

Neuron

6886

Neuron

6890

Neuron

7736

Neuron

7740

Neuron

7801

Gamma=0

seed=101

Gamma=0

seed=7607

Gamma=50

seed=42

Gamma=50

seed=101

Gamma=50

seed=7607

Gamma=100

seed=42

Gamma=100

seed=101

Gamma=100

seed=7607

A.9 Other clustering consistency metrics

ARI is the most popular metric for pairwise cluster comparisons [19], which we think is more appropriate than set-based methods. To ensure we are not missing anything, we also considered alternative metrics such as Fowlkes-Mallows, completeness, homogeneity, and v-measure (a particular case of mutual information) and repeated the analysis. The results show the same ordering of methods A.9. Do the metrics reflect biological significance? We reasoned that if neuronal embeddings determine the response function of a neuron in the model, then two neurons that have similar

embeddings under one instance of a model, should also have similar embeddings under another instance of this model. Therefore, measuring similarity of clustering is a reasonable proxy.

20

40

60

80

100

Number of clusters

0.1

0.2

0.3

0.4

0.5

Fowlkes-Mallows

=0

=50

=100

lognorm

=10

=0, 7ch

=50, 11ch

=100, 12ch

=0.003

A.10 t-SNE reliability

Please note that we used t-SNE only for visualization purposes to show the presence of the density modes in the data. While t-SNE plots are indeed not sufficient to make any conclusions about the data, many papers showed that they are a good tool for exploratory data analysis [72] and we additionally checked trustworthiness metric [18] after performing t-SNE. Trustworthiness shows higher scores for more regularized models with varying numbers of neighbors confirming the presence of the density modes (Fig. 14).

A.11 Adaptive regularization pruning

It turns out that pruning does not further increase ARI for the adaptively regularized model (Fig. 15). Thus, regularizing neurons in a non-uniform fashion is already effective enough that additional pruning does not improve it. We speculate that this is because some noisy neurons become sufficiently regularized without pruning, while before such regularization was only achieved with pruning and regularization combined.

A.12 Computational resources

All of the experiments were performed on a local infrastructure cluster with 8 NVIDIA RTX A5000 GPUs with 24Gb of memory each. 2 models could be trained on a single GPU simultaneously in ≈ 3 hours. Pruning experiment, without parallelization, would take a week. No extensive cpu resources are required. Optimal Gabor search, without parallelization takes around 12hours for Gaussian readout model and 3 days for the factorized readout model. We also pre-saved optimal Gabors in batches to improve the speed, which required ≈ 100Gb of memory.

A.13 Broader impact

Our work is an important step towards obtaining a functional cell types taxonomy for the primary visual cortex neurons. Such classification can significantly improve our understanding of the brain and help on the way to understand the mechanisms and develop the treatment for different neurodegenerative disease.

20

40

60

80

100

Number of clusters

0.20

0.25

0.30

0.35

0.40

v-measure

20

40

60

80

100

nember of neighbors

0.86

0.88

0.90

0.92

0.94

0.96

0.98

trustworthiness

=0

=50

=100

=0.003

lognorm

=10

20

40

60

80

100

Number of clusters

0.1

0.2

0.3

0.4

0.5

ARI

=100, 16ch

=100, 12ch

=50, 16ch

=50, 11ch

=0, 16ch

=0, 10ch

lognorm

=10, 16ch

lognorm

=10, 13ch

=0.003, 16ch

=0.003, 12ch

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Introduction provides a clear list of paper contributions

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly admit that nevertheless the improvement was done on ARI, it could have been improved more. The Discussion section discussed the ponetial acpects we have not addressed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In appendix there are 2 sections describing the model and training pipeline we followed and modifications to it. We also report the seeds and neuron ids for the pictures and the data subsampling procedure when applicable and specific vizualization settings.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The open source available dataset was used in the experiments. The code will be public upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have 2 section in the appendix specifying hyperparameters. The data splits for train-test-validation are fixed in the open version of the dataset we used. We do not provide the specific ids for 14,000 neurons used for ARI and vizualization because it is supposed to be generalizable and the specific ids are not important.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The 1-sigma error bars are present in the majority of the plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 'Computational resources' in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All employers are payed fair. Open accessed data was used. No ethics concerns are involved in the experiments.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader impact section in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the dataset and code used from Sensorium 2022 Benchmark competition ([60]) and mention the open source 'neuralpredictors' package. Both are publically available for research purposes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The only new asset is code and it will be available upin publication with extensive documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work demonstrates that overparameterized neural network models, which have many non-unique solutions, can lead to inconsistencies in representing the mouse visual cortex. It suggests a novel approach called \"adaptive regularization,\" where the regularization parameters in the loss term are learnable rather than fixed. The study also examines other approaches, such as normal regularization and pruning, and finds that these methods are beneficial for improving representation consistency. The proposed method significantly enhances consistency. This contribution is significant to the computational neuroscience community.\n\nSTRENGTHS:\n1. This work addresses a critical problem in computational neuroscience: reproducibility. It proposes solutions to mitigate this issue, which have significant impact on the computational neuroscience community. \n\n2. The authors rigorously define the consistency of computational models and thoroughly examine how regularization strength affects neuronal properties.\n\nWEAKNESSES:\n1. The study is limited to only one type of model, the Rotational Equivariance CNN.\n\n2. In the Methods section, the terms \"Embedding\" and \"Mask\" are not defined. Moreover, Lp and L1 are described as functions of model parameters, but no parameters are defined in the text. This lack of clarity poses an issue for understanding. For instance, Lp could be interpreted in two different ways: as a function of the core but not the readout, or as a function of both the core and the readout. The same ambiguity applies to L1.\n\n3. There are two types of computational neuroscience models regarding their outputs. The first type is task-optimized models, whose outputs are task-related, such as object class, object representation, or action. These models are trained to perform downstream tasks, such as supervised object classification, reconstruction, or playing games [1,2,3]. The second type is neural response fitting models, whose outputs are predicted neural responses, and they are trained to predict these responses directly. However, the reasons why the authors chose response fitting models over task-optimized models are not mentioned.\n\nReference\n\n[1] Performance-optimized hierarchical models predict neural responses in higher visual cortex\n\n[2] Unsupervised neural network models of the ventral visual stream\n\n[3] Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments\n\nQUESTIONS:\n1. Oother models that, despite lacking rotational equivariance, still demonstrate high capability in predicting neural responses measured by electrodes [1]. Why did the authors choose the Rotational Equivariance CNN? What was the motivation behind this choice?\n\n2. Figure 2A shows the models' performances. However, what about the factorized readout with L1 regularization? Does the predictive performance decrease?\n\n3. Why did the authors choose a log-normal prior for the learnable coefficient?\n\n4. Why did the authors not examine adaptive regularization for factorized readout models? The implementation should be straightforward, similar to what was done for Gaussian.\n\n5. Why does the factorized model have one γ in Table 1?\n\n\nReference\n\n[1] https://www.brain-score.org/vision/\n\nLIMITATIONS:\nThis study investigates the consistency of neuronal properties and the prediction performance of regularized models. However, the models used by the authors are trained to predict neural responses. There is another type of computational model that is trained to perform downstream tasks, with neural-like representations emerging from the training. Future work could consider these latter models.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a systematic investigation of the use of deep neural network fits to biological neurons as the basis for neuron cell type classification. The authors explore how various factors such as regularization and model pruning can influence both the predictive model fit and the consistency of the neural clustering.\n\nSTRENGTHS:\nThis paper presents a well-motivated, calculated set of experiments for evaluating how well deep neural networks can be used as models of mouse visual cortex. With increasing interest in this area and increasingly bold claims, work like this is a breath of fresh air. The controlled experiments and attention to detail provides compelling evidence that though DNNs may be our best working computational model of visual cortex, the are likely not a global optimum.\n\nWEAKNESSES:\nOverall, the biggest weakness I think is some lack of explanation. As someone who is very familiar with work in computational neuroscience but not a neuroscientist myself, there were a few things that felt under-explained and it wasn't clear if it was because I'm not the correct audience or because the authors did not provide enough context in the text. The authors do a good job of providing clear motivation for some of the fundamentals (L24-53) but then leave out context when things get a little more technical specialized (i.e. comparisons between readout mechanisms, significance of biological properties and tuning indices, etc) as a result it feels unclear what the reader is expected to know before reading.\n\n- The font in the figures is a little unprofessional. Comic sans? In this economy?\n- Figure 2G is hard to parse. I recommend splitting up the histograms\n- The intuitive explanation of the factored vs gaussian vs adaptive readouts is not super clear to me. It might be worthwhile to spend more time on it, especially to motivate your proposed adaptive mechanism.\n- minor typos (\"initialiation\" in L183)\n- L216 is nonsensical. I think some words are out of order\n- Confidence intervals in Table 1 would be nice to gauge significance. Also, the table format makes it a little hard to visualize. Perhaps a graphical representation would be better.\n\nQUESTIONS:\n- Why is the adaptive model not pruned in the experiments shown in fig 4? I feel like it makes sense to include it in the pruning experiments if possible for consistent comparison across experiments.\n\n- I think more explanation of table 1 is warranted. First, what is the ideal case? It's not clear to me that the optimal computational model has neurons with precise tuning curves for these chosen functional biological properties. If this is obvious to the authors, then perhaps there should be explicit motivation in the text. Why these biological properties and not others? Do biological neurons in HVC demonstrate each of these properties robustly (and not others)? Is it fair to expect that neurons in silica should each individually exhibit these properties? Second, the difference in NMAE of tuning indices between the factorized and the gaussian readouts seems very large. Large enough that I would expect more discussion of it in the text. How big of a difference in score is this really, is there something that can put it in context? If the biological property fits are a big deal, then is this a nail in the coffin for gaussian readout? Is gaussian readout a hack for improving overall fit while sacrificing something more important? Or is the functional biological properties just a cool trick that would be nice to have but not essential.\n\nLIMITATIONS:\nLimitations are clear.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies models trained to predict the responses of neurons in visual cortex. The model has a shared multilayer network core followed by a final layer which maps the core features into individual neuron responses. The key question that this paper asks is, How reproducible are the individual neuron properties and \"cell-type\" clusterings inferred by such techniques? The main conclusion is that sparsity-inducing regularization is important for finding consistent cell properties. Pruning certain channels in the core is also shown to improve consistency.\n\nSTRENGTHS:\nConsistency and reliability of these methods for understanding visual cortical processing is important for the scientific interpretation of these models. The design of the approach seems sound. A number of metrics are used, both scores as well as response properties used in neuroscience.\n\nWEAKNESSES:\n* I found the comic sans font and cartoon diagram style distracting in Figs 1-4.\n* It isn't clear that much is gained from the t-SNE plots in Fig 2. Some evidence of different clusters, yes, but it's known that these kind of plots can be deceiving.\n* How do you pick the optimal regularization parameters? By what metric/cross-validation procedure? (Line 242 gives $\\gamma=10, \\sigma=0.1$) Validation performance is reported but I'm unclear of the splitting procedure and whether there was a train/valid/test 3-way split or not.\n* Minor points are made in \"questions\" section\n\nQUESTIONS:\n* Line 35 \"well described\" -> \"well-described\" typo\n* Line 62 \"We address the identifiability problem, by proposing\" typo, remove comma\n* Fig 1 caption \"sampling receptive field positions form a Gaussian\" typo\n* Is the clustering shown in Fig 1B truly \"rotation invariant\" or related to cyclic permutations of the vector indices? It is unclear from the text. The figure does not depict a rotation (i.e. action by an orthogonal matrix), since the orbit is elliptical rather than circular and it is not centered at the origin.\n* Fig 3A has $\\sigma=01$ which should be $\\sigma=0.1$\n* Fig 3B axis \"amount of such weights\" sounds strange\n\nLIMITATIONS:\nThe authors mention that their work is limited to a particular type of core architecture. It's also limited to being applied to just one dataset, whereas open data in other animals (or collected by other research groups, e.g. the Allen Institute's work) are available. I am not suggesting the authors do this in their revision but that they mention it as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Reproducibility of predictive networks for mouse visual cortex,\" explores the reproducibility of neuronal embeddings in the mouse visual cortex using deep predictive models. By introducing adaptive regularization and iterative feature pruning, the authors address key issues related to model overparameterization and provide a robust framework for achieving consistent functional representations of neurons. The work lays the groundwork for future research aimed at developing reliable and interpretable models for understanding neuronal function. \n\nThe primary goal is to investigate the stability and reproducibility of neuronal function embeddings derived from deep predictive models. These models aim to predict neuronal responses to sensory inputs and have been proposed to define functional cell types via unsupervised clustering. The paper addresses the concern that deep models are often highly overparameterized, leading to multiple solutions that can represent the same neuronal function, thereby questioning the reliability of embeddings for downstream analysis.\n\nThe paper demonstrates that L1 regularization, which was used in early models, is crucial for obtaining structured and consistent neuronal embeddings when newer readout mechanisms are used. A novel adaptive regularization scheme is introduced, which adjusts the strength of regularization for each neuron. This method improves the consistency of neuronal embeddings across different model fits while maintaining predictive performance. The paper proposes an iterative feature pruning strategy to reduce the dimensionality of performance-optimized models by half without losing predictive performance. This pruning improves the consistency of neuronal embeddings concerning clustering neurons.  The consistency of neuronal embeddings is evaluated using the Adjusted Rand Index (ARI) of clustering partitions across models, correlations of predicted responses across models, and the consistency of tuning indexes describing known nonlinear functional properties of visual neurons.\n\nSTRENGTHS:\n- Adaptive Regularization Scheme: The introduction of an adaptive regularization method that adjusts the regularization strength per neuron is a significant novelty. This approach helps achieve better consistency in clustering neuronal embeddings without compromising on predictive accuracy.\n\n- Iterative Feature Pruning: The feature pruning strategy to address overparameterization in deep models is another novel contribution. By systematically reducing the model’s dimensionality, the authors enhance the robustness and consistency of the neuronal embeddings.\n\n- Comprehensive Consistency Evaluation: The paper provides a comprehensive evaluation of model consistency across different dimensions (embedding clustering, predicted responses, and tuning curves). This thorough approach highlights the robustness and reproducibility of the proposed methods.\n\n- The paper is well-written with generally sufficient referencing to the previous methods.\n\nWEAKNESSES:\n- Bias from Regularization and Pruning: While the adaptive regularization and pruning strategies improve consistency, they may introduce biases that affect the biological validity of the neuronal embeddings. Over-regularization, for instance, can reduce the model’s expressive power and lead to less biologically plausible representations.\n\n- The study focuses primarily on models with a rotation-equivariant convolutional neural network (CNN) core. The authors acknowledge that other core architectures, such as regular CNNs or Vision Transformers, were not evaluated. This limitation means the findings may not generalize across different model architectures.\n\n- Despite the improvements in clustering consistency, there is a trade-off between consistency and predictive performance. The pruning and regularization strategies, while improving consistency, sometimes result in a drop in predictive performance, which is not ideal.\n\n- The introduction of adaptive regularization adds another layer of hyperparameters (e.g., the log-normal hyperprior parameter) that need to be carefully tuned. This increases the complexity of the model training process and may require significant computational resources.\n\n- The focus on achieving high consistency in clustering and embeddings may overshadow other important factors, such as the interpretability and biological relevance of the model outputs. Balancing consistency with these factors is crucial for developing useful predictive models in neuroscience.\n\nQUESTIONS:\n- Is ARI a metric you developed or did it exist before? If it existed before, it requires referencing. \n\n- Have you considered evaluating other core architectures, such as Vision Transformers or regular CNNs, to see if the findings generalize across different model types? What specific reasons led you to choose a rotation-equivariant CNN core for this study?\n\n- Could you elaborate on the potential biases introduced by the adaptive regularization and pruning strategies? How do you mitigate these biases? Have you explored other regularization techniques or pruning strategies that might offer a better balance between consistency and predictive performance?\n\n- Are there specific aspects of model performance that are most critical for maintaining biological plausibility?\n\n- The adaptive regularization introduces additional hyperparameters. How do you approach hyperparameter tuning to ensure optimal model performance? What are the computational costs associated with this tuning process? \n\n- Why did you choose the Adjusted Rand Index (ARI) and other metrics for evaluating consistency? Are there alternative metrics that might provide additional insights? How do you ensure that the chosen metrics accurately reflect the biological significance of the neuronal embeddings?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}