Learning to cluster neuronal function
Authors: Nina S. Nellen
Polina Turishcheva
Michaela Vystrčilová
Shashwat Sridhar
Tim Gollisch
Andreas S. Tolias
Alexander S. Ecker
Venue: NeurIPS 2025 poster ()
Keywords: reproducibility
predictive models for visual cortex
neuroscience
inductive bias
clustering
Primary area: neuroscience_and_cognitive_science
TL;DR: Explicit clustering bias added during training improves structural consistency of cell embeddings but does not reveal clear cell types in mouse V1

Abstract: Deep neural networks trained to predict neural activity from visual input and behaviour have shown great potential to serve as digital twins of the visual cortex.
Per-neuron embeddings derived from these models could potentially be used to map the functional landscape or identify cell types. 
However, state-of-the-art predictive models of mouse V1 do not generate functional embeddings that exhibit clear clustering patterns which would correspond to cell types.
This raises the question whether the lack of clustered structure is due to limitations of current models or a true feature of the functional organization of mouse V1. 
In this work, we introduce DECEMber -- Deep Embedding Clustering via Expectation Maximization-based refinement -- an explicit inductive bias into predictive models that enhances clustering by adding an auxiliary $t$-distribution-inspired loss function that enforces structured organization among per-neuron embeddings.
We jointly optimize both neuronal feature embeddings and clustering parameters, updating cluster centers and scale matrices using the EM-algorithm.
We demonstrate that these modifications improve cluster consistency while preserving high predictive performance and surpassing standard clustering methods in terms of stability.
Moreover, DECEMber generalizes well across species (mice, primates) and visual areas (retina, V1, V4). 
The code is available at https://github.com/Nisone2000/DECEMber, https://github.com/ecker-lab/cnn-training.

Learning to cluster neuronal function
Nina S. Nellen, Lt. Polina Turishcheva, 1tt." Michaela Vystrčilová,
Shashwat Sridhar, Tim Gollisch, 2-5 Andreas S. Tolias, 6-9 Alexander S. Ecker1.10."
Institute of Computer Science and Campus Institute Data Science, University Göttingen, Germany 2 University Medical Center Göttingen, Department of Ophthalmology. Göttingen, Germany Bernstein Center for Computational Neuroscience Götrngen, Götringen, Germany
Cluster of Excellence "Multiscale Bioimaging: from Molecular Machines Networks of Excitable Cells" (MBExC), University of Göttingen, Germany
Else Kröner Fresenius Center for Optogenetic Therapies, University Medical Center Göttingen, Germany Department of Ophthalmology. Byers Bye Institute, Stanford University School of Medicine, CA, US
Stanford Bio-X, X, Stanford University, Stanford, CA, US
Wu Tsai Neurosciences Institute, Stanford University, CA, US Department Electrical Engineering. Stanford University, CA,, US Max Planck Institute for Dynamics and Self- Self- Organization, Göttingen, Germany
Shared contribution
"nina wwwelleeeenngoeettnneeeeeee "(turishcheva, ecker)ecss uni-goettingen.de
Abstract
Deep neural networks trained to predict neural activity from visual input and be- haviour have shown great potential to serve digital twins of the visual cortex. Per-neuron embeddings derived from these models could potentially used map the functional landscape or identify cell types. However, state-of-the-art predictive models of mouse V1 do not generate functional embeddings that exhibit clear clustering patterns which would correspond to cell types. This raises the question whether the lack of clustered structure is due to limitations of current models or a true feature of the functional organization of mouse V1. In this work, we introduce DECEMber - Deep Embedding Clustering via Expectation Maximization- -based refinement explicit inductive bias into predictive models that enhances clus- tering by adding an auxiliary
cetrributiioninssiiee loss function that enforces structured organization among per-neuron embeddings. We jointly optimize both neuronal feature embeddings and clustering parameters, updating cluster centers and scale matrices using the EM-algorithm. We demonstrate that these modifica- tions improve cluster consistency while preserving high predictive performance and surpassing standard clustering methods in terms of stability. Moreover, DECEM- ber generalizes well across species (mice, primates) and visual areas (retina, V1, V4). The code is available at https: /tte::///ttttee com/ wat.ssone0000DEEHEE0 /ttps://ght thub. cwwcccoc/cca/oo com/ecker-oo lab/ cnn con-training.
Introduction
Understanding whether neurons form discrete cell types or lie on continuum is fundamental question in neuroscience [1]. Previous research has extensively investigated the morphological and electrophysiologica properties of neurons in the visual cortex. While discrete anatomical and transcriptomic classifications have been proposed [2-4], recent work on the mouse brain suggests a
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

more continuous organization [5, 6]. Significantly less attention has been devoted to the neurons' functional properties. Each neuron can characterized by a function that maps high-dimensione sensory inputs to its one-dimensional neuronal response. These functions are highly non-linear, making their analysis complex. Discrete functional cell types are well established in the retina [7] but their existence remains unclear in the mouse visual cortex.
Recently, deep networks showed great potential for predicting neural activity from sensory input [8-15] and also in inferring novel functional properties [16-18]. These networks learn per-neuron vectors of parameters, which are interpreted as neuronal functional embeddings. There were several attempts to use these embeddings to reveal the underlying structure of neuronal population functions through unsupervised clustering [13, 19-21]. However, in none of these studies vell-separated clusters emerged, raising the question of whether distinet functional cell types exist among excitatory neurons in the mouse visual cortex. A central challenge is cluster consistency: How reliably are neurons grouped into the same cluster across different model runs? Clustering metrics such as the Adjusted Rand Index (ARI) [22], which evaluates cluster assignment agreement across different seeds or clustering methods and similarity metries between individual neurons' remained relatively low [13]. These low scores show that clustering results lack the stability and distinctiveness necessary to making strong claims about biological interpretations.
In this work, we incorporate an explicit clustering bias into the training of neuronal embeddings to improve the identifiability of functional cell types, One could view it as model-driven hypothesis testing: if clear functional cell types exist then such bias should improve the model performance. embeddings structure and/or cluster consistencies.
To improve the cluster separability of neuronal embeddings we took inspiration from Deep Embedding Clustering (DEC) [23] and introduced a new clustering loss, which combines updating clusters' locations and shapes along with learning feature representations. We measured the consistency of clustered features across models fitted on different seeds by computing ARI on their clustering results. Additionally, we examined how the clustering loss strength influenced models' performance.
Our contributions are
We adapted the DEC-loss [23] to allow for non-isotropic multivariate clusters of different sizes by learning a multivariate mixture model [24].
We improved cluster consistency while maintaining state-of-the-art predictive model performance.
We showed that our method generalizes well, improving cluster consistency across different species, visual areas, and model architectures.
2 Background and related work
Predictive models for visual cortex. In comparison to task-driven networks [25-28], pioneering data-driven population models [10, 29, 30] introduced the core-readout framework, which separates the stimulus-response functions of neurons into a shared nonlinear feature space (core) and per-neuree specific set of linear weights the readout. The core is shared among all neurons and outputs nonlinear set of basis functions spanning the feature space of the neuronal nonlinear input-output functions of dimension (height x width x feature channels). The early models were extended by including behavioral modulation [17, 31], latent brain state [11, or the perspective transformations of the eye [12]. The core architecture was improved by introducing biological biases such rotation-equivarinn core [14] to account for orientation selectivity in VI neurons [33], extending to dynamic models with video input [9. 12, 15, 18, 31] using transformer architectures [34].
Klindt et al. [35] introduced a factorized readout for each neuron, comprising a spatial mask Mn specifying its receptive field (RF) position and feature weights. This approach was refined by Lurz et al. [36], who proposed the Gaussian readout, replacing the full spatial mask with a pair of coordinates (z,, Mn) drawn from learned normal distribution. To predict the neuronal response the model computes the dot product between the neuron's 's weight vector (per-neuron embedding) and each feature map at the RF location. For later visual layers, like V4, the receptive field location is not necessarily fixed. Therefore, Pierzchlewicz et al. [37] introduced an attention readout, which indicates the most important feature locations for a neuron n depending on the input image.
While different readouts exist, few works have examined their consistency. Turishcheva et al. [13] showed that factorized readouts produced more consistent neuronal clusters than Gaussian readouts,

---

A
Feature map
Gaussian readout
B
E-step
M-step
KLYI1- N(
old
neuron
neuron
Core
point
Ks.YN N(H422
neuron N
neuron
model training
Cluster centers and shape updates
(embedding updates)
Figure 1: A: Model architecture: The model consists of neuronwise shared core outputting feature map of size (height x width x feature channels) and neuron specific Gaussian readouts. They consist of receptive field position and a weight vector. The RF position chooses the vector in the feature map which is then combined with the neuron's 's weight vector by a dot product to get the neuron's response. B: Clustering procedure: We re're clustering the readouts with an additional loss to incorporate the cluster bias into the features. We update the clustering parameters (cluster centers and scale matrices) with an EM step of mixture model in Alg.
despite lower predictive performance. They addressed this by introducing adaptive log- lognoorm regular- ization to balance model expressiveness and feature consistency. However, the ARI scores were still not high enough to claim distinct cell types. Moreover, their work involved a rotation-equivariant convolutional core and required post-hoc alignment procedure [38] to interpret the cluster structures.
Deep embedding clustering. Deep Embedding Clustering (DEC) [23] combines clustering with representation learning. introduced clustering loss that simultaneously drives learning the cluster centroids and encourages the feature representation to separate the clusters. After pretraining a deep autoencoder without the clustering loss, the cluster centers μı are initialized using k-means [39]. DEC then minimizes a Dullback-Leiblle (KL) divergence of soft cluster assignments Q and an auxiliary target distribution defined as follows:
(1 ( Qi
P Pij = Σy Ry
(2.1)
with fÚ = Σ Pij- (2.2)
Ey (1+11
The 91y8 the probabilities of sample 2, belonging cluster and represented by Student's t-distribution with unit scale and degree of freedom y being set to 1. The target distribution (Eq. (2.2)) is chosen such that it:
Original values 9: denote the soft assignment probability of a data point belonging to cluster j. Squaring 9j and then re-normalizing makes high-confidence assignments more dominant while further diminishing the influence of low-confidence ones. "emphasizes high-confidence data points." high (i dominates d / meaning that points strongly associated with a cluster contribute more to Pij-
"strengthens predictions."
"normalizes loss contribution of each centroid to prevent large clusters from distorting the hidden feature space." Without fj. larger clusters could dominate the feature space since they would contribute disproportionately to the loss. By dividing by fa the impact each cluster normalized, ensuring that smaller clusters are not overshadowed by larger ones.
Guo et al. [40] extended this approach by jointly optimizing the clustering objective and the autoencoder's reconstruction loss, enabling the model learn clusters while preserving the local structure of the feature space.
3 DECEMber - Deep Embedding Clustering via Expectation Maximization-baser refinement
DECEMber combines training a predictive model of neuronal responses with the learning clustered feature embeddings by optimizing loss inspired by Deep Embedding Clustering and iteratively updating cluster parameters using the EM algorithm. We now describe our approach (illustrated in Fig. 1, described in Alg. 1).
3

---

Algorithm Model Training with clustering loss
Inputs: Degrees of freedom v, clustering weight 3, core parameters 0, neuronal embeddings (readout) Z
Output: Parameters Hjj Ej, 0 and Z
Pretraining: Train the predictive model by optimizing model w.r.t. 0 and Z for m epochs
Initialize: Cluster centers Hú with k-means and diagonal scale matrix 2 as within-cluster variance for epoch to T do for minibatch b in dataset do (1) E-step (Expectation): Compute
1.1 Soft assignments qıj = (((ss115.-...'
(3.2)
1.2 Latent scales dij y+ "3 (5)
(3.3)
(2) M-step (Maximization): Update parameters
En 9:30ay
2.1 Update Hi 2 0ig
(3.4)
2.2 Update Σj = Ee (
(3.5)
Σx 9u
(3) Gradient step: Optimize predictive model parameters
3.1 Minimize T= mmoddll + BKT(QQI w.r.t 0, Z
92//1, and f, = =,,,0 q3
with Pij = Σ2
return μ4, E, 0, Z
Predictive model for visual cortex. We build on a state-of-the-art predictive model [8] for responses ri of neurons i = I,.,, N to visual stimuli E R xw xTxC Here H' and W' are height and width of the input, T' time if the input is video and C' is the amount of channels: C for grayscale or C'= 3 for RGB. For static visual input (images), T = and could be ignored. If behavior variables such as pupil size, locomotion speed, and changes pupil size are present, they are concatenated to the stimuli as channels [8]. The model combines a shared convolutional core & with neuron-specnt Gaussian readouts 1i (Fig. IA). The core outputs a feature space ((s) TR. Hx We denote the core's 's parameters by 0. The readout [36] v RR #xWK r> IR first selects the features from & at the neuron's 's receptive field location (xtg ys) using bilinear interpolation, which we write with slight abuse of notation as (())z.m, E IR* k resulting in feature vector o; R It then computes the predicted neuronal response f; = IT oi where E RK is the neuron- -sprcicc readout weight (its functional embedding), overall
(s(s) v0((())) = =B)"
(3.1)
EM step to update cluster parameters. Instead of directly learning the cluster centroids via gradient descent, we updated them after each batch using the EM algorithm applied the Student's I-mixture model, (z;; θ) = Σj=1 Si (2;; My; Ej, v), [24] where degree of freedom controls the probability mass in the tails (if + so the t-distribution becomes Gaussian). The density of the multivariate Student's t-distribution is:
K
[+ [ 1,)" (2s mb))
felzi (g, v)
(3.6)
Ez
N(4 Ho {Σ,) Gamma u 11 3) du.
(3.7)
with the latter being the so-called shape-rate form of the t-distribution [41]. This interpretation is useful because introducing the Gamma distribuud latent variable u allows closed-form M-step updates for μı and E,, whereas direct likelihood optimization in t-mixture model does not generally admit closed-form solutions.
The full procedure is summarized in Alg. and alternates between: (1) E-Step: Compute soft cluster assignments jiy (Eq. (3.2)) - the probability of feature i belonging to cluster j and the latent scaling factors Usy (Eq. (3.3)). (2) M-Step: Update cluster means μ4 (Eq, (3.4) and (diagonal) scale matrices

---

Σj (Eq. (3.5)). (3) Gradiet step: Update the parameters of the core and readout via one iteration of stochastic gradient descent.
Clustering loss on readout weights. To encourage a well-clustered structure on the neuron-specific readout weights, we augment the standard model loss with clustering objective. Specifically, we minimize the KL divergence between soft cluster assignments Qiş (Eq. (3.2)) and target distributions Pij (Eq. (2.2)):
N log
Teluster KT,(Q(2)) =
Ps
(3.8)
Oij
This auxiliary loss encourages the embeddings to form J distinct clusters.
Xie et [23] use a pretrained autoencoder with well-separated embeddings and model soft cluster assignments using Student's t-distribution with fixed unit scale. However, this setup is too con- strained for our regression setting. where the mean and scale of the embeddings 2; are restricted by the regression loss. By adopting TMM, where each cluster is characterized by both its center μj and scale matrix Σj. we allow the clustering structure to adapt more flexibly during training.
Experiments
Clustering loss hyperparameters. For the clustering loss, we fixed the degrees of freedom to V = 2.1, just above the threshold where the variance - becomes defined (only for y > 2). To balance model ficxibility and robustness, we allowed each cluster to have its own diagonal scale matrix Σ.j which alloed for different variances per embedding dimension while preventing overfitting. For each dataset, we adjusted the clustering strength ,3 E R such that it is the same order of magnitude the model loss at initialization.
Pretraining and cluster initialization. Before adding the clustering loss, we pretrained the baseline model for epochs, such that the model could already predict the responses reasonably well. We explored m = 5, ....., ..... 40 to assess how the length of pretraining (PE) affected our results. We followed Turishcheva et al. [13] for the pretraining procedure, minimizing the following loss:
Lioacll = -p + Lrog =
1 N
ΣΣ (fa ra log fa) Liveg
(4.1)
where Lp the Poisson loss that aligned per-image ( L,..., model predictions fu with observed neuronal responses ra since neuron's firing rates follow a Poisson process [42], and Lieg is the adaptive regularizer that was shown to result in improved embedding consistency [13].
After pretraining, we initialized the cluster centroids μ4, with k-means [39] and the diagonal scale matrices Σ, as the Letuster as in Eq. (3.8) and scaled with B.
variances. We continued training using Lmmode! + BLehuster with
Evaluation of model performance. Building on previous research [8, 14, 16, 34, 43, 44], we evaluated the model's 's predictive performance by computing the Pearson correlation (across images in the test set) between the measured and predicted neural responses, averaged across neurons.
Evaluation of embedding consistency. We wanted to assess the relative structure of the embedding space: Do the same groups of neurons consistently cluster together across models with different initial conditions? To quantify this notion, took DECEMber's cluster assignments and measured how often neuron pairs are assigned to the same group using the Adjusted Rand Index (ARI) [22], which quantifies the similarity between two clustering assignments, X and Y. The ARI remains unchanged under permutations of cluster labels. ARI equals one if and only the two partitions are identical and it equals zero when the partitions agreement no better than random.
To compare DECEMber with a baseline, we extracted neuronal embeddings from the fully converged default model and fitted Gaussian Mixture Models (GMMs) using the same number of clusters as DECEMber, diagonal covariance, and regularization 10-6 6 We then computed the ARI across three GMM partitions from bascline models initialized with different seeds, using fixed GMM seed.
5

---

A
B
C
6
16
12
Cluster
Cluster2
Weights 21
Epochs
Figure 2: A: PDF of 21 (blue) and 22 (orange) of the underlying true normal distribution and distribution with unit scale estimated by DEC-loss. The two I-distributions as well as the normal distributions are highly overlapping. B+C t-SNE projection of toy data after training with DEC loss (B) DECEMber (C). We first pretrain simple linear regression model by minimizing MSE-loss for 30 epochs. Then we are training jointly with MSE and KL loss. B: Visualization of clustering with the DEC loss. All features are assigned to one cluster. Clustered structure is still visible. C: Clustering the learned features with DECEMber. All features get assigned to the right cluster. D: Norms of learned cluster centers μك and μig for the DEC-loss. It is clearly visible that the cluster centers collapse after only a few iterations whereas updated cluster centers via DECEMber 1i and μig are converging towards their true mean μ1 and μ2, with || = v128 x 11.3 and ||æ2||2 = v128. 1.5 se 17.
Visualization. To visualize the neuronal embeddings, we employed t-SNE [45], following the guidelines of [46]. Specifically, we set the perplexity to N /000, the learning rate to and early exaggeration to N.10. To be comparable with prior work [13, 20]. we randomly sample 2,000 neurons from each of the seven mice in the dataset and used the same neurons across all visualizations.
5 Results
DEC-loss needs learned scale: toy example illustration. To assess whether the DEC-loss provides a useful clustering when applied to model weights that are restricted by a regression loss instead of autoencoder embeddings,
constructed simple toy example consisting of linear neurons whose responses are given as Yiy = zi'az + Eij. where 2, are the neurons" weights, Iy the stimuli and Eij Gaussian noise. We generated 1100 white noise stimuli, each the form 2i IR-28 with Ty ~ (0, 1). We generated 1000 neuronal embeddings z; such that they would naturally form 2 clusters. We created 300 weights of the form 2; ~ (N122))) J J128) and 700 z; ~ (V.5. 12s). sampled Gaussian noise around 0 with variance that
To finally get the neuronal responses matches a chosen signal to noise ratio (SNR). In the here shown example we used SNR=2. detailed discussion about SNR can be found in the Appendix A.I.
We pretrained a linear regression model on this data for 30 epochs by minimizing the MSE of predicted and learned responses. After that we continued training, by jointly minimizing the KL divergence on the learned centers and the MSE using the DEC-loss versus DECEMber. We used early stopping as well as a learning rate scheduler. In theory the model should learn the weights of the two clusters centered at μp (1,.... 1) E 128 and μ22 (1.5.... 1.5) E 128 and assign 300 neurons to cluster 1 and 700 neurons to cluster 2.
We found that the vanilla DEC loss fails identify clusters even in this simple toy example, where cluster weights are well-separable after pretraining. This is because DEC employs a Student's s-t distribution with fixed unit scale parameter for all clusters, which is too large given how close the two weight distributions of clusters and 2 are (Fig. 2A). As the magnitude of the weights is given by the regression problem, the scale of the distribution needs to be adjusted appropriately during training. When this is not done (as in vanilla DEC), the cluster centroids fá and fig rapidly collapse to a single point after only few iterations (Fig. 2D). Even though this toy example the true covariance of the clusters has unit scale after the short pretraining phase the within cluster variance is much lower leading to collapse the DEC-centers. This happens because there exists degenerate optimum of the KL divergence: If all cluster centers equal μl LJ c, plugging them into Eq. (2.1)

---

A
1.0
True 10 (o00) midget
0.
0.8
0.8
0.6
0.8
0.4
Iuge off
CMN
K-mnans
outs
Predeted
Number of Cusstrn
Figure 3: RI across seeds for GMM, k-means and DECEMber. B: DECEMber predictions. Pretraining length: 25 epochs. Corresponding test correlation: 0.805 + 0.068 (std). C: ARI across 3 seeds for DECEMber over different number of clusters. ARI shows a clear peak at 4 clusters.
[1+1====[]2+-]] - (1+1//2
4. = (, 2 1) (F117 = gives us P.j =
as Jj Ly, (Eq. (2.2)) which means the KI. divergence KT.(PIIQ) = O, which of course is not meaningful solution. In DEC, this minimum not usually found in practice because clusters are initialized with sufficient separation after pretraining.
To avoid this collapse, instead used a multivariate E-distribution with (diagonal) scale matrices 2; for each cluster, updating both position and scale with an EM step (Alg. 1). On the same toy example, this approach succeeded in finding good cluster separation (Fig. 2C), and the cluster centers μor and fi converged towards the true underlying locations (Fig. 2D).
DECEMber accurately classifies retinal ganglion cells and outperforms conventional clustering approaches. To check whether DECEMber works on real data, we applied it on marmoset retinal ganglion cells (RGCs) where the existence of discrete cell types is well established [47]. We used data from two male marmoset retinas published by Sridhar et al. [48], where the neural activity was recorded using a micro-electrode array while presenting grayscale natural movies.
As we observed substantial differences between the two retinas' temporal response features (poten- tially due to temperature variation [49]), we followed Vystrčilová et al. [15] and trained separate model for each retina to avoid clustering by retina. We trained the model on all reliably responding cells (N- = 235). However, not all of them corresponded to known primate RGC type and thus were not assigned cell type label. When evaluating DECEMber, we only used the labeled cells. The first retina contained responses of four cell types (78 midget-OFF-like cells, 40 parasol-OFF-like cells, 5 ON-like cells, and 22 large-OFF cells, further details on classification are in App. A.2).
We trained baseline version of CNN model [15] separately without our proposed clustering loss, using three random seeds. Baseline clustering was then performed post hoc using GMM and k-means. Subsequently, we continued training the model with our clustering loss, again using three seeds.
Applied to marmoset retina data, DECEMber achieved reliable classification across cell types, with high clustering consistency (ARI 0.96 +0.01) for 4 clusters while maintaining high predictive performance of 00.1 1 +0.07. It surpassed both GMM and k-means, (Fig. 3A) effectively separating even highly unbalanced groups, such the ON-cells, resulting in almost perfect confusion matrix (Fig. 3B). In contrast to k-means, which is sensitive to initialization (Suppl. Fig. 12), DECEMber exhibited greater robustness and aligned more closely with the ground truth labels while the model retained high predictive accuracy. When we tested DECEMber for different amount of clusters we see clear peak in ARI the true number of clusters 4 (Fig. 3C).
DECEMber enhances local structure among embeddings and hurts performance once it domi- nates the overall model loss. Next, we asked if DECEMber could help to find functional cell types in visual area without clear known cluster separation. We used SENSORIUM 2022 dataset and baseline architecture train model to predict responses of mouse primary visual cortex grayscale images. for seven mice (more detail on data in App. A.5). Previous work [13, 20] observed density modes in the functional embeddings of mouse VI neurons (Fig. 4A) and hypothesized that these modes may correspond to discrete functional cell types. To investigate whether these patterns reflect true discrete and distinct cell types, we applied DECEMber (Alg. 1), hypothesizing that such types exist, DECEMber would help to separate them. As the number of excitatory cell types in the mouse
7

---

A
C
D
E
0.40
G 0:40 10a
102
10'
H 0.45
10
164
207
0.40
8 10
B 10
Tost corraation
8:10"
10'
0.38
0.30
A 8-10"
0.37
0.30
10
0.96
0.37
0.20
0.35
0:36
0.10
10'
10
15
25
0.14
0.16
0.18
0.20
0.22
clusters
Number clusters
Figure 4: A: I-SNE of baseline model without clustering loss. B-E: I-SNE projections of model with clustering bias for different multipliers 3 and tuned learning rates (Ir). All models use 15 clusters, 10 and Ir 0.008. C: 3 10a and Ir 0.008. D 10 and Ir = 0.007. E: 8 = 102 and Ir = 0.003. F: Corresponding model performances of the models with clustering bias and differing weights B, tuned learning rates described in B-D, G: Predictive performance vs. ARI for different 3. We can see that ARI increases with the increase of B until the model performance drops. After that ARI doesn't 't increase further. Here we fixed the number of clusters to 15. H: ARI for different clustering weights 3 with optimal learning rates, PE = 10.
PE 10 and seed 100. B:
A
B
PELS
PE:20
0.40
0:40
0-11
0:40
PED9
PE:40
PE:30
0.38
0.30
0.36
5:10"
033
PE 10
0 0:25
0.36
P:10"
037
PE 20
0.32
P:10"
0:15
0.36
0.2%
Baselne
0.002
0.005
8,006
0.007
5.008
0.35
0.603
0.004
0.505
0.006
0.007
0.008
10
40
50
Figure 5: A-B: Learning rate tuning for 3 (A) and length of pretraining (B). We fixed amount of clusters to 15. If the learning rate is too high the clustering loss starts oscillating due to learning rate scheduling leading
massive drop performance. C ARI for different number of pretraining epochs vs baseline model. For each number of pretraining epochs we used the optimal learning rate and set 3 10 for all experiments. All settings of DECEMber have better cluster structures after 15 clusters at latest. It is visible that 10 pretraining epochs generate the best clustered embeddings.
visual cortex remains unclear, with estimates ranging from 20 to 50 [20, 50], considered range of - 5, ..., 60 in increments of 5.
We tested wide range of loss strengths 3, to find the optimal value to balance the clustering nd the model loss. As 3 increases, t-SNE vizualization suggests improved qualitative separation of clusters in the embedding space (Fig. 4B-E). However, this comes cost: when the clustering loss becomes dominant, the model's predictive performance drops significantly (Fig. 4F). This made us question if the qualitative structure in t-SNE plots is meaningful. To answer this question, we quantified clustering consistency using ARI between three model fits with different seeds and found that clustering consistency noticeably improved compared to the GMM baseline (Fig, 4H).
We see the ARI improvement as long as 3 does not hurt performance (3 < 10 Fig. 4G. However, once /3 > 104, performance starts suffering (Fig. 4F) and the ARI does not improve anymore (Fig. 4G), suggesting that the qualitative structure is created by removing functionally relevant heterogeneity between neurons. While ARI values double compared to the baseline model, there is no clear peak around a certain number of clusters. We would expect ART to peak noticeably at the "true" number of clusters as shown for the retina ganglion cells (Fig. 3C) if such structure existed. This suggests that mouse VI likely lacks discrete functional cell types. Still, the clear improvement indicates meaningful local structure in functional embeddings.
8

---

Consistency of embeddings depends on length of pretraining. To validate our conclusions that mouse VI lacks discrete functional cell types, we performed extensive tuning of DECEMber by using different numbers of pretraining epochs before turning on the clustering loss, different clustering strengths 8 and tuned learning rates to optimize model predictive performance.
Across all settings, DECEMber achieved higher ARI scores than the baseline, indicating better consistency of embeddings (Fig. 4H, Fig. 5C).
0.396
coneiia Comelatin 0.392
0.394
B: 10*
We found that the optimal learning rate varied depending the number of pretraining epochs (Fig. 5A), and also depended on the clustering loss strength 3 (Fig. 5B). Importantly, the choice of the number of pretraining epochs had minimal effect on the overall predictive performance if the learning rate optimally chosen, with differences staying within the standard deviation across runs. We tuned on the validation set (Fig. 5B), and checked the test set (Fig. 6). However, we observed a distinct peak in ARI for 10 pretraining epochs in the case of the mouse visual cortex (Fig. 5C). range of cluster settings, we did not observe sharp maximum at any specific cluster count.
$.000
0.30n
20 Fretralning epach
10
36
Figure 6: The choice of pretrain epoch doesn't 't influence perfor- mance when we' 'e using an op- timal learning rate.
While the ARI improved across
DECEMber improves embeddings across different datasets and model architectures. To ensure that DECEMber generalizes across architectures, modalities, and species, we additionally tested on data from the mouse retina and macaque visual cortex area V4. We did not extensively tune hyperparameters, we only decreased the learning rate (Ir) to stabilize the baseline model training for both datasets, and set /3 as described in Sec. 4 (exact settings in App. 15). More extensive tuning of the Ir, 39 or the number of pretraining epochs can lead to better results. For both datasets we preserved the performance of the original models (App. A.9).
For the mouse retina we used both the data and the models from Höfling et al. [18]. As for the marmoset retina, we trained separate model for each retina account for the temperature differences between retinas. Given the limited availability of cell-type labels, we included all cells in our analysis and evaluated cluster consistency across varying numbers of clusters. For details on dataset averaging and per-dataset analysis, see App. A.10.
For macaque V4 data we used spiking extracellular multi- electrode recorded responses of neurons to gray-scale nat- ural images shown to awake macaque monkeys [51] and the model from Pierzchlewicz
A
baseline
B
0.6
PE:S
0.5
PE:10
ARI
0.4
2E:20
0.3
0.2
al. [37], which had a differ- readout architecture- tention readout instead of the previously used Gaussian read- out. We trained the model on 1000 cells and measured the ARI across three model seeds. The embedding consistency doubled using our method (Fig. 7B). This shows that DECEMber is robust not only across different data modalities but also across architectures. For more analysis of the monkey dataset see App. A.11.
0.1
Number of clusters
Figure 7: ARI on A mouse retina [18], weighted across six models. B monkeys V4 [51].
6 Discussion
In this work we introduced DECEMber, an additional training loss with explicit clustering bias for predictive models of neuronal responses. DECEMber enhances cluster consistency, while keeping state-of-the-arr predictive performance. It robust across different data modalities (electrophysiology and calcium imaging), species (mice, primates) and visual areas (retina, V1, V4). We also showed that DECEMber is robust across both static (mouse VI, macaque V4) and dynamic (retinas) cores and multiple readout architectures the Gaussian readout and the attention readout.

---

We see DECEMber as model-driven hypothesis test: if clear functional cell types exist, then incorporating this bias should improve model performance and/or the embedding structure, which we measure as cluster consistency. While improvements are observed across datasets and architectures, our main focus was mouse VI, where the existence of discrete excitatory cell types remains debated. Our results support the idea that excitatory neurons in mouse visual cortex form functional con- tinuum rather than discrete clusters. This finding is consistent with recent work studying different modalities by Weiler et al. [52]. Tong et al. [19], and Weis et al. [6], who independently found no clear boundaries in morphological or electrophysiological features. In line with Zeng [1], we argue that future efforts to define mouse VI cell types should emphasize multi-modality combining functional, morphological, and genetic data. This approach has proven fruitful in the retina, where functional types alone are coarser than those derived from multiple modalities [7, 21].
Given the generality of our clustering loss, which model-agnostic and not tied to specific architecture, we believe it holds promise for use in multi-modal models aiming to define cell types or in broader unsupervised representation learning contexts.
Connection to other works learning neuronal embeddings. There are other works [53-59], which all learn neuronal embeddings in some way, but none them explicitly enforce or optimize for clustering, which is the main goal of DECEMber. Specifically, Nemo [53], NeurPIR [54] and NuCLR [55] are contrastive methods, DECEMber not. Nemo [53] and NeurPIR [54] embeddings functions of imput (current activity, autocorrelogram). for DECEMber neuronal embeddings are time- and input- invariant weights of the predictive model (they embed the neuron's 's full input-output function). Nemo [53], NeuPRINT [56], NetFormer [57] do not model visual stimuli. NeuPRINT, NetFormer, POYO [58], NEDs [59] have time-invariant model weights, but both predict neuronal activity based masked previous neuronal activity while the regression model in our paper predicts neuronal activity based on visual stimuli. It might be interesting to integrate DECEMber clustering loss with these works [56-59] but we leave it for future work.
Limitations. DECEMber requires a predefined number of clusters. When this is unknown, multiple runs with varying cluster counts and seeds are necessary in combination with an evaluation ARI-like metric to identify the optimal configuration. This increases the computational cost. Choosing an appropriate clustering strength 3 is also crucial for balancing ARI and model performance and further work is needed to determine the optimal pretraining duration.
Moreover, operating in nigh-dimensional feature spaces introduces additional challenge: the cluster covariance matrices can become large and ill-conditioned, with tiny diagonal values, hitting the limits of numerical stability. We address this issue by clamping small values, though this solution is heuristic rather than principled. Furthermore, high-dimensional settings require a sufficient number of data points to prevent overfitting of the scale matrices. Additionally, we currently assume t-distributed feature space via a t-mixture model, but this can be adjusted if a more suitable prior over the embeddings is known.
7
Acknowledgments
We thank Suhas Shrinivasan, Max F. Burg, Larissa Höfling, Thomas Zenkel, Konstantin F Willeke, Fabian Sinz.
This work was supported by the Deutsche Forschungsgemeinschatt (DFG, German Research Foun- dation) - project IDs 432680300 (SFB 1456, project B05) and 515774656 and by European Research Council (HRC) under the European Union's 's Horizon Europe research and innovation programme (Grant agreement No. 101041669). AST acknowledges support from National Institute of Mental Health and National Institute of Neurological Disorders And Stroke under Award Number U19MHI 14830 and National Eye Institute award numbers R01 EY026927 and Core Grant for Vision Research T32-EY-002520-37, We gratefully acknowledge the computing time granted by the Re- source Allocation Board and provided on the supercomputer Emmy/Grete NHR-Nord @ Göötingen as part of the NHR infrastructure. The calculations for this research were conducted with computing resources under the projects nim00010 and nim00012.
10

---

References
[1] Hongkui Zeng. "What is cell type and how to define it?" In: Cell 185.15 (2022), pp. 2739- 2755.
[2] Javier DeFelipe al. "New insights into the classification and nomenclature of cortical GABAergic interneurons". In: Nature Reviews Neuroscience 14.3 (2013), pp. 202-216.
[3] Marcel Oberlaender et al. "Cell type- aipe-ppecfiic three-dimensional structure of thalamocortical circuits a column rat vibrissal cortex". In: Cerebral cortex 22. (2012), pp. 2375-2391.
[4] Henry Markram et al. "Reconstruction and simulation of neocortical microcircuitry". In: Cell
163.2 (2015), pp. 456-492.
[5] Federico Scala al. "Layer 4 of mouse neocortex differs cell types and circuit organization
between sensory areas".. In: Nature communications 10.1 (2019), 4174.
[6] Marissa A. Weis al. "An Unsupervised Map of Excitatory Neuron Dendritic Morphology in the Mouse Visual Cortex". In: Nature Communications 16.1 (Apr. 2025), p. 3361. ISSN: 2041-1723, DOI: 10. 10110038//44 441467714611/1 112111177-2-----0 020-0005555555 (Visited on 04/10/2025).
[7] Tom Baden et al. "The functional diversity of retinal ganglion cells in the mouse". In: Nature 529.7586 (2016), pp. 345-350.
[8] Konstantin F. Willeke et al. The Sensorium competition on predicting large-scale mouse primary visual cortex activity. 2022. arXiv: 2206 08666 [q-bio. NC]. URL: https:
arxiv. org/ abs/20bs..22...////// 08666.
[9] Polina Turishcheva al. "The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos". In: ArXiv (2024), arXiv-2305.
[10] Ján Antolik et al. "Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses to Natural Scenes". In: PLOS Computational Biology 12.6 (June 2016), pp. 1-22. DOT: 10. 1371 / jurnal. pectnnaalpbb........ 1004927. URL. https: /ooi........ org 10 / journal. pcbi 1004977.
[11] Finn Schmidt et al. Modeling dynamic neural activity by combining naturalistic video stimuli and stimulus independent latent factors. 2024. URL: https pettps//neurripe: cc/ /virtull/
2024/101471
[12] Eric Y. Wang et al. "Towards Foundation Model of the Mouse Visual Cortex". In: bioRxiv (2023). DOr: 10. 1101 / 2023. 03. 533548. eprint: https / /www. biorxiv org/ Content/early 2023 /33 03/ / 2023 03 21 533548. full. pdf. URl: https: //wp:://www. biorxiv http://ootett/tteeeeeeeeeeeee wttp://ttttetette/////////////// 240222233332222 03. 23..2.5533 23.5544...
[13] Polina Turishcheva et al. Reproducibility of predictive networks for mouse visual cortex. 2024.
arXiv: 2406 12625 [q-bio. URL: htt.t:tttss///rttt org/ htgg//bs/240066 /t/p://rr.....rrr/aa///000.... 12625.
[14] Alexander Ecker al. "A rolation-equivaariant
cta.:peccciiiiiattooooooolllaaaa convolutional neural network model of primary p9ster/922.
visual cortex". In: (2019). URL: URt::httpps//kkcc.cc/vvrttttt
[15] Michaela Vystrčilová et al. "Convolutional neural network models of the primate retina reveal
adaptation to natural stimulus statistics", In: bioRxiv (2024). Dor: 10. 1111 03 06. 583740. eprint: https: //ww///ww//// biorxiv http://.rrrooototteeeeeeeeeo
content/ /aaeett/aaaaae pearteett/errlyy/22///////////
03.
waapttteaaarlrrreeeee 06 583740300.. full. fdf.. URL: https //ww://wwww./ biorxiv www.rrvww.rrr.ootttee///////////
09/2024. 03. 06. 5887440.
[16] Edgar Y Walker al. "Inception loops discover what excites neurons most using deep predictive models". In: Nature neuroscience 22. 12 (2019), pp. 2060-2065.
[17] Katrin Franke et al. "State-dependent pupil dilation rapidly shifts visual feature selectivity".
In: Nature 610.7930 (2022), pp. 128-134.
[18] Larissa Höfling et al. "A chromatic feature detector in the retina signals visual context changes".
In: Elife 13 (2024), e86860.
[19]
Rudi Tong al. "The feature landscape of visual cortex". In: bioRxiv (2023). DOI: 1101/ 2023. 11. 03 565500. eprint: https: // /www. biw.xiio.v..... http///otteeeeel/////oeoe/ 05 / 11. 03. 565500 full pdf, URL: https : www biorxiv org/ content/
early/2023/ 1 20/02////10////2//2 2202..2221.111 11. 03.03..... 565500.
[20] Ivan Ustyuzhaninov et al. "Digital twin reveals combinatorial code of non-linear computations in the mouse primary visual cortex". In: bioRxiv (2022), pp. 2022-02.
[21] Max F. Burg et al. Most discriminative stimuli for functional cell type clustering. 2024, URL:
https: //iclr. cc/virtual/2022 http://0022/ppttt119922/ 19993..

---

[22] Lawrence Hubert and Phipps Arabie. "Comparing partitions". In: Journal of classification 2 (1985), pp. 198-218.
[23]
Junyuan Xie, Ross Girshick, and Ali Farhadi. "Unsupervised Deep Embedding for Clustering Analysis". In: Proceedings of The 33rd International Conference on Machine Learning. Ed. by Maria Florina Balcan and Kilian Q. Weinberger. Vol. 48. Proceedings of Machine Learning Research. New York, New York, USA: PMLR, 20-22 Jun 2016, pp. 478 378-487. URL: https: / //proceedings.m.......... press v48/xieb166..6 html.
[24] Geoffrey J. McLachlan and David Peel. "Robust mixture modelling using the distribution". In: Statistical Science 15.1 (2000), pp. 1-19. URL: https : / /ttppppppl....//////// smp...................... uq. edu. au/
feoffcclcclaan aaaaaanmmmmmaaa_ htlppppps._______10p11_0pp_00p .d0.
[25] Daniel LK Yamins et al. "Performance-optinn
hierarchical models predict neural responses in higher visual cortex". " In: Proceedings of the national academy of sciences 111.23 (2014),
pp. 8619-8624.
[26] Charles F Cadieu et al. "Deep neural networks rival the representation of primate IT cortex for core visual object recognition". In: PLoS computational biology 10.12 (2014), c1003963.
[27] Santiago A Cadena et al. "Deep convolutional models improve predictions of macaque V1 responses to natural images". In: PLoS computational biology 15.4 (2019), e1006897.
[28] Galen Pogoncheff, Jacob Granley, and Michael Beyeler. "Explaining V1 properties with a biologically constrained deep learning architecture". In: Advances in Neural Information Processing Systems 36 (2023), pp. 13908-13930.
[29] Eleanor Batty et al. "Multilayer Recurrent Network Models of Primate Retinal Ganglion Cell Responses". In: International Conference on Learning Representations. 2016. URL: https: / //ppi.pp....// semanti www.olarccca.......om....oaa.... www.s/ollrrrrrrooooooeooeoo Corpus 390002941..
[30] Lane T. McIntosh et al. "Deep Learning Models of the Retinal Response to Natural Scenes". In: Advances in Neural Information Processing Systems. Vol. 29. Curran Associates, Inc.,
2016, pp.
[31] Fabian Sinz al. "Stimulus domain transfer in recurrent models for large scale cortical population prediction on video". In: Advances in neural information processing systems 31 (2018).
[32] Mohammad Bashiri et al. "A flow-based latent state generative model of neural popula- tion responses to natural images". In: Advances in Neural Information Processing Sys- tems. Ed. by M. Ranzato et al. Vol. 34. Curran Associates, Inc., 2021, pp. 15801-15815. URL: https // proceedings neurips. cc paper files / paper 2021 / file/ / 04522292282222222233366666622222 pdf.
[33] Andrew Y. Tan et al. "Orientation Selectivity of Synaptic Input to Neurons in Mouse and Cat Primary Visual Cortex". In: Journal of Neuroscience 31.34 (2011). Erratum in: J Neurosci. 2011 Oct 12:31(41):1483) pp. 12339-12350 12350. DOI: 10. 111111//2ERRRA JNEUROSCI 2039- 11. 2011. Bryan Li et al. "VIt: large-scale mouse v1 response prediction using vision transformer". In: arXiv preprint arXiv:
[34]
(2023).
[35] David Klindt et al. "Neural system identification for large populations separating "what" and "where""" In: Advances in neural information processing systems 30 (2017).
[36]
Lurz et al. "Generalization in data-driven models of primary visual cortex". In: bioRxiv (2020). DOI: 10. 1111/1010001111 10. 05. 326256. eprint: https: //www
Konstantin-Klenenn
biorxiv. content/ oreexivvoorg/ootettt///oo eeaeeelllly//// 10 2020 10 326256 full pdf. URL: 11//11100///0// .0../.020..110 10. 05.336256666
https://www biorxiv...vi.ii.....
[37] Pawel Pierzchlewicz al. "Energy guided diffusion for generating neurally exciting images".
In: Advances in Neural Information Processing Systems 36 (2023), pp. 32574-32601.
[38] Ivan Ustyuzhaninov et al. "Rotation-invariant clustering of neuronal responses in primary visual cortex". In: International Conference on Learning Representations. 2019.
[39] James MacQueen. "Some Methods for Classification and Analysis of Multivariate Obser- vations". In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. 1967, pp. 281-297.
[40] Xifeng Guo et al. "Improved deep embedded clustering with local structure preservation". In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. 1JCAI' Melbourne, Australia: AAAI Press, 2017, pp. 1753-1759. ISBN: 9780999241100 103.
[41] Geoffrey J
and David Peel. Finite mixture models. John Wiley & Sons, 2000.
12

---

[42] Peter Dayan and Laurence F. Abbott. Theoretical Neuroscience: Computational and Mathe- matical Modeling of Neural Systems. MIT Press, 2005.
[43] Brett Vintch, Anthony Movshon, and Eero Simoncelli. "A convolutional subunit model for neuronal responses in macaque V1". In: Journal of Neuroscience 35.44 (2015), pp- 14829- 14841.
[44] Max F Burg et al. "Learning divisive normalization in primary visual cortex". In: PLoS computational biology 17.6 (2021), e1009028.
[45] Laurens van der Maaten and Geoffrey Hinton. "Visualizing data using t-SNE". In: Journal of muchine learning research 9.Nov (2008), pp. 2579-2605.
[46] George C. Linderman and Stefan Steinerberger. Clustering with I-SNE, provably. 2019. DOI:
10. 1137 011///1112111331 URL: hattss:///oi. org/ 1t///.0.11113 1137 181121131...
[47] Rania A Masri et al. "Survey of retinal ganglion cell morphology in marmoset". In: Journal of Comparative Neurology 527.1 (2019), pp. 258-258.
[48] Shashwat Sridhar and Tim Gollisch. Dataset Marmoset retinal ganglion cell responses naturalistic movies and spatiotemporal white noise. Apr. 2025. DOI: 10. 12751/ NODE. 3DFITI. URL: https: // doi. gin. g node org / 10 12751 / g node. 3dfiti (visited on 05/16/2025).
[49] Zhijian Zhao al. "The temporal structure of the inner retina at single glance". In: Scientific reports 10.1 (2020), p. 4399.
[50] Nathan W Gouwens et al. "Classification of electrophysiological and morphological neuron types in the mouse visual cortex". . In: Nature neuroscience 22.7 (2019), pp. 1182-1195.
[51] KF Willeke et al. "Deep learning-drii characterization of single cell tuning in primate visual area V4 unveils topological organization. bioRxiv. 2023". In: doi. org/10. wtg/10.1101/2223.05 12 (2023).
[52] Sebastian Weiler et al. "Functional and structural features of L,23 pyramidal cells continuously covary with pial depth mouse visual cortex". In: Cerebral Cortex 33.7 (2023), pp. 3715-
3733. DOI: 10. 1093/cercor/ ctt.////roc/////////e /bhoc333.
[53] Han Yu et al. "In vivo cell-type and brain region classification via multimodal contrastive learning". In: The Thirteenth International Conference on Learning Representations. 2025.
URL.: https: http///ppeeeeeeeeeeeeeeee feett//formm/ddd101111001
[54] Wei Wu al. "Neuron Platonic Intrinsic Representation From Dynamics Using Contrastive Learning". In: The Thirteenth International Conference on Learning Representations. 2025.
URL: https: /taps://oppae http:///ppeeeeeee///m www. wet.t/orrraiieeeaaaaaFFaa0
[55] Vinam Arora et al. "Know Thyself by Knowing Others: Learning Neuron Identity from Popu- lation Context". In: The Thirty-ninth Annual Conference on Neural Information Processing
Systems. 2025. URL: https: http///ppprreeeeeeeeeeee/eeeeeee net ffrtrmrrm id 2tt22t3333BB
[56] Lu Mi et al. "Learning Invarinn Representations for Individual Neurons from Population Dynamics". In: Thirty-seventh Conference on Neural Information Processing Systems. 2023.
URL: https: / /oppnreviiiwwwccmmooooo net footfforrue ftr./oormmm//// id=nn3331111111 6Xmnx.
[57] Ziyu Lu et al. "NetFormer: An interpretable model for recovering dynamical connectivity in neuronal population dynamies", In: The Thirteenth International Conference on Learning Representattinss 2025. URL: /estts::/openree
net/ tuee/foruee forum?dd-bbTTiWksss/
[58] Mehdi Azabou al. "A Unified, Scalable Framework for Neural Population Decoding". In: Thirty-seventh Conference on Neural Information Processing Systems. 2023. URL: https: http//orrmmddss//0sssrrree
/ openrevies
[59] Yizi Zhang et al. "Neural Encoding and Decoding Scale". In: Forty-second International Conference on Machine Learning. 2025. URL: https: /openrevieeeeeeee net/ forum/ormmm id= vOdz3zhSCj.
[60] Shashwat Sridhar et al. "Modeling spatial contrast sensitivity in responses of primate retinal ganglion cells to natural movies". In: bioRxiv (2025). DoI: 10. 1101//024222...... 03 05 583449. eprint: https: : / www. biorxiv. . org / content / eerrlyyy22255 / 04 / 09/ 2024 03 05. 583449. , full. full.pdf.. URL: https: //wp///ww...// biorxiv org wtw.://www..rreooooe/eer/ar/////
2024. 03. 05.05...5.. 583449.
[61] Edward B Fowlkes and Colin L Mallows. "A method for comparing two hierarchical cluster- ings". In: Journal of the American statistical asxociation 78.383 (1983), pp. 553-569.
13

---

[62] Andrew Rosenberg and Julia Hirschberg. "V-measure: A conditional entropy-bart -bssed exter- nal cluster evaluation measure". In: Proceedings of the 2007 joint conference on empiri- cal methods in natural language processing and computational natural language learning (EMNL.--CNNLL). 2007, pp. 410-420.
[63] Alexander Strehl and Joydeep Ghosh. "Cluster ensembles a knowledge reuse framework for combining multiple partitions" In: Journal of machine learning research 3.Dec (2002),
pp. 583-617.
14

---

A Appendix
A.1 Signal to noise ratio for toy example
J.S-Ae CATEATS
Tot Carcaatonn
DEC
Dialarce
66-
02
0.0
0.003 0.51
3.5
0.5 SKW
10
0.001
5.5
0.05
0/
0.001
0.3 0.5
SAI
SNI
Figure 8: A: Clustering accuracy for predicted vs. ground-truth label for DEC vs. DECEMber for different Signal-to-Noise ratios. B: Predictive performance on test set. The distance of the norms of the 2 cluster centroids || - /|//|| for DEC vs. DECEMber. The ground truth distance is V128 1.5
V128 5.66.
Data in mouse VI is recorded using 2-photon calcium imaging which is known to be noisy. To investigate how robust DECEMber is towords noise we varied the SNR in the toy example setting. As described in the toy example generated clean responses as the dot product of the stimuli and the created network weights. To simulate noisy observations, we added Gaussian noise independently for each neuron and stimulus but with the same variance. The noisy responses are thus given by Vi =/ øj + Eij. where z are the neurons' weights, Iÿ the stimuli and fij Gaussian noise. Since both the input and the noise mean-centered, the Signal-to-No -o-isse Ratio of the neuronal population
N of N neurons and M stimuli is defined SNR 1 Σ i1=1 Σ
Var
In this setup, we vary the SNR by adjusting the noise variance, thereby controlling the noise power in the simulation. We varied the SNR between 0.001 and 10 and trained in the same way before. We calculated the Pearson correlation on a left out test set. Both DEC and DECEMber converge to similar performance since the MSE drives the learning of the model's weights (Fig. Fig. 8B). We that even for low SNR DECEMber successfully separates the clusters (Fig. Fig. 8A), though the distance not ideal before SNR is above O. However, for DEC cluster collapse happens independently of SNR (Fig. Fig. 8C). Since the weights (e.g. neurons) are generated based on predefined clusters (means of the Gaussians), we know the ground truth label for each neuron. To evaluate the cluster accuracy of DEC and DECEMber we measure the proportion of neurons that are assigned to the correct cluster.
A.2 Retina gagnlion cells
To select reliable cells from the marmoset RGCs dataset [48], we used the same reliability assess- ment of each cell's responses to visual stimuli as [60]; only reliable cells were used for model training. The model architecture was also taken from [15]. For clustering evaluations using DE- CEMber, k-means, and GMM, we considered only cells for which cell-type labels were avail-
A
B
Parasol ON
Irue parasel parasol
Parasol OFF
38
ON
able.
Predicted
The dataset contains recordings from two dif- ferent retinas of male marmosets. The second retina (not analyzed the main part of this pa- per) includes 38 parasol-OFF and 35 parasol- ON cells which are well separable (Fig. 9B). We trained our models all reliable cells from
Figure 9: A: DECEMber predictions. Pretraining length: 20 epochs. Same predictions for GMM and k-means. All methods have ARI 1. B: t-SNE projections of the corresponding cells.
15

---

this retina as well and tested DECEMber with varying pretraining lengths (which did affect cluster consistency). All three clustering meth- ods-GMM, k-means, and DECFMMer--sucesseeee and robustly identified the two cell types, as visualized Fig. 9A with ARI=I.
Cell type labels We used the same cell-type classification procedure as in [60] (Methods section 4.5), clustering the cells using KMeans++ algorithm on features extracted from receptive-field estimates obtained using spike-triggered averaging from responses to spatiotemporal white-noise, and from autocorrelograms computed on responses white-noise and naturalistic movies. The cell-type labels in our analysis differ from the ones used in the original publication because did not exclude cells that violated the tiling of spatial receptive fields.
A
CMM
1.0
B
k-means
-1.0
midget
72
76
0.8
Thu parasal
OFF
parto
32
0.4
0.4
large
0.2
0.2
22
22
Predicted
Predicted
0.95
0.90
0.85
0.80
0.75
0.70
midgen DFF parasal OFF
large OFF
Pretraining
26
30
Figure 10: All plots show evaluations of seed 4 of the trained marmoset RGC model [15] for retina I used in the main part of this paper. A: GMM predictions. B: k-means. C: f-SNE projections of the corresponding cells. D: ARI for different length of pretraining. Longer pretraining seems to be beneficial with ARI stabilizing after pretraining of 20 epochs.
A.3
ARI-stability for k-means and GMM marmoset RGC
1.0
Figure 11: Clustering stability of k-means and
0.9
GMM on retina 1 for marmoset RGC. We started with single baseline RGC model of retina (seed 2) and performed k-means and GMM clustering (4 clusters each), varying the random seed (42, 10, 100) for both algorithms. Clustering was done on all cells the model was trained on, but ARI calculated using only labeled cells.
0.8
0.7
0.6 GMM
k-means
16

---

A.4
ARI stability for GMM on mouse V1
1.0
Figure 12: Clustering stability GMM on mouse VI. We trained baseline model of mouse V1 for one seed (42). We then did GMM for clusters ranging from 5 to 60 the ground truth is not known varying just the seed for the initialization the GMM. It's clearly visible that GMM becomes unstable if the amount of clusters is large.
0.8
0.4
10
50
60
Number of clusters
A.5
Sensorium data details.
The model was trained on the SENSORIUM 2022 dataset [8], which contains neural responses to natural images recorded from seven mice (a total of 54,569 neurons). Recordings were made from excitatory neurons in layer and of the primary visual cortex using two-photon calcium imaging. In addition neural activity, the dataset includes five behavioral variables: locomotion speed, pupil size, the instantaneous change in pupil size (estimated via second-order central differences). and horizontal and vertical eye position, all of which are incorporated into the model. Three of them locomotion speed, pupil size, and the instantancous change in pupil size were appended to the grayscale images and are used input to the core, while pupil horizontal and vertical position were used as input to the shifter a model part shifting the readout receptive field locations depending on where the mouse looking. The validation set contains responses to roughly 500 and set to 5000 images per mouse.
A.6 Additional clustering metrics show qualitatively consistent results with ARI
Monkey
A
D
PE:5
0.7
PE:20
V-manure
FM--core
V-measure
FMM-scre
0.5
0.3
10
20
30
10
20
30
10
20
10
20
Number of clusters
Figure 13: Different clustering consistency metrics for monkey V4 and mouse retina datasets. Same as in the main paper, mouse retina is weighted across six models using all neurons, monkey V4 model trained on a subset of 1000 neurons. The order of lines is same as for ARI, confirming its results qualitatively, V-measure is biased towards bigger amount of clusters due to the set-based nature.
Clustering quality can be evaluated using metrics beyond ARI. ARI measures the similarity between two clusterings by checking whether pairs of points are assigned to the same cluster in both. The Nowlkes-Mallowe index [61] (Fig. 14) also compares two partitions but does not adjust for chance; the geometric mean of precision and recall, based on how consistently point pairs are clustered together.
Other common metrics homogeneity, completeness, and V-measure [62] are asymmetric and compare one clustering against reference (typically ground truth). Homogeneity measures whether each cluster contains only members a single class, while completeness checks whether members of given class are assigned to the same cluster. Swapping the roles of predicted and true labels inter- changes homogeneity and completeness. V-measure, equivalent normalized mutual information (NMI [63]) and it is the harmonic mean of the two. As in our case do not have ground truth,
17

---

B: 10
B: 10
Fovilos-Maillnn Mas-MMlloos Score
0.5
B: 10*
A: 10*
v-measure
0.35
10
8: 10' Baseline
10*
0.30
0.3
0.25
0.2
0.20
20
40
60
20
40
60
Number clusters
Number of clusters
Figure 14: V-measure and Fowlkes-Mallows--core for PE 10, 15 clusters.
compute the metrics with all possible seed pairs, which leads to homogeneity, completeness, and V-measure being equivalent (Fig. 14).
A.7 Influence Degree of freedom as a hyperparameter
Clusters: Clusters: 10
0.42
0.40
0.36
0.34
0.32
0.30
0.28
2.5
5.0
7.5
10.0 12.5 15.0 17.5 20.0
Figure 15: DECEMber, with PE = 10, 1==00008, 8 = 10 with different degrees freedom v.
We've tested the influence of the degree of freedom on ART for both and clusters 15 and can't. really see difference.
A.8 Comparison with the rotation equivariant baseline
Turishcheva et al. [13] is the only work to date that specifically addresses neuronal embedding consistency, and thus serves as our baseline for comparison. We use the "Mggorm 10 condition from their paper and compare to our consistency results in Fig. 16. Our approach achieves comparable consistency levels while climinating the need for a rotation-cquivariant core, thereby removing the post-hoc alignment step and improving predictive performance from s 38. 1% (Fig. A in [13]) to 39.5% (Fig. 4F).
A.9 Models performances on mouse retina and macaque V4 data
Baseline
PE
PE 10
PE
Mouse retina
0.4727 + 0.0008
0.4695 + 0.0009
0.4732 + 0.0008
0.4727 + 0.0009
Macaque V4
0.308 0.004
0.308 + 0.006
0.304 0.005
0.305 0.003
mouse retina and macaque V4 data for the models reported in the main Table I: Performances
paper (Sec. 5). Mouse retina is weighted as described in App. A.I0.All performances are on the held-out test set. The values averaged across all cluster counts. Seeds were 42, 101 and 7607. For GMM baseline the seed was 42.
18

---

0.45
PE:5
PE:30
0.40
PE:9
PE:40
0.35
PE:10
Baseline
0.30
PE:11
Rotation
PE:20
equivariant
ARI
0.25
0.20
0.15
0.10
0.05
10
20
30
40
50
60
Number of clusters
Figure 16: DECEMber, with PE = 10; DECEMber cluster consitency matches the rotation- equivariant model from Turishcheva et [13].
A.10 Further analysis of mouse retina data
Averaging across datasets For the mean ARI across datasets weighted ARI lines like Proul E, WiP4, where w; = / Thotal with Ylour the number of neurons in the current model, Phtal the number of neurons in all six models, and l4 is the average ARI score across three seeds for the current model. We used the law of total variance and computed the variance as o =
E; wi [o + (p+ [ where the first term captures within-dataset ARI variability and the second term captures between-datasee ARI variability. Fig. 17 shows the ARIs per models. We can see that the fewer neurons were present in the models the less the improvement was.
Dataset 0698360, n-936
Dataset 4rdcebc, n33
Dataset 5e62060, n-2787
0.7
baseline
0.6
PE:10
PE:15
0.5
0.4
Dataset 3tallBb,
Dataset n-2089
Dataset n-530
0.1
10
20
25
30
10
15
30
35
15
20
Number of clusters
Figure 17: ARI per retina. is the number of neurons in the model
A.11 Further analysis of monkeys data
For monkey V4, we trained models for 5, 10 to 20 and 25 clusters, as original work reported 12 clusters for 1000 cells. For 144 cells there were no labels and 100 cells ahd "not properly clustered" label. Therefore, decided to only the 1000 labeled cells. For results of models trained on all
19

---

A
ba line
B
Figure 18: A ARI be- tween models seeds for models trained on all 1244 neurons.
0.6
PE:5
PE:10
FE:20
APLO22
ARI
0.4
30
B Confusion matrix be- tween predictions of the model trained on 1000 neurons and labels sug- gested in Willeke et [51]
Thun
101G
20 11
10
20
Number of clusters
cells Fig, 18A. While the trends and values qualitatively similar the model trained only on a 1000 neurons subset, the standard deviation corridor seems to be wider, likely due to some of the "not properly clustered" neurons being in between the distinct groups. Please note that the labels from Willeke et al. [51] are rather a suggestion but not ground truth as they were not verified using independent biological measurements. For the confusion matrix of our labels and labels from Willeke et al. [51] see Fig. 18A. Same as for Burg et al. [21], our labels do not perfectly match the ones proposed in Willeke al. [51].
A.12
Bayesian inference criterion BIC
1.24
2.32
1.23
2.34
Likellhoot
1.22
2.36
1.21
-2.38
1.20-
-2.40
11.19
-2.42
1.18-
-2.44
1.17-
-2.46
1.16 6
10
20
50
20
50
Number of clusters
Number clusters
Figure 19: A: Log likelihood estimation. B: Calculation of BIC.
As suggested by reviewer, BIC could be an alternative way to find the underlying number of clusters. We used model with 10 pretraining epochs and 3 10
After training the model, we calculated the log-likeli likelihoodd of the neurons
N
log C log
txi Mg Eb),
where (. M4: Σx) is the multivariate 1-distribution with mean Mg, covariance Ex, and degrees of freedom M.
We computed the BIC as BIC log [ (2 K 1) log(N).
where is the number of neurons, d the dimensionality of the embeddings, K is the number of clusters.
The likelihood grows with the number of clusters whereas BIC falls with the number clusters (Fig Fig. 19) indicating the lack of clear cluster peak, which agrees with ARI (Fig Fig. 4H). BIC also scales with the number of model parameters which in our case increase a lot when the number of clusters increases making it not the most suitable measure in our case.
20

---

A.13
Compute requirements
All our models can be considered standards. A single mouse retina model requires less the 10Gb of GPU memory and trains under 20 minutes of walltime. A single mouse VI model requires under 2 hours of walltime. A single marmoset retina model uses 40Gb GPU and trains for under 16 hours of walltime. single monkey model requires 24Gb of memory and trains for under hours of walltime.
in terms of compute by modern deep learning model
12Gb of GPU memory and trains for
We use a local infrastructure cluster with NVIDIA RTX A5000 GPUs with 24Gb of memory each for mouse experiments. For mouse retina, marmoset retina, and monkey V4 we used 40Gb NVIDIA A100.
A.14 Broader impact
Our work contributes to building more reproducible models, which are more suitable for making biologically meaningful statements. It is even more related to derive functional taxonomy of cell types in the primary visual cortex, which can enhance our understanding of brain function and support the development of treatments for neurodegenerative diseases.
A.15 Experimental settings
For marmoset RGC dataset, we used the three layer CNN described in [15]. We trained it for maximum of 1000 epochs, stopping early if validation correlation did not improve for 20 epochs. The learning rate of both pretraining and training with the clustering loss was initially 0.005 and reduced during training using the ReduueeLROPPateau learning rate scheduler, patience 10 and minimal learning rate 1e 1e-8 For SENSORIUM 2022, used their model and training hyperparameters for the baselines training. Pretraining duration, learning rates and clustering strength / is reported in every experiment. For mouse retina, we followed Hofling et al. [18] model and training hyperparameters, changing only learning rate from 0.01 to 0.005 to improve baselines stability. Clustering strength was set to 0.001 across all experiments. For monkey V4 data we followed model and training hyperparameters from [37], again only changing the learning rate from 3 10 to 5 10 to improve baselines stability. Clustering strength was set to 0.001 across all experiments. Changing learning rate in both cases did not impacted performance in more than std boundaries.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The introduction clearly state the contributions an bullet point list at the end. Limitations as discussed Sec. as separate paragraph.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, have limitations paragraph in the Discussion (Sec. 6).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide code, experiments parameters and use open-source available datasets (three out of four). In order to access monkey dataset [51], please contact the original paper authors.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
We provide the code for repeating experiment the data and we use open- source available data for our experiments. The only exception is data from Willeke et al. [51], which could be available upon request from the original authors.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
public/guides Saliddds/CooeSuumissseee cniiiisssoooiiyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips.cc/ppii
http//uddes/eee/ /Coo//ooddeessseeeeeee sooolliiiciioooociccccooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: App. A.15
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report std corridors error bars in the vast majority all of our plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: App. A.13
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https httpp::////euiipp/ httpp///llc...ticccciiieeeeeeeee
Answer: [Yes].
Justification: Our research does not include any human data experiments. We believe research does not have any immediate societal impact and potential harmful consequences are interested in fundumental biological question and models reproducibility. We also believe that the research process was done in a harmless way. All the animal datasets used in this paper were collected primarily for different studies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Section App. A.14
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., dentoyment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers that produced the code package or dataset (Höfling et al. [18] for mouse retina data and models, Willeke for monkey V4 data [51], Pierzchlewicz
et al. [37] for monkey V4 model, Sridhar et al. [48] for marmoset RGC data and Vystrčilová al. [15] for the marmoset RGC models.). Openly available assets are Pierzchlewicz et al. [37] under the CC-BY-NC 4.0 license, which allows non-commercial use, and Höfling et al.
[18] data under CC BY-NC-SA 4.0 license, again allowing us to use data and models for research purposes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcoee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the code for the mouse V1 experiments, which includes our additional loss. As our code builds upon SENSORIUM 2022 codebase [8], well documented and straightforward to use.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should not be described.
28


Reviews:
[{"review_id": "j7O7OmYBtO", "reviewer": "Reviewer_mk7Y", "summary": "The authors introduce a method for clustering neurons by their functional properties.  They call this method DECEMber - Deep Embedding Clustering via Expectation Maximization-based refinement.  This algorithm works by adding a latent t-distributed class variable onto an existing predictive model.  After pre-training on the predictive model, EM steps with the t-distributed latents are alternated with updating the model parameters.  They tested this approach with a simple toy model of linear neurons separated into two functional clusters.  They also demonstrated classification of retinal ganglion cells, where it performed extremely well.  Interestingly, application of this approach to mouse V1 data suggests no clearly definable clustering as exists in the retina.  In this case, imposing the clustering prior appears to hurt performance of the models.  They also found that the length of pretraining had an effect on the consistency of obtained clusters.", "questions": "Questions are all given above in Strengths and Weaknesses.", "limitations": "The authors reasonably address limitations of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The notion of cell type is a question of broad importance in neuroscience, and the ability to define cell types via functional properties (let alone connecting these with other definitions of cell type) would facilitate substantial progress.  The authors’ method is straightforward and interesting and has the advantage that it can be combined with a broad class of models.  \n\nPersonally, I would have presented the model a bit differently, but this could be a matter of taste.  I found the introduction of the distributions p and q, followed by the description of the EM algorithm a bit confusing.  My approach would have been to define the latent space and modeling distributions which would be determined via the EM algorithm, and then present the update equations that are derived from that formulation via the E and M steps.  I feel like the logic of the approach would be clearer.  \n\nA potential weakness of the work is that it is unclear to what extent the underlying model can affect the results.  As an example, the neurons in the toy model are linear (and deterministic if I understand correctly?).  How does the approach perform with a reasonable model mismatch, i.e. how important is it to have an underlying model that is roughly correct?  In the case of the toy model, what would adding a stochastic term to the neuron outputs do to the results?  How would they change by the SNR of the resulting neurons?  You might easily imagine that the noisier the neurons, the more difficult the clustering problem, although on the other hand the strength of a clustering approach is the potential to share information across many neurons that have similar properties. (As the model develops the clustering, the clustering will of course feedback to the neuron models via the parameter update).  Conceptually, if you haven’t reasonably captured the appropriate set of features to describe the neurons, then clustering should be a lost cause.\n\nTo that last point:  what are the perfomances (in terms of Pearson correlation) for the toy model and the retinal models?  How do they compare to the V1 results?  Can you identify a systematic relationship between clustering and model performance? (I mean beyond the observation that imposing too strongly the clustering prior has a negative impact on performance.). For example, as mentioned above, if you decrease SNR by adding noise to the toy model, what does this do to the clustering outcomes?  Do they remain consistent?  This question is particularly important for the mouse V1 results because mouse visual neurons are extremely noisy in response to commonly used stimuli in neuroscience.  \n\nOn a side note, is there a reason you treat nu (in the t-distribution) as a hyperparameter and not something to be optimized in the EM algorithm?  I have no particular objection to this but wondered if there was a technical or scientific reason.  \n\nminor:\n\nOn line 140:  “the probability of feature i”,  should this read “neuron i”?", "was_revised": true, "final_justification": "I stand by my original score.  This is a nice paper and an excellent contribution.  The extra analyses are nice and will add value to the paper.", "rebuttal": "Thank you for your positive review, appreciating that our method addresses a *“question of broad importance in neuroscience”* and can be *“combined with a broad class of models”*.  \nBelow we are happy to address your questions and comments from the 3rd, 4th and 5th paragraph:\n* *Toy model from the manuscript*:  \nYou say, *“the neurons in the toy model are linear (and deterministic if I understand correctly?”*.  \nYes, you are correct that the toy model is linear and deterministic. Its primary purpose was not to test for robustness to noise, but to create a minimal setting where we could clearly demonstrate a specific failure mode of the original Deep Embedding Clustering (DEC) loss. The vanilla DEC loss fails because it assumes a fixed unit scale for the clusters, causing the centroids to collapse. Our toy example shows how introducing a learnable scale matrix Σj is essential to solve this problem.\n* *Signal-to-noise ratio*:  \nYou also suggested: *“In the case of the toy model, what would adding a stochastic term to the neuron outputs do to the results? How would they change by the SNR of the resulting neurons?”*  \nAs you correctly pointed out, its signal-to-noise ratio is a very important aspect of the data and as you suggested, we added a stochastic term in the toy example  to address it.  \nSpecifically, we generated:\n    * 1100 of images $I$, where each image is a 128-dimensional vector drawn from a standard Gaussian: $x_i \\sim  \\mathcal{N}(0, I_{128})$. We do a train/test split with a train set containing 90% and the test set containing 10% of the data.\n    * 1000 of neurons $N$, where the neuron weights are drawn from two Gaussian distributions to form uneven clusters: first “cell type” as  $w_j \\sim$ $ \\mathcal{N}($**0**$, I_{128}),  \\text{for } j = 1, \\dots, 700$, and second “cell type” as $w_j \\sim$ $ \\mathcal{N}($**1**$, I_{128}),  \\text{for } j = 1, \\dots, j = 701, \\dots, 1000$  \nThe clean neural responses were simulated as a dot product between the images $I$ and neurons $N$.\nTo simulate noisy observations, we added Gaussian noise independently for each neuron and image: $\\boldsymbol{\\varepsilon} \\sim \\mathcal{N}(0, \\sigma^2)$.\nThe noisy responses are thus given by: $y_{ij} = w_j^T x_i + \\varepsilon_{ij}$.   \nSince both the input $\\mathbf{I}$ and the noise $\\boldsymbol{\\varepsilon}$ are mean-centered, the Signal-to-Noise Ratio of neuron $j$ (SNR_j) is defined as:\n$\\mathrm{SNR}_j =$ $\\frac{\\operatorname{Var}_i(w_j^T x_i)}{\\operatorname{Var}_i(\\varepsilon_ij)}$ (both $i$ and $j$ are indexes for $\\varepsilon$ but openreview struggles to display it correctly).  \nIn this setup, we vary the SNR by adjusting the noise variance $\\sigma^2$, thereby controlling the noise power in the simulation.  \nWe varied the SNR between 0.001 and 10 and same as in the paper toy example we first pretrained the regression model on MSE for 30 epochs and then finetuned with DEC or DECEMber + MSE loss for 150 more epochs. We calculated the Pearson correlation on a left out test set. Both DEC and DECEMber converge to similar performance since the MSE drives the learning of the model’s weights.\nBelow you can find the table with the distance between cluster centers given different SNRS (the ground truth distance between the cluster centers is $\\sqrt{128}  \\sim 11.3$). \n**We see that even for low SNR DECEMber successfully separates the clusters**, though the distance is not ideal before SNR is above 1.\nHowever, for DEC cluster collapse happens independently of SNR.\nSince the weights (e.g. neurons) are generated based on predefined clusters (means of the Gaussians), we know the ground truth label for each neuron. To evaluate the performance of DEC and DECEMber we measure the proportion of neurons that are assigned to the correct cluster.  \n|SNR|Distance between cluster centers||Predictive accuracy (correlation)||Cell classification accuracy||\n|---|---|---|---|---|---|---|\n||DEC|DECEMBER|DEC|DECEMBER|DEC|DECEMBER|\n|0.001|1.968|3.908|0.000|0.000|51.6|50.1|\n|0.01|1.213|4.370|0.022|0.025|61.0|79.0|\n|0.05|0.197|6.993|0.100|0.109|79.0|98.1|\n|0.1|0.048|8.828|0.180|0.189|59.2|99.8|\n|0.5|0.015|10.494|0.489|0.494|51.4|100|\n|1|0.007|10.76|0.641|0.642|52.6|100|\n|2|0.011|10.95|0.771|0.771|50.2|100|\n|5|0.010|11.058|0.890|0.890|51.5|100|\n|10|0.015|11.075|0.941|0.940|50.3|100|\n\n* *Relationship Between Model Performance and Clustering:*  \nYou asked *“Can you identify a systematic relationship between clustering and model performance?”*, which is a very interesting point.  \nYou are right, *“conceptually, if we haven’t reasonably captured the appropriate set of features to describe the neurons, then clustering should be a lost cause”*.\nE.g. on a toy example for the smallest SNR=0.001 the cell types accuracy is worse than a random guess (the probability that a random guess matches the true label\n$P(\\text{predict}=i) \\times P(\\text{true label}=i) = 0.7^2 + 0.3^2 = 0.58$).\nHowever, even the minimal catch of meaningful signal already significantly improves the clustering accuracy for DECEMber (SNR=0.01 gives 2.5% correlation and 79% cell types classification accuracy). And of course, better model performance helps to improve the clustering: On the toy example above we see the cell types classification accuracy improving up to SNR=0.5, where it saturates as it reaches the best possible score.  \n**The models we use for real data, including mouse V1, are close to the state-of-the-art** [2] and actively used for closed-loop [3] or in-silico experiments [4,5]. Therefore, we believe they do capture meaningful features even though the correlations are far from 1. \nThe fact that DECEMber improves cluster consistency on the very noisy mouse V1 dataset provides strong evidence that DECEMber is effective in the high-noise, low-SNR regime that is typical for in-vivo neuroscience data, even though the models are not perfect.  \nYou also state that *“the strength of a clustering approach is the potential to share information across many neurons that have similar properties.”* That’s correct only if the clusters are very homogeneous. If there is a lot of within-cluster variance, there is not so much information to share. Pulling all neurons towards the cluster mean would actually hurt the performance, because the neurons indeed are different.  \nYou also mentioned in your summary that *“imposing the clustering prior appears to hurt performance of the models.”* It doesn’t actually hurt performance unless we overdo it. For smaller beta~=10^4 (Fig. 4H), predictive performance is maintained but embedding consistency improves substantially. The experiment of increasing the clustering weight, β, to the point where performance drops is intentional. It demonstrates that aggressively forcing a clustered structure can erase functionally relevant details, but does not improve embeddings’ consistency. This result supports our conclusion that excitatory neurons in the mouse visual cortex likely form a continuum rather than discrete types. **It is not a weakness of the method – it is a property of the data!**\n\n* *$\\nu$ as a Hyperparameter:*   \nIn the 5th paragraph, you ask if there is a reason we *”treat* $\\nu$ *(in the t-distribution) as a hyperparameter”*.  \nWhile it is possible to optimize ν in a t-mixture model, it does not have a simple closed-form update in the M-step, unlike the mean and scale matrices. We tried both having as a learnt hyperparameter on toy data and also explored the values between 2-20 on mouse V1 data, however, it had little to no impact on clustering outcomes or overall model performance, which is consistent with the original DEC paper [1], so we chose to fix $\\nu$ as a hyperparameter for simplicity and computational efficiency. We will add a section in the appendix summarizing these findings.  \n|ARI ± std|clusters|$\\nu$|\n|---|---|---|\n|0.416 ± 0.012|5|2.1|\n|0.417 ± 0.013||5|\n|0.418 ± 0.013||10|\n|0.42 ± 0.012||20|\n|0.299 ± 0.025|10|2.1|\n|0.299 ± 0.025||5|\n|0.301 ± 0.025||10|\n|0.303 ± 0.027||20|\n\n* *Line 140*: yes, “feature i” is equivalent to “neuron i”\n\nWe thank you for the biologically meaningful questions and helping us to improve our toy model. We will integrate it into the final manuscript.  \nWe are happy to address any follow-up questions as well.\n\n*References*:  \n[1] Xie, J., Girshick, R., & Farhadi, A. (2016, June). Unsupervised deep embedding for clustering analysis. In International conference on machine learning (pp. 478-487). PMLR.  \n[2] Willeke, K. F., Fahey, P. G., Bashiri, M., Hansel, L., Blessing, C., Lurz, K. K., ... & Sinz, F. H. (2023, August). Retrospective on the SENSORIUM 2022 competition. In NeurIPS 2022 Competition Track (pp. 314-333). PMLR.  \n[3] Walker, E. Y., Sinz, F. H., Cobos, E., Muhammad, T., Froudarakis, E., Fahey, P. G., ... & Tolias, A. S. (2019). Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12), 2060-2065.  \n[4] Ustyuzhaninov, I., Burg, M. F., Cadena, S. A., Fu, J., Muhammad, T., Ponder, K., ... & Ecker, A. S. (2022). Digital twin reveals combinatorial code of non-linear computations in the mouse primary visual cortex. BioRxiv, 2022-02."}, {"review_id": "Dbd3vXTo9d", "reviewer": "Reviewer_ZNMh", "summary": "This paper proposes DECEMber, a deep learning clustering model designed to learn meaningful latent neural representations. The model is trained in a two-stage process. Initially, a pre-training step establishes a feature representation using an architecture composed of a shared core and neuron-specific Gaussian readouts. This core generates a common feature map, and individual neuron responses are computed via a dot product between their weights and a selected feature vector, guided by each neuron's receptive field. Subsequently, the model is fine-tuned for clustering by introducing an additional loss to the objective function, leveraging the EM algorithm to iteratively learn and refine cluster assignments. The authors validate the clustering capabilities of DECEMber across a range of datasets, including a numerical simulation and complex biological data from the retina and visual cortex of mice and non-human primates.", "questions": "- The authors stated that the model avoids the trivial solutions and mode collapse issues inherent in DEC by learning a covariance matrix for each cluster. However, the mechanism by which this prevents mode collapse is not fully clear. Could the authors please elaborate on precisely how learning this covariance matrix confers this stability? For instance, if the clustering stage were initialized with a unit scale factor for all clusters, as is common, what prevents the model from converging to a trivial solution before the covariances are sufficiently learned? \n\n- In the analysis of the marmoset retina dataset, it was noted that the model clusters neurons by their source retina, suggesting a batch effect. As a result, the authors trained each retina separately. This outcome is somewhat unexpected given the model’s architecture. A key motivation for learning a shared neural embedding in the first stage is presumably to produce representations that are invariant to nuisance factors such as batch effects. Could the authors clarify why the model, despite this design, fails to integrate data from the two retinas?", "limitations": "The authors have already mentioned some important limitations of the proposed deep clustering method. In my view, the most critical limitations are:\n\n•\tThe need to predefine the number of clusters during training. The model lacks a strategy for adaptively determining the optimal number of clusters.\n\n•\tThe model's performance appears to be sensitive to the number of pretraining epochs, the $\\beta$ hyperparameter, and the learning rate.\n\n•\tThe proposed framework is not a sufficiently robust exploratory tool for uncovering both discrete and continuous sources of variability in the dataset.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\n- Important Problem Domain: The paper tackles an important and challenging problem of identifying functional cell types by learning neuronal functional embeddings. This is a core question in neuroscience, and novel computational approaches are highly valuable.\n\n- Comprehensive Datasets: The authors validate their model on a sufficient combination of numerical simulations and relevant real-world datasets.\n\n**Weaknesses**\n\n- Suboptimal Training Strategy: The two-stage training process that includes pre-training a core unit followed by clustering-specific fine-tuning, is a suboptimal design. This separation can prevent the model from learning features that are informative for the final clustering task from the outset. A joint, end-to-end training approach would likely yield more task-relevant embeddings. This limitation is highlighted by the ambiguity in hyperparameter tuning (e.g., Figures 5 and 7), where it is unclear how many pre-training epochs are optimal.\n\n- Unsuited for Discovering Cluster Number:  The proposed framework is not an appropriate choice for addressing the fundamental question of how many functional cell types exist in the data or whether they are organized into discrete or continuous structures. The model itself does not provide a mechanism for model selection. The authors attempt to use the Adjusted Rand Index (ARI) for this. While the ARI measures similarity between clustering results or against a known ground truth, it does not indicate the correct number of clusters. A more appropriate method, like using the Bayesian Information Criterion (BIC), would be required to provide evidence for a specific number of clusters. I do not think the paper's claims about continuous versus discrete organization are not well-supported by its own analysis.\n\n- Clustering Algorithm: The decision to build upon Deep Embedded Clustering (DEC) may not be the most suitable choice. Prior studies, IDEC [1] have highlighted that the clustering loss used in DEC can distort the feature space by forcing embeddings toward predefined cluster centroids. This approach risks compromising the local structure of the data, potentially resulting in learned representations that fail to preserve the intrinsic organization of the original neural activity.\n\n- Insufficient Baselines and Performance: The comparison to other methods is lacking. The paper only includes simple baselines like GMM and K-means, while ignoring more powerful deep clustering techniques (e.g., methods based on contrastive learning like [2] or IDEC).\n\n- The model's reported predictive performance seems low. It is primarily reported as a correlation score (Figure 4), which consistently remains below 0.4 across different experiments, raising questions about the model's practical utility.\n\n- Supplementary Materials: There appears to have been a mistake during the upload of the supplementary material. The supplement is attached to the main manuscript file, and the separately uploaded supplementary file seems to be missing content\n\n-------------------------\n[1] Guo, Xifeng, et al. \"Improved deep embedded clustering with local structure preservation.\" Ijcai. Vol. 17. 2017.\n\n[2] Shen, Yuming, et al. \"You never cluster alone.\" Advances in Neural Information Processing Systems 34 (2021): 27734-27746.", "was_revised": true, "final_justification": "I will keep my score. In my opinion, the current version of the manuscript requires major revisions to more demonstrate the impact and novelty of the method, and clearly distinguish its results. At present, it is unclear how other comparable methods, such as IDEC, would perform on visual cortex datasets. The authors may need to consider adding a few well-established benchmarking datasets to showcase the superiority of their method compared to others.", "rebuttal": "Thank you for your detailed feedback, acknowledging that we are solving an important problem on a “comprehensive” real-world dataset. \n\nWe believe that the concerns raised are mostly due to misunderstandings.\n*Let us first address the criticism (weaknesses) you raised:*\n\n- *Suboptimal Training Strategy / Clustering Algorithm:*   \n**We do train jointly**. After a brief pretraining phase, the clustering loss is combined with the Poisson loss, similar in spirit to IDEC [1], thereby preserving local structure. To avoid biasing neuronal structure in any way we initialize neuronal embeddings as a constant at the beginning. Therefore, the model first needs to learn minimal feature separation for clustering to be meaningful. Additionally, IDEC also “pretrain a stacked denoising autoencoder before performing clustering”. Hence, we are already following your suggestions.  \nDifferent optimal amounts of pretraining epochs across datasets are more likely explained by different model architectures and very different nature of data in different datasets (electrophysiology vs calcium imaging, hundreds of neurons to thousands of neurons in a model and different signal-to-noise ratios). \n\n- *Discovering Number of Clusters*:  \nWe would like to clarify why ARI is suitable to detect the number of clusters: ARI measures the agreement between two partitions of points by evaluating whether pairs of points are assigned to the same cluster in both partitions. When a ground-truth number of clusters exists, the ARI should peak at that value because points are consistently grouped together. This makes our clustering approach well-suited even in scenarios where the true number of clusters is not known in advance. We validated it on a marmoset retina, where N=4 is the ground truth number of clusters and we indeed see a clear ARI peak at 4 clusters.\n| Clusters | ARI (mean ± std) |\n|----------|------------------|\n| 2 | 0.78 ± 0.07|\n| 3 | 0.52 ± 0.31|\n|**4** | **0.95 ± 0.01**|\n| 5| 0.78 ± 0.08|\n| 6| 0.77 ± 0.15|\n| 7| 0.81 ± 0.08|\n| 8| 0.85 ± 0.05|\n| 9| 0.79 ± 0.04|\n| 10| 0.85 ± 0.07|\n| 15| 0.71 ± 0.07|\n| 20| 0.74 ± 0.06|\n| 25| 0.67 ± 0.04|\n| 30| 0.49 ± 0.02|\n| 40| 0.41 ± 0.04|    \n\n    We will integrate this analysis in the final version of the paper.  \n    As you suggested, BIC could be an alternative way to select the number of clusters.\n    As BIC requires held-out neurons to estimate the test likelihood, we did it on mouse V1 datasets (marmoset retina is too small for it as it has ~150 neurons).\n    For each mouse we randomly hold-out 200 neurons for each of the seven mice, which were trained together with the model but were ignored during the clustering procedure which fitted the TMMs. We used a model with 10 pretraining epochs and β=10^4. After the model was trained we a) computed the likelihood on the hold-out neurons and b) performed an additional E-step on the held-out neurons, calculating the soft assignments, such that we could evaluate ARI on only these 200 neurons as well.  \n    We computed BIC as $BIC = -2 log L + (2 \\cdot k \\cdot d + 1) \\cdot log(N)$, $N$ is the number of samples ($N = 200 * 7$), $d$ is the dimensionality of the embeddings ($k=128$ in our case),  $k$ is the number of clusters and $log L$ is the likelihood computed as $log L =  \\log\\left( \\frac{1}{K} \\sum_{k=1}^{K} t(x \\mid \\mu_k, \\mathrm{var}_k, \\nu) \\right) $. \n    As we have not observed any significant changes in the likelihood, BIC just linearly grows with the number of clusters (number of model parameters), indicating the lack of clear cluster peak, agreeing with ARI (Fig. 4Hin the paper).  \n    |Clusters|Likelihood|BIC|\n    |---|---|---|\n    |5|375.3 ± 27.8|8529 ± 55.6|\n    |10|348.5 ± 15.7|17855 ± 31.5|\n    |15|355.0 ± 34.4|27115 ± 68.7|\n    |20|335.1 ± 3.2|36427 ± 6.3|\n    |25|382.9 ± 32.2|45605 ± 64.4|\n    |30|387.2 ± 42.7|54869 ± 85.3|\n    |35|351.2 ± 30.2|64213 ± 60.5|\n    |40|371.2 ± 33.4|73446 ± 66.8|\n    |45|357.4 ± 26.6|82746 ± 53.2|\n    |50|367.7 ± 22.1|91998 ± 44.3|\n\n\n- *Insufficient Baselines:* We use post-hoc GMM/k-means clustering baselines due to the lack of more suitable alternatives. Our method already resembles IDEC as we do joint optimization to preserve local structure. We cannot use vanilla DEC since it forces unit-scale embeddings, but our embeddings are model weights whose scale  is determined by the scale of neuronal activity. Thank you for pointing us to the “You Never Cluster Alone” paper [2]. While interesting, its contrastive approach would require augmentation of model weights—our per-neuron embeddings. As it’s not clear how one would do data augmentation in this case, this would constitute a separate research project.\n\n- *Low predictive performance:* The predictive performance is not low – it’s close to the state of the art. It just reflects that predicting neuronal responses in mouse V1 is a hard task. Our numbers are comparable to the best models on Sensorium 2022 [3] – Table 3. A few percent can be gained by ensembling multiple models. We show the potential for this improvement by ensembling models across 8 seeds - this gave us 41% correlation for models with 5, 10 and 20 clusters (there were 8 seeds for each number of clusters, we did not ensemble across a number of clusters). Please note that this is a qualitative comparison as sensorium 2022 live and final test sets are not publicly available, we used test sets from 5 fully publicly available mice to compute the test metrics. \n\n- *Supplementary Materials:* We apologize for the confusion. We intended to upload all supplementary materials together with the main text to preserve hyperlink functionality, with a single plot being added later. We will integrate all materials into a single document.\n\n*With respect to your questions:*\n\n1) *Mode collapse:* To clarify: we are not addressing general DEC mode collapse, but adapt DEC to our specific setting, where embedding scales are not arbitrary. In standard DEC, unit-scale clusters are acceptable because the latent space of the autoencoder can adapt. In our case, the embeddings to be clustered are model weights. Their scale is constrained by the scale of neuronal activity. Changing their magnitude would harm the predictive model. Therefore, we need to adjust the scale and shape of the clusters. To do so, we introduce scale matrices that let the model capture anisotropic cluster structure and non-unit scale. As noted in line 159, these matrices are initialized from empirical within-cluster variances, computed after a few pretraining epochs using only the Poisson loss—providing a data-driven, meaningful starting point.\n\n2) *Batch effects:* As you correctly noticed, we do train several models for mouse and marmoset retinas. We do it due to the differences in the experimental conditions (presumably different temperature) which influences the speed of the responses so much that this would be the most trivial dimension to separate the neurons for different groups [4, 5]. While several works try to address this issue and integrate several retinas into the same model [6, 7, 8], this is still an active area of research, which is not the main focus of our paper. \nWe also would like to note that DECEMber is not a specific model per se, but rather an additional loss that we integrate with various models. This also means that models for monkey V4, mouse V1 and mouse retina all have slightly different architecture, especially on the readout layers. And for mouse V1 and monkey V4, we do train jointly on several animals, as the experimental conditions between subjects there are comparable and DECEMber does not suffer from batch effects there. \n\n\nWe will add IDEC[1] into the related works section and integrate the BIC analysis to the appendix.\n\nWe hope these comments shed some light on our motivation, technical and design choices. We also hope that it would help you to reconsider some criticism.\n\n*References*:  \n[1] Guo, Xifeng, et al. \"Improved deep embedded clustering with local structure preservation.\" Ijcai. Vol. 17. 2017.\n[2] Shen, Yuming, et al. \"You never cluster alone.\" Advances in Neural Information Processing Systems 34 (2021): 27734-27746.  \n[3] Willeke, K. F., Fahey, P. G., Bashiri, M., Hansel, L., Blessing, C., Lurz, K. K., ... & Sinz, F. H. (2023, August). Retrospective on the SENSORIUM 2022 competition. In NeurIPS 2022 Competition Track (pp. 314-333). PMLR.  \n[4] Zhao, Z., Klindt, D. A., Maia Chagas, A., Szatko, K. P., Rogerson, L., Protti, D. A., ... & Euler, T. (2020). The temporal structure of the inner retina at a single glance. Scientific reports, 10(1), 4399.  \n[5] Fritsches, K. A., Brill, R. W., & Warrant, E. J. (2005). Warm eyes provide superior vision in swordfishes. Current Biology, 15(1), 55-58.  \n[6] Höfling, L., Szatko, K. P., Behrens, C., Deng, Y., Qiu, Y., Klindt, D. A., ... & Euler, T. (2024). A chromatic feature detector in the retina signals visual context changes. Elife, 13, e86860.  \n[7] Vystrčilová, M., Sridhar, S., Burg, M. F., Gollisch, T., & Ecker, A. S. (2024). Convolutional neural network models of the primate retina reveal adaptation to natural stimulus statistics. bioRxiv, 2024-03.  \n[8] Gonschorek, D., Höfling, L., Szatko, K. P., Franke, K., Schubert, T., Dunn, B., ... & Euler, T. (2021). Removing inter-experimental variability from functional data in systems neuroscience. Advances in Neural Information Processing Systems, 34, 3706-3719."}, {"review_id": "AQkqZeSe6y", "reviewer": "Reviewer_2CDw", "summary": "This paper addresses a core question in computational neuroscience: whether neuronal functions constitute discrete cell types, particularly in the mouse primary visual cortex (V1). Existing methods cluster neuron embeddings learned by predictive models, but the results of such clustering lack consistency. The authors propose a new method called DECEMber, which introduces an explicit clustering inductive bias by adding an auxiliary clustering loss, inspired by deep embedding clustering, to the predictive model's loss function. The core innovation of this method is its use of a multivariate Student's t-mixture model combined with the Expectation-Maximization (EM) algorithm to update clustering parameters, thereby adapting to the non-isotropic features of neuronal embeddings.\n\nThe authors validate the method's effectiveness on a \"positive control\" dataset where discrete cell types are known to exist—marmoset retinal ganglion cells—and apply it to several other datasets, including mouse V1, mouse retina, and macaque V4. The main conclusion is that although DECEMber improves clustering consistency, it does not find strong evidence supporting the existence of discrete functional cell types in mouse V1, suggesting its functional organization is more likely a continuum.", "questions": "see  Weaknesses", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n(1)Importance of the Problem: The paper investigates a fundamental and challenging problem in neuroscience—the identification of functional cell types. Applying machine learning methods to test this biological hypothesis holds significant scientific value.\n\n(2)Clear, Hypothesis-Driven Framework: Structuring the method as a form of \"model-driven hypothesis testing\" is a clever and rigorous approach. That is: if discrete functional types do exist, introducing a clustering bias should improve certain performance metrics of the model.\n\n(3)\"Positive Control\" Experimental Design: The experimental results on the marmoset retina dataset, known to have discrete cell types, are convincing. DECEMber achieved an extremely high ARI score (0.96) and near-perfect classification results on this dataset (Fig. 3B), which strongly demonstrates the method's potential under ideal conditions.\n\n(4)Exhaustive Experiments and Transparency: The authors conducted extensive experiments across multiple species (mouse, primate), multiple brain regions (retina, V1, V4), and multiple model architectures (Gaussian readout, attention readout). Furthermore, the paper is clearly written and candidly lists several limitations of the method in the discussion section.\n\n3.Weaknesses\nDespite the aforementioned strengths, there are several significant weaknesses that make this paper unsuitable for publication at NeurIPS.\n\n(1)Limited Novelty: The core of the method is combining a clustering loss with a prediction loss. This idea is not new. DECEMber is essentially a combination of existing techniques: Deep Embedding Clustering (DEC, 2016), t-mixture models (TMM, 2000), and the EM algorithm. While its application in the field of neuroscience is new, from a machine learning methodology perspective, its conceptual contribution is incremental and does not meet the high standards of NeurIPS.\n\n(2)Complex Method and Highly Sensitive to Hyperparameters: DECEMber introduces a large number of key hyperparameters that require careful tuning, including the clustering loss weight β, the number of pre-training epochs m, the number of clusters J, and the learning rate. One of the paper's core claims is that the \"model improves clustering consistency while maintaining high predictive performance\". However, Figure 4F clearly shows this is a direct trade-off, not maintenance. When the β value is increased to obtain better t-SNE visualizations, the model's predictive performance drops significantly. Performance is only unaffected at small β values (≤10⁴), but at that point, the improvement in consistency is also limited.\n\n(3)Tedious Tuning Process: Figures 5A and 5B show that the model's performance and consistency are sensitive to the choice of learning rate and the number of pre-training epochs when trying to achieve optimal parameters. For example, on the mouse V1 data, the ARI peaks at 10 pre-training epochs. This means users need to perform an expensive grid search to find the optimal hyperparameter combination for each new task, which greatly undermines the method's practicality and robustness. The authors themselves admit that selecting an appropriate β is crucial and that the pre-training duration requires \"further work\" to determine.\n\n(4)The Method's Value Is Not Realized: For mouse V1, the paper's main scientific conclusion is that discrete functional cell types \"likely do not exist,\" but rather form a continuum. While this is a valuable scientific finding, it fundamentally undermines the motivation for proposing this complex clustering method. If the core problem the method aims to solve does not have a clustered solution, what is the point of introducing this complex clustering mechanism? This makes the primary application scenario of the method seem somewhat contradictory.\n\n(5)Methodological Flaw: The authors acknowledge that to address potential numerical stability issues with high-dimensional covariance matrices, they adopted a \"heuristic rather than principled\" approach of truncating smaller values. Relying on such an ad-hoc solution in a core algorithmic component is a significant methodological flaw.\n\n(6)Inconsistent and Potentially Misleading Baseline Comparison: A core argument of the paper is that DECEMber outperforms standard methods like GMM and k-means in clustering stability. However, the evidence for this claim is contradictory. In the main results (Fig. 3A), the GMM baseline has an ARI of about 0.85 and shows huge error bars, indicating its instability across different model seeds. However, in the appendix (Fig. 10), the authors show that on the embeddings from a single pre-trained model, GMM clustering itself is highly stable (ARI ≈ 1.0), while k-means is unstable. The discrepancy between these two results is not explained in the text. This strongly suggests that the instability of GMM in Figure 3A stems mainly from the differences in the embedding spaces learned by the predictive model itself under different random seeds, not from the GMM algorithm itself. If so, comparing an end-to-end trained method like DECEMber with a post-hoc GMM clustering on unstable embeddings is unfair. This inconsistent presentation severely weakens the core claim that DECEMber is superior to the baselines.\n\n(7)Poor Data Visualization Hinders Understanding of Core Results: The presentation of several figures in the paper is flawed, making it difficult for the reviewer to evaluate its key arguments.\n\n①Fragmented Presentation of the Core Trade-off in Figure 4: Figure 4 is central to the paper's argument regarding the mouse V1 results. However, the reader must jump back and forth between subplot 4F (predictive performance) and subplot 4H (clustering consistency ARI) to understand the trade-off between performance and consistency. A more direct and convincing visualization would be to plot these two key metrics on the same graph (e.g., x-axis for ARI, y-axis for predictive performance, with different colors/symbols for β values). The current fragmented presentation obscures rather than clarifies the core issue.\n\n②Coarse and Inconsistent Presentation of Generalization in Figure 7: The authors claim that DECEMber has good generalization properties, but the presentation in Figure 7 fails to support this claim convincingly.\n\n③Overly Brief Figure Caption: The caption for Figure 7 is extremely uninformative, only stating the dataset sources. It does not explain the trends or key findings in the figure, leaving the reader unable to understand its intent independently.\n\n④Inconsistent Evaluation Criteria: The x-axis ranges (number of clusters) in Figure 7A (mouse retina) and Figure 7B (macaque V4) are different. More importantly, their evaluation methods differ: A is a weighted average across 6 models, while B is a simple average across 3 model seeds. This lack of a unified evaluation framework makes cross-dataset comparison meaningless and severely undermines the claim of \"generalization\".\n\n(8)Ambiguous Contribution to the Core Scientific Question: The authors position this method as a tool to test for the existence of discrete functional cell types in mouse V1. However, even with DECEMber, the best ARI value obtained on mouse V1 is still very low (around 0.25). While this is a relative improvement over the baseline (around 0.15), an absolute value so close to random (ARI=0) can hardly be considered a \"meaningful\" cluster structure. Therefore, concluding that \"mouse V1 likely lacks discrete functional cell types\" based on an improvement from 0.15 to 0.25 is a weak argument. The method's improvement is too small to provide strong evidence for answering this important biological question.", "was_revised": true, "final_justification": "The innovation of this article is limited, as many existing works have achieved similar results, and the author has not made a comprehensive and fair comparison. So I choose to reject it.", "rebuttal": "Thank you for your thoughtful feedback, highlighting that we are solving an important problem with a *\"clear, hypothesis-driven framework\"*.\nBelow we address your specific comments from the review:\n\n*(1) Limited Novelty*:  \nOf course our method builds on prior work (DEC, EM, TMM), but it introduces key innovations:\nThe novelty is the combination of solving a multivariate non-linear regression task (how neurons respond to images) jointly with clustering the neurons’ readout weights. Unlike DEC, which clusters input samples by altering the model’s latent space and enforcing unit-scale clusters, we cluster model weights directly, constrained by the Poisson loss, which is jointly optimized with the clustering loss. This requires significant adaptation, including introducing EM updates, learning the scale matrix and an appropriate curriculum of first pre-training for the regression task. DEC serves more as inspiration than as a direct combination of methods.\nOur model provides a hypothesis test for a core neuroscience question—whether distinct functional cell types exist in mouse V1. Currently no such method exists!\nThe components may be known, but their integration, adaptation, and application are novel and scientifically meaningful.\n\n*(2) Method Complexity*:  \nDECEMber has several hyperparameters, but not all need tuning. For example, β just needs to roughly balance clustering and Poisson losses, which can be estimated in 1–2 few epochs runs. In several experiments (marmoset, mouse retina, monkey V4), we did exactly this, without sweeping β.\nRegarding Figure 4: It is correct that very large β>=10^5 hurt performance (Fig. 4F). However, consistency improves substantially compared to the baseline already with much smaller β~=10^4 (Fig. 4H), in which case the high predictive performance is maintained. Thus, **there is no trade-off**. We will improve the figures to make this clearer in the final version.\nThe experiment of increasing the clustering weight, β, to the point where performance drops is intentional. It demonstrates that aggressively forcing a clustered structure can erase functionally relevant details, but does not improve embeddings’ consistency. This result supports our conclusion that excitatory neurons in the mouse visual cortex likely form a continuum rather than discrete types. **It is not a weakness of the method – it is a property of the data!**\nFinally, t-SNE plots were used only for visualization, not for tuning β—we’ll make this clearer in the text.\n\n*(3) Tedious Tuning*:  \nYou're right that careful hyperparameter selection matters and we appreciate the opportunity to clarify DECEMber’s practicality and robustness.\nWe believe that the degree of tuning required is manageable. **Even sub-optimal hyperparameters yield substantial benefits**. The ARI for mouse V1 data does peak at 10 pretraining epochs, but Fig. 5C shows that nearly all tested settings outperform the baseline across a range of cluster counts. This suggests DECEMber offers more consistent embeddings than standard methods, even without a perfectly tuned grid search.\nOn other datasets (mouse retina and macaque V4), we explicitly avoided extensive tuning—only making minor learning rate adjustments for stability and choosing β as described above (1 additional run to estimate clustering loss order) and 3 different pretraining epochs to try out—and still observed a doubling of ARI without compromising predictive accuracy (see App. A6 for predictive performances). This highlights DECEMber's robustness and strong out-of-the-box performance.\nWe acknowledged these limitations in our discussion. As with many flexible deep learning methods, some tuning is expected, but the resulting gains in embedding stability and interpretability justify the effort.\n\n*(4) Value of the Method:*  \nYou ask “*If the core problem the method aims to solve does not have a clustered solution, what is the point of introducing this complex clustering mechanism?”* – The point is that **it is an open scientific question whether the solution is clustered**. Neurons are widely thought to form discrete types morphologically and genetically, and in the retina, this holds for function as well [6]. But whether V1 neurons cluster functionally is unknown. Our method tests this: if discrete types exist, clustering should improve model structure or performance (lines 314–316). This holds in the retina (Fig. 3), where functional types are known. \nMoreover, as reviewer gTJC suggested, varying cluster count for marmoset retina reveals a clear peak at the ground-truth: 4 clusters, again showing DECEMber capability to reveal distinct cell types if they exist.\n\n| Clusters | ARI (mean ± std) |\n|----------|------------------|\n| 2 | 0.78 ± 0.07|\n| 3 | 0.52 ± 0.31|\n| 4 | **0.95 ± 0.01**|\n| 5| 0.78 ± 0.08|\n| 6| 0.77 ± 0.15|\n| 7| 0.81 ± 0.08|\n| 8| 0.85 ± 0.05|\n| 9| 0.79 ± 0.04|\n| 10| 0.85 ± 0.07|\n| 15| 0.71 ± 0.07|\n| 20| 0.74 ± 0.06|\n| 25| 0.67 ± 0.04|\n| 30| 0.49 ± 0.02|\n| 40| 0.41 ± 0.04|\n\nThat is, the marmoset retina dataset provides the positive control that our method does what it is supposed to do. \nIn mouse V1 (Fig. 4), in contrast, clustering does not improve predictive performance and ARI stays below 50%, with no clear peak — suggesting a functional continuum. This negative result is meaningful and supports recent findings [1-3]. Rather than undermining the method, it highlights the method’s value for quantitatively testing, not assuming, biological structure.\n\n*(5) Methodological Issue:*  \nClipping small values is a common and practical approach to address numerical stability and it worked well in all our experiments. This is really a small detail that has no practical implication for the method.\n\n*(6) Baseline Comparison:*  \nWe appreciate your careful observation. You note that *“comparing an end-to-end trained method like DECEMber with a post-hoc GMM clustering on unstable embeddings is unfair”* — This is precisely the motivation for our method, because no other end-to-end method for this problem exists. Therefore, we believe that comparison with post-hoc baselines is both necessary and meaningful. If you are aware of alternative baselines that would offer a fairer comparison, we would be grateful for suggestions.  \nTo be clear: Across multiple DECEMber runs, the random seed of the predictive model's parameters also varies. The fact that combining training the predictive model and clustering leads to more consistent embeddings (and, hence, clustering) is one of our main contributions.\n\n*(7) Visualizations:*  \nWe will improve the figures and captions.  \n1 -We will add the suggested plot to the manuscript.\nAs we cannot add plots for the rebuttal, below is a table. It shows that for a fixed number of clusters when β is increasing ARI increases as well and the predictive performance stays roughly constance before β>=$10^5$, where we see a rapid drop in performance without any gain in ARI. \n| ARI (mean) | Predictive performance | β | Cluster |\n|---|---|---|---|\n| 0.263 | 0.398 | 10 | 5 |\n| 0.378 | 0.396 |$10^2$| 5 |\n| 0.403 | 0.398 |$10^3$ | 5 |\n| 0.416 | 0.397 |$10^4$| 5 |\n| 0.419 | 0.395 |$10^5$| 5 |\n| 0.421 | 0.385 |$10^6$| 5 |\n| 0.422 | 0.372 |$10^7$| 5 |\n| 0.193 | 0.398 | 10 | 10 |\n| 0.267 | 0.397 |$10^2$| 10 |\n| 0.286 | 0.398 |$10^3$| 10 |\n| 0.298 | 0.396 |$10^4$| 10 |\n| 0.301 | 0.394 |$10^5$| 10 |\n| 0.304 | 0.377 |$10^6$| 10 |\n| 0.305 | 0.256 |$10^7$| 10 |\n| 0.14 | 0.398 | 10 | 20 |\n| 0.189 | 0.398 |$10^2$| 20 |\n| 0.205 | 0.397 |$10^3$| 20 |\n| 0.218 | 0.397 |$10^4$| 20 |\n| 0.221 | 0.391 |$10^5$| 20 |\n| 0.224 | 0.372 |$10^6$| 20 |\n| 0.225 | 0.352 |$10^7$| 20 |\n\n2-4 -  Fig 7 caption will be : *“ARI across three seeds per model. A: Mouse retina [19], averaged over six models reflecting different experimental conditions (see App. A.7). B: Monkey V4 [51], three seeds. DECEMber improves ARI across both datasets and architectures, consistent with mouse V1 (Fig. 4H), supporting its generalization.”*\n\nWe will extend Fig. 7A/B to 60 clusters for consistency with Fig. 4H. However, **the evaluation method is the same across Figures 4H, 7A, and 7B** - all ARI values are computed over three seeds per model. For the mouse retina, we train six separate models to account for different experimental conditions (temperature), which strongly affect response speed and would trivially separate neuron groups [3,4]. ARI curves are averaged over three seeds and weighted by neuron count. See Appendix A.7 for details and per-model ARIs.  \n\nOverall, these results further support our finding: DECEMber improves consistency across diverse datasets and models, when enough neurons are available.\n\n*(8) Ambiguous Contribution:*  \nWe appreciate your concern as it's an important point to understand.\nFirst, to clarify: in Fig. 4F, the best ARI on mouse V1 is ~0.40 (not 0.25) for 5 clusters, with a baseline of ~0.25 (not 0.15) — a ~60% improvement, consistent across cluster counts.\n\nThat said, we agree the absolute ARIs are low, as are other metrics (e.g., Fowlkes–Mallows score ~0.5, Appendix A.4), even when the clustering bias is strong. But this is not a model weakness — **it’s the core scientific finding**: no method, including DECEMber, reveals clear, stable clusters. This supports the idea that mouse V1 excitatory neurons form a functional continuum rather than discrete types (if discrete functional cell types existed, we’d see an ARI peak at some cluster count—not a decaying trend, as we saw in point 4 for the marmoset retina). \nWe hope this reply helps to clarify our contributions,methodology, and revisit the evaluation. \n\n*References*:  \n[1] Weis et al. (2025) doi: 10.1038/s41467-025-58763-w  \n[2] Tong et al. (2023) doi: 10.1101/2023.11.03.565500  \n[3] Weiler et al. (2023) doi: 10.1093/cercor/bhac303  \n[4] Zhao et al. (2020) doi: 10.1038/s41598-020-60214-z  \n[5] Höfling et al. (2024) doi: 10.7554/eLife.86860  \n[6] Baden et al. (2016) doi: 10.1038/nature16468"}, {"review_id": "2tPm5fTfk2", "reviewer": "Reviewer_gTJC", "summary": "This paper points out a problem in neural activity predictive models: Although they can achieve high predictive performance, the extracted features do not exhibit clear clustering patterns that correspond to cell types. This paper then proposes to add a clustering loss to the neural activity prediction objective to explicitly enforce clustering. The clustering loss is also a novel one that improves the existing deep embedding clustering loss, which failed to separate a toy dataset. Then the paper goes on to validate their method across 4 datasets (retinal ganglion cells, mouse v1, mouse retina, and monkey v4) and demonstrates a good generalization across species and visual areas. By using their method on mouse v1, the authors conclude that mouse v1 cell types may not have distinct functional types, which aligns with other findings in literature.", "questions": "This is overall a very strong paper, and I do not have many questions.\n\n1. May I suggest an additional experiment on varying the number of clusters (k ranging from 1 - 10, for example) on the retinal ganglion cell dataset? Plot ARI to see if it peaks at the true number of types. This would demonstrate that the method can recover a correct cluster count when discrete classes exist. This experiment would further support your claim that “We would expect ARI to peak noticeably at the ‘true’ number of clusters if such a structure existed. This suggests that mouse V1 likely lacks discrete functional cell types.”\n2. Do you think your method can be improved so that it can handle data across multiple subjects or sessions?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written, easy to follow, and well-motivated.\n* The paper proposes a novel clustering loss to address that the vanilla DEC loss fails to separate a simple toy dataset\n* The paper's algorithm is validated on a wide range of datasets, across species and visual areas. I like that the authors first validate the algorithm with a toy dataset for a sanity check, then go on to real datasets in increasing complexity (retinal ganglion cells, mouse v1, mouse retina, and monkey v4).\n\nWeaknesses:\n* As the authors have already discussed in the Limitations section of the paper, this algorithm requires tuning of different hyperparameters, including the number of clusters, the strength of the clustering loss, and the number of pertaining epochs. This could mean the algorithm is hard to use in practice.\n* The clustering algorithm does not seem to be able to handle data from different subjects/trials altogether.", "was_revised": true, "final_justification": "I appreciate that the authors have conducted an additional experiment on the retina data to demonstrate the validity of their approach. Overall, the authors have done thorough experiments and analyses, and I am particularly excited that the proposed method can lead to the conclusion that mouse v1 neurons may not have functional subtypes. I think this is a novel and meaningful contribution to the research community.", "rebuttal": "Thank you for appreciating our paper, noticing novelty and validation *“on a wide range of datasets, across species and visual areas”* and toy data.\n\nWe are happy to address you comments and questions below:\n### **Questions**\n**Q1**: *Additional experiment on marmoset retina*: Thanks a lot for suggesting an additional experiment with varying the number of clusters on the retinal ganglion cell dataset. The table below reports ARI values per cluster (and std across 3 seeds). We indeed observe a clear peak around 4 clusters, as it should be for this retina (Fig.3 in the original manuscript), so ARI does “peak noticeably at the ‘true’ number of clusters if such a structure exists.”\nWe will add this  experiment as a figure into the final version of the paper.\n\n| Clusters | ARI (mean ± std) |\n|----------|------------------|\n| 2        | 0.78 ± 0.07      |\n| 3        | 0.52 ± 0.31      |\n| 4        | **0.95 ± 0.01** |\n| 5        | 0.78 ± 0.08      |\n| 6        | 0.77 ± 0.15      |\n| 7        | 0.81 ± 0.08      |\n| 8        | 0.85 ± 0.05      |\n| 9        | 0.79 ± 0.04      |\n| 10       | 0.85 ± 0.07      |\n| 15       | 0.71 ± 0.07      |\n| 20       | 0.74 ± 0.06      |\n| 25       | 0.67 ± 0.04      |\n| 30       | 0.49 ± 0.02      |\n| 40       | 0.41 ± 0.04      |\n\n**Q2**: *Multiple sessions*: Actually, our method is capable of handling data from different subjects/trials altogether, in fact, it is already done for mouse visual cortex and macaque V4. We do indeed handle sessions separately for mouse and marmoset retinas due to the differences in the experimental conditions (presumably different temperature) which influences the speed of the responses so much that this would be the most trivial dimension to separate the neurons for different groups [1,2]. While several works try to address this issue and integrate several retinas into the same model [3,4,5], this is still an active area of research, and is not the main focus of our paper.\n\n### **Weaknesses**\n**W1** *Hyperparameters*: Our method indeed introduces additional hyperparameters, but not all of them need extensive tuning. Specifically, β just scales the clustering loss to the same order as the Poisson loss, which can be done with 1-2 short runs. However, finding the optimal pretraining epoch or number of clusters indeed increases the amount of necessary experiments. But since there are no competing methods which achieve the same without hyperparameter tuning, we believe that this degree of tuning is manageable and not unusual.\n\n**W2** *Multiple sessions*: See **Q2**\n\nWe thank you once again for constructive feedback and helping us to improve our paper. We are happy to answer any follow-up questions.\n\n**References**:  \n[1] Zhao, Z., Klindt, D. A., Maia Chagas, A., Szatko, K. P., Rogerson, L., Protti, D. A., ... & Euler, T. (2020). The temporal structure of the inner retina at a single glance. Scientific reports, 10(1), 4399.  \n[2] Fritsches, K. A., Brill, R. W., & Warrant, E. J. (2005). Warm eyes provide superior vision in swordfishes. Current Biology, 15(1), 55-58.  \n[3] Höfling, L., Szatko, K. P., Behrens, C., Deng, Y., Qiu, Y., Klindt, D. A., ... & Euler, T. (2024). A chromatic feature detector in the retina signals visual context changes. Elife, 13, e86860.  \n[4] Vystrčilová, M., Sridhar, S., Burg, M. F., Gollisch, T., & Ecker, A. S. (2024). Convolutional neural network models of the primate retina reveal adaptation to natural stimulus statistics. bioRxiv, 2024-03.  \n[5] Gonschorek, D., Höfling, L., Szatko, K. P., Franke, K., Schubert, T., Dunn, B., ... & Euler, T. (2021). Removing inter-experimental variability from functional data in systems neuroscience. Advances in Neural Information Processing Systems, 34, 3706-3719"}]

Author rebuttal:

Meta-review: This paper proposes a deep learning technique to train a predictive model of visual neurons of the brain while simultaneously optimizing for cell type clustering. The authors demonstrate that their model is effective in both predicting neural responses and clustering cell types, with evidence showing generalization across species and brain areas.

The reviewers provided thorough evaluations that identified both strengths and limitations of this work. While there are certainly areas for improvement (additional baseline models, better visualizations, etc.) the proposed method demonstrates clear value and effectiveness despite its straightforward approach. The dual optimization for neural prediction and cell type clustering is a meaningful contribution that addresses an important challenge in neuroscience.

Some reviewers expressed concerns about methodological limitations and presentation quality. However, after careful consideration of both the reviewers' comments and the authors' rebuttals, the core contributions appear sound and the experimental validation adequate. 

Given the significance of this problem and the demonstrated effectiveness of the proposed approach, I believe this work would provide value to neuroscience. The method offers a practical tool that researchers can incorporate into their analysis pipelines and test rigorously in diverse applications. While the work may represent an incremental advance rather than a major breakthrough, such contributions are essential for building the methodological foundation needed for progress in computational neuroscience.

Therefore, I recommend acceptance.
Decision: Accept (poster)