Most discriminative stimuli for functional cell type clustering
Authors: Max F Burg
Thomas Zenkel
Michaela Vystrčilová
Jonathan Oesterle
Larissa Höfling
Konstantin Friedrich Willeke
Jan Lause
Sarah Müller
Paul G. Fahey
Zhiwei Ding
Kelli Restivo
Shashwat Sridhar
Tim Gollisch
Philipp Berens
Andreas S. Tolias
Thomas Euler
Matthias Bethge
Alexander S Ecker
Venue: ICLR 2024 poster (2024)
Keywords: clustering
discriminative stimuli
interpretable
optimization
expectation-maximization
functional cell types
digital twins
feature visualization
pre-image search
maximally exciting image
Primary area: applications to neuroscience & cognitive science

Abstract: Identifying cell types and understanding their functional properties is crucial for unraveling the mechanisms underlying perception and cognition. In the retina, functional types can be identified by carefully selected stimuli, but this requires expert domain knowledge and biases the procedure towards previously known cell types. In the visual cortex, it is still unknown what functional types exist and how to identify them. Thus, for unbiased identification of the functional cell types in retina and visual cortex, new approaches are needed. Here we propose an optimization-based clustering approach using deep predictive models to obtain functional clusters of neurons using Most Discriminative Stimuli (MDS). Our approach alternates between stimulus optimization with cluster reassignment akin to an expectation-maximization algorithm. The algorithm recovers functional clusters in mouse retina, marmoset retina and macaque visual area V4. This demonstrates that our approach can successfully find discriminative stimuli across species, stages of the visual system and recording techniques. The resulting most discriminative stimuli can be used to assign functional cell types fast and on the fly, without the need to train complex predictive models or show a large natural scene dataset, paving the way for experiments that were previously limited by experimental time. Crucially, MDS are interpretable: they visualize the distinctive stimulus patterns that most unambiguously identify a specific type of neuron.

Published as a conference paper at ICLR 2024
MOST DISCRIMINATIVE STIMULI FOR FUNCTIONAL CELL TYPE CLUSTERING
Max F. Burg Thomas Zenkel45, Michaela Vystrčilova", Jonathan Oesterle
Larissa Höfting Konstantin F. Willeke 1,26, Jan Lause Sarah Müller Paul G. Fahey Zhiwei Ding Kelli Restivo&), Shashwat Sridhar Tim Gollisch10-12, Philipp Berens Andreas S. To.1 Thomas Fuler Matthias Bethge's, Alexander S. Ecker
International Max Planck Research School for Intelligent Systems, Tübingen, Germany 2 Institute of Com- puter Science and Campus Institute Data Science, University of Göttingen, Germany, Tubingen AI Center, University of Tubingen, Germany, 4 Institute of Ophthalmic Research, University of Tübingen, Germany, & Centre for Integrative Neuroscience, University of Tübingen, Germany, Institute for Bioinformatics and Medical Informatics, Tübingen University, Germany, Hertie Institute for AI in Brain Health, University of Tübingen, Germany, Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA," Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA, 10 University Medical Center Göttingen, Department of Ophthalmology, Germany 11 Bemstein Center for Computational Neuroscience Göttingen, Germany 12 Cluster of Excellence "Multiscale Bioimaging: from Molecular Machines Networks Excitable Cells" (MBExC), University Göttingen, Germany Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA, 14 Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany. Contact: max. bethgelab.org, ecker@ cs. uni-goettingen..ee
ABSTRACT
Identifying cell types and understanding their functional properties is crucial for unraveling the mechanisms underlying perception and cognition. In the retina, functional types can be identified by carefully selected stimuli, but this requires expert domain knowledge and biases the procedure towards previously known cell types. In the visual cortex, is still unknown what functional types exist and how to identify them. Thus, for unbiased identification of the functional cell types in retina and visual cortex, new approaches are needed. Here we pro- pose an optimization-basse obtain functional clusters of neurons using Most Discriminative Stimuli (MDS). Our approach alternates between stimulus optimization with cluster reassignment akin to an expectation- expectation-maximias maximization algorithm. The algorithm recovers fune- tional clusters in mouse retina, marmoset retina and macaque visual area V4. This demonstrates that our approach can successfully find discriminative stimuli across species, stages of the visual system and recording techniques. The resulting most discriminative stimuli can be used to assign functional cell types fast and on the fly. without the need to train complex predictive models or show large natural scene dataset, paving the way for experiments that were previously limited by ex- perimental time. Crucially, MDS interpretable: they visualize the distinctive stimulus patterns that most unambiguously identify specific type of neuron.
clustering approach using deep predictive models to
INTRODUCTION
Animals perceive the world through an intricate network of neurons in the visual system. These neurons are organized into cell types that can be identified by their distinct responses to visual stimuli. Such functional cell typing has been demonstrated in the mouse retina (Baden al., 2016), where functional cell types also match in their gene expression and morphology (Goetz al., 2022). In higher visual areas, it has proven difficult to identify cell types based on their response patterns, although recent attempts point at mix of continuous and discrete functional types in mouse primary visual cortex (V1) (Ustyuzhaninov et al., 2020; 2022) and primate cortical area V4 (Willeke et al., 2023).
Functional cell type identification currently often requires domain knowledge to handcraft suitable visual finger-printing stimuli (Farrow & Masland, 2011; Baden al., 2016; Franke al., 2017)

---

Published as a conference paper at ICLR 2024
Digital twin
Stimulus
Visual system
Activity
Prediction
Digital twin
Stimulus
Netron
MMA
Ama Ana
Neoun Neuran
Nevrmn?
AML
Mmm
Aurr
Maximally discriminative clustering algorithm
Cluster reassignment
MDS
orot
Cluster
Cluster 2
Adtivty
Adtivtyy
MDS
until converged
Optimize
Oplimizee
StmuuAs optimization
Neurons
Naurons
Neurons
Most discriminative stimulus (MDS) clustering based on digital twin. A. Digital twin model trained to mirror responses of neurons in the visual system. B. MDS clustering (Algorithm 1) iterates between optimizing MDS drive neurons within one cluster while suppressing all others and reassigning neurons to the cluster associated with the MDS they respond most to.
Figure 1:
and to select distinctive response features for classification (Farrow & Masland, 2011). This biases classification towards known cell types and might make it hard to discover new types. Also, in many brain areas the functional types are still unknown hence no "ground truth" exists to tailor stimuli towards. Therefore, the study of both well known and less understood visual areas would benefit from more neutral, data-driven approach that does not require domain knowledge. One idea would be to directly cluster cell's responses to unbiased naturalistic stimuli. However, the receptive field locations vary within simultaneously recorded population of neurons, which means that each cell would "see" a different part of the stimulus. Hence, even cells with identical input-output function will respond differently to the same stimulus, preventing direct clustering of their responses.
Fortunately, resolving this issue has recently come within reach: artificial neural networks have been proposed as "digital twins" of the visual system these networks receive visual stimuli as inputs and predict how a specific biological neuron would respond to that stimulus with high fidelity (Fig. IA; Antolik et al. (2016); Batty al. (2016); McIntosh Sinz et al. (2018); Walker et al. (2019); Lurz et al. (2020); Burg et al. (2021); Willeke et al. (2022); Ustyuzhaninov et al. (2022)). Such digital twins make it possible to analyze the computational properties of visual neurons in silico. For instance, digital twins allow to center a stimulus on each individual neuron's 's receptive field (Ustyuzhaninov et al., 2022) and to optimize it to maximally drive single neuron, yielding its Maximally Exciting Input (MEI), that visualizes the neuron's 's preferred feature (Walker et al., 2019; Bashivan et al., 2019). Bashivan et al. (2019) attempted to make MEIs more specific by optimizing images that do not only drive neuron, but also suppress all others. However, this is often not possible in practice, because multiple neurons of the same functional cell type are present, which respond similarly.
al. (2016); Klindt al. (2017);
If we knew which cells belong to the same functional type, we could easily extend this approach and find Most Discriminative Stimuli (MDS) at the cell type level stimuli that drive one cell type while suppressing all others when presented centered on each cell's receptive field. With the result- ing MDS, we can discover the unique visual features that the different cell types preferably process, rendering MDS to be efficient finger-printing stimuli for easy cell type identification in future ex- periments. As it is unknown which neurons belong which cell type priori, we need cluster cells into types and find the MDS for each type in a joint procedure.
Here, we tackle this task in an Expectation-Maximizaate Maximization (EM) style manner: We alternate be- tween assigning neurons to cell type clusters, and optimizing MDS for each cluster (Fig. 1B). We empirically validate our novel clustering algorithm by benchmarking the resulting clusters against well-established retinal ganglion cell (RGC) types (Baden et al., 2016; Field & Chichilnisky, 2007). We further demonstrate that our method found meaningful functional clusters when little domain knowledge is available, in cortical area V4. Our clustering works across species (mouse. mar- moset, macaque) and recording techniques (two-photon imaging, electrophysiology.. and leveraging MDS finger-printing stimuli can identify cell types 20% faster than traditional methods.
2

---

Published as a conference paper at ICLR 2024
Algorithm 1 Most discriminative clustering algorithm
Randomly assign neurons into clusters
while cluster assignments change do
> Initial clustering
M-step: Optimize MDS for all clusters with objective function Eq. (1)
E-step: Re-assign each neuron to cluster whose MDS drives it the most end while
while cluster assignments change do for each cluster de
> Determine the optimal number of clusters
Randomiy assign this cluster's neurons into candidate sub-clusters
Run EM clustering on sub-clusters until convergence
Keep candidate sub-clusters if they improve mean objective (Je)s and remove empty clusters end for end while
return Cluster assignments, most discriminative stimuli
2 RELATED WORK
Cell type clustering. Cell types form the building blocks of neural circuits across the brain (Sanes & Masland, 2015). To understand how these cell types perform computations in the visual system, a functional classification based on light responses may be most relevant (Vlasits et al., 2019). In the mouse retina, functional cell types have been studied extensively (Farrow & Masland, 2011; Baden et al., 2016; Franke et al., 2017), especially for retinal ganglion cells: RGCs be hierarchically divided by response polarity into ON, OFF and ON-OFF types, and then further by response dy- namies into transient and sustained types (Farrow & Masland, 2011). Using more complex stimuli, Baden et al. (2016) identified a finer cluster structure of at least 32 functional RGC types, many of which were later matched morphological and genetic types (Bae al., 2018; Tran et al., 2019; Goetz et al., 2022; Huang et al., 2022). As the Baden et al. (2016) functional clustering for RGCs is the most comprehensive available to date, we focus on their cluster labels in the retina experiments presented here. In cortex, functional cell types have been harder to define. For instance, the fin- gerprinting stimuli used in the retina rely mainly on temporal characteristics of full-field brightness changes or known properties such as direction selectivity probed with moving bars. How to find ap- propriate stimuli for the diverse spatial selectivity patterns observed visual cortex (Walker al., 2019; Ustyuzhaninov et al., 2022; Tong et al., 2023) is less clear. Ustyuzhaninov et al. (2020; 2022) modeled responses to natural images in mouse primary visual cortex (V1) with a convolutional neu- ral network (CNN), where each neuron's 's response is modeled by a linear readout from a shared feature space. They then used the readout weights as a vector representation of the neuron's 's func- tion. This approach captures the full selectivity of neuron, but it requires showing natural scenes and training a predictive model to infer a neuron's cell type, whereas our approach can identify a neuron's is cell type in new experiments quickly and without model training.
Image optimization. Optimizing input images has been used to maximize the response of units in artificial neural networks for object classification (Erhan et al., 2009; Zeiler & Fergus, 2014; Olah et al., 2017) and to distinguish between different classification models (Golan et al., 2020). Build- ing on these ideas, Deep Neural Network (DNN) models of neural activity were used to generate maximally exciting inputs (MEIs) for biological neurons. These MEIS can drive single neurons in vivo (Bashivan et al.. 2019; Walker et al., 2019; Franke al., 2022: Hoefling et al., 2022; Willeke et al., 2023; Pierzchlewicz et al., 2023), even when using multiple diverse MEIs (Ding et al., 2023); inhibit single neuron (Fu et al., 2023), or drive a single neuron but suppress others (Bashivan et al., 2019). While MEIs were recently extended to drive a predefined group of neurons (Ustyuzhaninov et al., 2022), there is no technique that at the same time suppresses the activity of all other neurons. Here, we provide this missing technique and extend it into a flel-fiedged clustering algorithm.
MOST DISCRIMINATIVE STIMULUS CLUSTERING ALGORITHM
Digital twins are models trained to predict neural activity based on shown stimuli. They allow us to search the stimulus space for most discriminative stimuli (MDS) that optimally separate functional groups of neurons in their activity (Fig. 1A). Here, we simultancously generated MDS and clustered neurons in an EM-like fashion (Fig. 1B and Algorithm 1). We started by randomly assigning neurons
3

---

Published as a conference paper at ICLR 2024
into a small number of initial "seed" clusters. Then we alternated between (M) optimizing an MDS for each cluster exclusively activating that cluster's ' neurons, while avoiding to activate other clusters, and (E) re- re-assigning neurons to the cluster whose MDS drove them the most (Fig. IB). Periodicallyy, we split candidate clusters into sub-clusters that we kept if this improved the objective (see below).
In each M step, we optimized MDS &e (centered on the neuron's receptive field) to maximize the average response F of all neurons j in the target cluster c, i.e. Fr(2)) = (r)()))) while suppressing the average response of all other clusters, Fu (1te). We maximized the softmax-base objective
exp (F.(H))/T)
max J. = max R2
log
R Σ
K exp (Fa(#e)/T)
where the temperature parameter determines how strongly we penalize large responses in other clusters (we set = 1.6 which is the best within broad optimum of the final objective, see Fig. 14), This yielded one MDS per cluster. the E step, we re-assigned neurons to the cluster associated with the MDS to which they responded most strongly. We then alternated between the E- and M-step until cluster assignments did not change anymore.
To find the optimal number of clusters, we devised a procedure to determine if adding more clusters enhanced our overall clustering. We treated each cluster independently and split its neurons into sev- eral new sub-cluster candidates. Then we performed the EM clustering procedure on the sub-cluster candidates followed by a global optimization step on all clusters yielding optimal MDS and cluster assignments, and we kept new clusters if the mean objective across all clusters (Je)e improved (see Appendix A.1 for details). During the splitting procedure, we removed any clusters not containing any neurons. We then repeated cluster splitting until it did not improve the objective anymore. To finalize clustering, we optimized all MDS until convergence followed by final reassignment step.
4 DATA AND IMPLEMENTATION DETAILS
Mouse retinal ganglion cells. We used a mouse retinal ganglion cell (RGC) dataset. A digital twin model for this dataset was previously published by Hoefling et al. (2022). The neurons in this dataset were stimulated with an ultraviolet (UV) and green channel naturalistic video (30 Hz frame rate). The digital twin was an ensemble of five CNNs that were trained to predict the time-varying RGC response with a 30 frame context window. Each member of the ensemble consisted of a core shared between all neurons and a neuron-specific linear readout (see Appendix A.2 for details). All models used in this study share this core-readout structure.
We assigned neurons to Baden et al. (2016) functional types using a classifier that predicts the cell type based on soma size and the light response to two synthetic fingerprinting stimuli, a "chirp" and a moving bar (Qiu et al., 2023). We removed neurons with less than 25% assignment confidence and all functional types containing less than 50 modeled neurons, as they might have been under- represented in digital twin training. Further, displaced amacrine cells, which usually part of this dataset, were removed as well. This led to 2,448 RGCs of 17 types. To ensure MDS are not overfit to the cells they were optimized on and that they also work in future experiments, we computed MDS on a 80% training split of the cells and report results on the remaining held-out test cells. Splits retained relative functional group sizes. These type labels are no perfect ground truth, as they are subject to potential misassignments by the classifier.
Stimulus optimization. We synthesized MDS with the same parameters that Hoefling et al. (2022) used to optimize MEIS and summarize the key components here. We initialized the stimulus, a 50 frames (18 x 16) pixels x2 channels video-snippet, with Gaussian white noise. We fed this stimulus to the digital twin predicting a 20 frames response trace for each cell. The average predicted activity over the last ten frames was the input into the objective function (Eq. (I)). We optimized the stimulus by gradient ascent for ten steps with learning rate of 100 (the optimal learning rate within the range [10-3, 3 10*)). During algorithm development we verified that sufficient optimization steps were performed by inspecting the objective values during optimization. To ensure that the MDS stayed within the pixel intensity range used when training the digital twin (Hoefling et al., 2022), we jointly renormalized both channels' pixel values of each frame to L2 Lorm of 30 after each optimization step. Additionally, we clipped the pixel values to the minimum (UV: -0.913, green: -0.654) and maximum (UV and green: 6.269) light intensity of the stimulus projector that
4

---

Published as a conference paper at ICLR 2024
displayed the videos, where zero corresponds to the mean luminance across the stimuli. This avoids that single pixels can exceed the display range of the projector and is a typical procedure in stimulus optimization settings (e.g. Walker et al. (2019); Hoefling et al. (2022); Willeke et al. (2023)).
Simulation of experiment. To estimate how time-efficient MDS are in determining a functional type, we did an in silico experiment with simulated observational noise. We modeled the trial-to- trial variability for each neuron by a gamma distribution whose mean we set to the digital twin's MDS response prediction for that neuron. We the variance be proportional estimated the proportionality constant for each neuron from a subset of the stimuli in the dataset for which responses to three repeated stimulus presentations were provided. Then we sampled from the gamma distribution to generate 16 "trials" of activity and computed the neural responses by averaging the activity over the last ten frames (as in the stimulus optimization procedure). Model neurons were then assigned to the cluster whose MDS activated them the most.
the mean and
Marmoset retinal ganglion cells. We additionally evaluated our approach on 167 marmoset RGCs obtained from Multi-Electrode Array (MEA) recordings of spiking activity in one retina while pre- senting a naturalistic gray-scale movie. For this data, no fine-grained reference taxonomy of func- tional types exists. We optimized the mouse digital twin (Hoefling et al., 2022) for this dataset (four CNN layers, larger 21 pixel spatial filters in the first layer, see Appendix A.3 for details).
Macaque visual cortex area V4. Additionally, we used third dataset of extracellular multi- electrode recordings from macaque visual cortex area V4, covering spiking responses of 1,224 neu- rons to gray-scale natural images shown to awake macaque monkeys (Willeke et al., 2023). We kept only the 1,030 neurons for which Willeke et al. (2023) provided MEI-based cluster labels. Note that these single cell MEI clusters do not represent ground truth, they were not verified with in- dependent biological measurements. Again, we sampled train (80%) and test (20%) splits retaining relative cluster sizes. For digital twin details and MDS optimization parameters, see Appendix A.4.
5 RESULTS
Mouse retinal ganglion cells. We started by empirically validating our novel clustering algorithm on the mouse RGC dataset that was labeled with 17 well-established Baden et al. (2016) functional types. Our MDS clustering algorithm found seven functional clusters after discarding small clusters with less than ten cells. Clusters were reasonably well separated in terms of their mean response (Fig. 2A) to the MDS (Fig. 2B). While our MDS clustering yielded fewer clusters than that by Baden et al. (2016). the clusters we found comprised the well known functional cluster hierarchy of the retina (e-g. Baden et al. (2016); Farrow & Masland (2011), 1)), namely OFF, ON-OFF, fast ON, and slow ON types, and one color-opponent type that was recently highlighted as having very distinct selectivity (Hoefling et al., 2022). Clusters differed in their preference for temporal frequencies, receptive field center sizes, strength of UV preference, and surround strength and structure (Fig. 2C).
Next, we compared the 17 Baden et al. (2016) types to our MDS clusters (Fig. 2D). We found that groups of Baden et al. (2016) types with similar functional properties matched well with our MDS clusters (color-coded labels in Fig. 2D based on clustering hierarchy in Baden et al. (2016)). Off- diagonal entries in Fig. 2A.D show that not all fine types could be perfectly separated by MDS. However, the block-diagonal structure (black boxes in Fig. 2D) indicates that cell types were only confused within groups of very similar function. All OFF and ON-OFF types (orange labels in Fig. 2D) mapped almost exclusively to MDS clusters 2 and 3, with a tendency of OFF cells mapping to cluster 2, and ON-OFF cells cluster 3. From the groups of Baden al. (2016) types, fast ON types ("ON loc tr OS", "ON trans", "ON high freq"; green in Fig. 2D) showed the clearest correspondence to a single MDS cluster, 38%, 93%, and 93% of cells of its three types mapping to MDS cluster 4, respectively. Slow ON types (blue in Fig. 2D) mapped mostly to clusters 5 and 6, with a preference for one of the two in most but not all types. In this group, one type ("ON DS sustained") even mapped exclusively to single MDS cluster. Despite its name, the "OFF suppressed I" type also mapped to ON MDS cluster 6, likely because this type has an ON response for some local stimuli (e.g. moving bar response in Baden et al. (2016)). The biggest exception of the slow ON types was "ON contrast suppressedd. which did not map strongly to either cluster or 6. but instead mapped to the color opponent MDS cluster 7 (Fig. 2D). Interestingly, this cluster showed the strongest one-to-one correspondence with a single Baden et al. (2016) type. The only other MDS cluster consisting of mostly one type was MDS cluster 1, which was dominated by
5

---

Published as a conference paper at ICLR 2024
C Functional properties
B MDS
Polarity (UV)
Polarity (green)
UV
T-freq. pref.
RF size
Time
UV
Green
pref.
(UV)
(green)
A Cluster mean responses to MDS
Simple
Simple
discciiinattiv stimulus
84
120
82
10
36
24
27
Ofi(Onn
Off[:On)
Simple
100
53
On(-Off)
On(-Off
Simple
24
28
100
57
Compl.
42
54
50
20
302
60
7-
Simple
Cluster mean response
Compl.
D Confusion with Baden et al. (2016) types
Confusion matrix of classifier predicting Baden et al. (2016) types
OFF loo 05 [14]
OFF slow [21]
10
ON-OFI [31]
ON-FFF DS [48]
50
55
(ONJOFF loo OS
ON lotor OS [118]
32
trens [26]
ON high teq [14]
feq [21]
ON sust [43]
alsha [38]
ONDS sust [14]-
200
ON sust 2 [12]
ON contr super [14]
ON 3 [13]
OFF [83]
OFF supor 2 [02]
33
lo 0S.08144 [1a]]]
OFF slow (2))
ONOFF lec [31]
ONOFF DS [48)
(ONJOFF loe OS (10]
ON to OS [118]
ON bonns (30)
ON high [14
ON feeq (211
ON sust (43)
DNDS sust [14]
ONDS pust [12]
ON contr auppr [14]
DS sust 3 [13)
OFF: euper [88)
OFF suppr (97)
[50] (10) (10][169](66]
ON mini aiphna
Types (2016)
Cluster [# neurans]
Type labels from Baden et (2016)
Figure 2: Most discriminative stimuli clustering mouse RGCs. Results held-out test neurons and empirical comparison to Baden et al. (2016) types shown. A. Cluster averaged digital twin response to the optimized MDS. Elements within column normalized to highest value. B. MDS video snippets per cluster decomposed into time, UV- and green-channel stimulus components by singular value decomposition. For display, spatial components were re-scaled to symmetric color scale between -1 (blue) and (red). C. Summary of the functional characteristics for each cluster (UV-pref: UV-preference, T-freq. pref.: Temporal frequency preference; Surr. Surround properties; RF: Receptive Field; +/ o /-: hhtihh//mdduu//ow)... D. Confusion matrix between MDS clusters and Baden et al. (2016) types (predicted by classifier for the mouse RGC dataset; Qiu al. (2023)). Annotations in same color belong to the same hierarchical functional type. Elements within rows were normalized to sum 1, annotated numbers display the number of neurons in percent. E. Confusion matrix of the classifier (Qiu et al., 2023) used to predict Baden et al. (2016) types for our dataset only having access the same type of information as MDS clustering between Baden al. (2016) type labels and predicted type labels on held out test data. Types confused by the classifier are merged into MDS clusters. Rows normalized (across Baden et al. (2016) types) to sum annotations displaying the number of neurons in percent.

---

Published as a conference paper at ICLR 2024
the "OFF suppressed 2" type, but also comprised other types, including both OFF and ON cells. Similar to "OFF suppressed 2", cells of "OFF suppressed I" could be found in most clusters. This is expected, given that both types likely comprise sub-types with distinct functional properties (see coverage factor in Baden et al. (2016)).
We further analyzed why most Baden et al. (2016) types mapped to two or more MDS clusters and why all MDS clusters comprised more than one Baden et al. (2016) type. One hint was that direction- and orientation selective Baden et al. (2016) types (DS and OS in Fig. 2D) were mixed into all three major cluster groups (colored in Fig. 2D), suggesting the clustering did not capture direction selectivity. Consistent with that idea, we found that none of the MDS exhibited significant lateral motion. This is likely because the used digital twin is limited in capturing direction selectivity by both architecture egpae-time-separablee convolution) and training data (sampling bias optical flow; Hoefling et al. (2022)). We note that this is limitation of the predictive model, not of our procedure. Also, the original Baden et al. (2016) types were constructed with additional information on direction selectivity of each cell that our MDS clustering did not have access to. The MDS clustering can therefore not distinguish cells based on their direction selectivity, and instead merged types with and without direction selectivity when they had otherwise similar functional properties. To confirm this, we re-created cell type labels for a fairer comparison using a reduced classifier. This classifier re-labeled cells using only information the MDS clustering also had access to, namely functional responses to "chirp" stimuli, but not direction selectivity or soma size. We found that the reduced classifier performed substantially worse (accuracy dropped from 79% 49% on test data) and also confused types that were merged by MDS clustering into single clusters (Fig. 2E). For instance, the reduced classifier confused OFF and ON-OFF types that were mapped to MDS clusters 2 and 3, the fast ON types which were mapped mostly to the single MDS cluster 4, and some of the slow ON types mapping to MDS clusters and (Fig. 2D,E). This indicates that without access to direction selectivity or morphological features (e.g. soma size) the finer Baden et al. (2016) types cannot be reliably distinguished. This also explains why our MDS clustering finds fewer clusters.
Robustness. To verify that our clustering algorithm returned consistent results across runs, we repeated MDS clustering with another random initialization of the stimulus and initial neuron as- signments into 30 instead of five clusters. Again the algorithm converged to seven clusters, which exhibited strikingly similar MDS as the run initialized with five clusters (Appendix Fig. 6). The sim- ilarity of cluster assignments across eleven comparison runs varying in initial number of clusters, initial neuron assignment, and MDS initialization was also high (median ARI: 0.67), suggesting MDS clustering is robust across initializations. Further, running MDS clustering for a different dig- ital twin architecture yielded similar results, suggesting robustness to the specific digital twin model used (see Appendix A.6 and Fig. 13 for details).
MDS provide time-ellicient on-the-fly cell type assignments. As MDS condense the unique response features of each cell type into a single, short stimulus, we next asked if they allow us to assign neurons to types faster than conventional methods. For fair com- parison with conventional approaches, we considered the clusters predicted by the classifier purely trained on the "'hirp"" responses (see above) as a baseline. For a comparable assessment between clusterings, we merged the 17 "conventional" clusters from the chirp-classiffer according to the Baden et al. (2016) hierarchy and matched them to the seven MDS clusters. The baseline classifier retrained on this task predicted the correct cluster label with 73.5% accuracy on held-out test set of neurons. Recording the required chirp responses (four stimulus repetitions) and assigning functional type labels for new neurons requires 2 min 12s s with this baseline classifier.
1.0
Fraction correct clasecti classified
Meen MDS accuracy
MDS repeats
Baseline clessifier
0.8
0.6
0.0
0.5
1.0
1.5
2.0
2.5
Presentation time (min))
Figure 3: Accuracy of iden- tifying MDS clusters under simulated observational noise for varying repeats. Presen- tation time for the full set of seven MDS. Mcan and standard deviation across ten simulations shown. MDS outperform the baseline after 1 min 45 (9 repeats).
We used the digital twin to simulate an experiment in which we want to obtain the functional type using only MDS. We showed each MDS video of 1.66 s length (50 frames) to each in-silico neu- ron and assigned them to the functional type of the MDS that elicits the highest simulated response. We simulated multiple MDS rep- ctitions to average out trial-to-trial variability. Repeating the MDS
7

---

Published as a conference paper at ICLR 2024
nine times outperformed the conventional approach in identifying the correct MDS cluster while saving 20% (27 sec) of experimental time compared to the chirp stimulus (Fig. 3). Note that identi- fying cell types in experiment requires only the pre-computed MDS. No digital twin training is required to use the MDS as finger-printing stimuli.
Marmoset retinal ganglion cells. Next, demon- strated that our approach is robust across species and recording techniques, using a dataset with the spiking responses of 167 marmoset RGCs to naturalistic videos recorded with multi-electrode arrays. MDS clustering yielded four functional clusters (Fig. 4). Even though the data contained only seven ON cells, the MDS algo- rithm recovered and even split them into slow and fast ON cluster (cluster 3 and 4, respectively). OFF cells were also grouped into a fast and slow cluster (clus- ter and 2. respectively). These clusters likely corre- spond to ON and OFF midget (slow) and parasol (fast) cells previously described in the primate retina (Field Chichilnisky, 2007) and are in good agreement with a clustering baseline based on the spike-triggered averages on white noise stimuli (see Appendix A.3 and Fig. 10).
Most disctiminative stimulus
Fast off
200
Slow
14
300
10
76
100
Slow
Fast on
Cluster mean response
Figure 4: MDS clustering on marmoset RGCs. Cluster averaged digital twin re- sponse to MDS normalized by mean pre- dictions across the twin's training stim- uli. Elements within column normal- ized to highest value. MDS space com- ponent displayed on symmetric color scale between -1 (black) and I (white).
We also verified that repeating MDS clustering with a different seed or initial number of clusters resulted in the same number of final clusters (Appendix Fig. 9). The similarity of clusters across nine comparison runs was also high (median ARI: 0.96). Although the number of cells is smaller and the taxonomy of functional types less refined for the marmoset retina, the results on this dataset show that MDS clustering finds meaningful clusters across species and recording techniques.
Macaque visual cortex area V4. Finally, we showed that our approach extends beyond the retina and also works for brain areas with little
37
20
46
100
27
37
13
22
38
25
22
11
18
47
domain knowledge about cell types. For that, we tested MDS clustering on spiking responses of neu- rons in macaque area V4, stimulated by gcay-scale images (Willeke et al., 2023). We discarded clus- ters containing less than six neurons in the test set, yielding 12 well separated MDS clusters (Fig. 5).
Most discriminative stimulus
13
14
100
15
42
41
22
28
36
10
11
21
15
100
38
10
18
31
23
10
39
41
14
59
100
30
25
22
26
10
15
29
26
10
59
19
56
100
49
55
44
1.
22
57
21
18
34
12
18
100
61
39
11
22
22
-9
16
21
60
100
60
10
18
16
Here, the MDS were individual images, which showed complex patterns with curvature or texture, typical for V4 (Bashivan al., 2019; Willeke et al., 2023). Interpreting these MDS suggests that V4 might compute such complex features on the cell type-level. Repeating clustering with different seed and ten instead of five initial clusters resulted in the same number of MDS clusters (Appendix Fig. 11) with reasonable similarity (ARI: 0.53). Our results suggest that the MDS algorithm also be successful in computationally more com- plex brain areas such primate V4.
16
48
300
18
45
100
14
100
10
12 MDS
Cluster mean response
Figure 5: MDS clustering on macaque V4 on held-out test neurons. Cluster averaged digi- tal twin response MDS normalized by mean predictions across the twin's 's training stimuli. Elements within column normalized high- est value. MDS displayed on symmetric color scale between -1 (white) and (black).
DISCUSSION
We presented a novel clustering algorithm that alternates between assigning neurons into func- tional clusters and optimizing most discriminative stimulus (MDS) for each functional cluster. We empirically validated our algorithm on three datasets across species, visual processing stages (mouse/marmoset RGCs, macaque V4) and data modalities (gray-scale and multi-channed image and video stimulation, multi-electrode array recordings and dalcium-imaginee and showed that MDS al- low to identify cell types faster than traditional methods.
8

---

Published as a conference paper at ICLR 2024
Comparison to other functional clustering methods. Willeke al. (2023) reported hints towards functional groups in macaque V4. They optimized a maximally exciting image for each single cell and clustered these MEls. In contrast, our algorithm computes MDS capturing unique functional properties that distinguish clusters of neurons based on their responses. In mouse V1, Ustyuzhaninov et al. (2022) clustered non-interpretable digital-twin model parameters they assume to represent neuron's 's fuunction In contrast, our algorithm clusters directly based on responses to interpretable MDS, not requiring expert knowledge extract parameters from the digital twin. Unlike both of these approaches, we empirically validated our clustering algorithm on data where type labels akin to biological ground truth exist (Baden al., 2016). In addition, once the cell types and their MDS have been established, our approach can identify neuron's type in new experiments without the digital twin, while both other approaches would need to retrain the digital twin. This suggests that MDS allow for fast online cell typing at recording time, enabling researchers to target a specific cell type they are interested in for data collection and experimental interventions.
Data-driven and interpretable cell type discovery with MDS. While previous clustering ap- proaches required domain knowledge to hand-craft fingerprinting stimuli (Farrow & Masland, 2011; Baden al., 2016; Goetz et al., 2022), approach identifies functional cell types by automatically generating a fingerprinting stimulus: the MDS. Therefore, our approach is particularly advantageous when domain knowledge is scarce and functional types yet be discovered, for example in V1 or - as we demonstrated - in V4: here, without incorporating domain knowledge, our clustering approach successfully identified plausible clusters.
The resulting MDS facilitate interpretability of a cluster's 's function, as it highlights a cluster's unique visual feature that distinguishes it from the features processed by other groups. For instance, the mouse RGC type "ON contrast suppressed" has been a subject of study for some time (Tien et al., 2015: Baden al.. 2016; Mani & Schwartz, 2017; Tien et al., 2022), but only recently Hoefling et al. (2022) found that this cell type responds strongly to center color opponency a property that was not revealed by previous fingerprint stimuli, but is clearly visible in our MDS. For complex visual features (e.g. in V4, Fig. 5), describing cluster's function in words might be more difficult and provides an interesting avenue for future work.
Limitations and future work. MDS clustering is applicable for any dataset rich enough to train digital twin, which is the case for many recent experimental studies (Yamins et al., 2014; Bashivan et al., 2019; Lurz et al., 2020; Sinz et al., 2018; Hoefling et al., 2022; Willeke et al., 2022: Tur- ishcheva et al., 2023; Wang et al.. 2023). While a systematic investigation of how the choice of a specific digital twin model affects downstream results is to date unavailable, our MDS and clusters were consistent with another digital twin architecture choice (Appendix A.6 and Fig. 13), suggesting certain degree of robustness to model details. Another promising direction for future work is ap- ply our method to new functional recordings where no comparison clustering exists and to validate with transcriptomic data, e.g. by testing experimentally which cells are driven by our MDS and subsequently identifying their transcriptomic types by PatchSeq (Liu et al., 2020) or spatial tran- scriptomics (Alon et al., 2021) in the same tissue. While the final judge of the MDS' faithfulness would be an in vivo experiment, single cell MEIs generated with the same optimizer and digital twins as we used were verified vivo (Willeke et al., 2023) and ex vivo (Hoefling et al., 2022), suggesting that our results might well generalize to an in vivo setting similar to other work on in vivo verification (Bashivan et al., 2019; Walker et al., 2019; Franke et al., 2022: Tong et al., 2023).
Algorithm extensions. ON-OFF RGCs are stimulated by both light increases and decreases. Solely optimizing a single, short MDS may not effectively capture both properties and thus fails to distin- guish ON-OFF from pure ON or pure OFF cells. To address this, method can be expanded to optimize multiple MDS for each cluster, showing promising initial results (see Appendix A.5 and Fig. 12). Additionally, identifying "OFF suppressed 2" cells, characterized by their high baseline fir- ing rate reduced through stimulation, might be improved by searching for a maximally suppressing stimulus. As currently set up. MDS clustering assumes discrete cell types, and future work could extend it towards accounting for recently reported continuity within or between cell types (Tasic al., 2016; Harris al., 2018; 88; Tasic et al., 2018; Stanley et al., 2020; Scala et al., 2021).
Conclusions. Our general-punn purpose functional cell type clustering algorithm and clusters' most dis- criminative stimuli could be a useful tool for the neuroscience community and spark further work on exploring functional groups across the visual system especially in areas where no fingerprinting stimuli are available - and help designing experiments previously limited by experiment time.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We will make our code, seeds, and used Python environment publicly available for reproducibility at www.//ccrr-oomoossiiiiiiiaaaeeae
Furthermore, we provide detailed method descriptions and parameters in sections Most discriminative stimulus clustering algorithm, Data and implementation details, and Appendix. For all results we quantified reproducibility of MDS clustering across runs with varying parameter initializations and we provide according visualizations in Figs. 6, 9 and 11.
ACKNOWLEDGMENTS
The authors would like to thank (in alphabetic order): Catie Nealley, Emmanouil Froudarakis, Federico D' Agostino, Gabrielle Rodriguez, Jiakun Fu, Katrin Franke, Kayla Ponder, Polina Tur- ishcheva, Saumil Patel, Taliah Muhammad, Tori Shinn. MFB, KFW, and SM thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS). This project has received funding from the European Research Council (ERC) under the European Union's 's Horizon Europe research and innovation program (Grant agreement No. 101041669). This publication was funded by the Deutsche Vorschungsgemminschaat (DFG, German Research Foundation)- Project-IID 432680300 SFB 1456. This work was supported by the German Research Foundation (DFG): SFB 1233, Ro- bust Vision: Inference Principles and Neural Mechanisms, TP4, project number: 276693517. MB and PB are members of the Machine Learning Cluster of Excellence "Machine Learning - New Perspectives for Science". funded by the Deutsche Forschungsgemeinschaatt (DFG, German Re- search Foundation) under Germany's 's Excellence Strategy EXC number 2064/1 - Project number 390727645. We thank the European Union (ERC, "Ne" ref. 101039115). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. We also thank the Hertie Foundation for support. This work was supported by the Tübingen AI Center. The work was supported by the Intel- ligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (Dol/BBC) contract number D16PC00003. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. AST acknowledges support from NSF NeuroNex grant 1707400. AST acknowledges support from National Institute of Mental Health and National Institute of Neurological Disorders And Stroke under Award Number U19MHI 14830 and National Eye Institute award numbers RO1 EY026927 and Core Grant for Vision Research T32-EY-002520-37. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U.S. Government. The authors gratefully acknowledge the computing time made available to them on the high-performance computers HLRN-IV at GWDG at the NHR Center NHR Göttingen.
REFERENCES
Shahar Alon, Daniel R Goodwin, Anubhay Sinha, Asmamaw T Wassie, Fei Chen, Evan R Daugh- arthy, Yosuke Bando, Atsushi Kajita, Andrew G Xue, Karl Marrett, et al. Expansion sequenc- ing: Spatially precise in situ transcriptomics in intact biological systems. Science, 371(6)28): eaax2656. 2021.
Ján Antolik, Sonja Hofer, James A Bednar, and Thomas D Mrsic-Flogel. Model constrained by visual hierarchy improves prediction of neural responses to natural scenes. PLoS Computational Biology, 12(6)ee104997) 2016.
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Proceed- ings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, SODA '07, pp. 1027-1035, USA, January 2007. Society for Industrial and Applied Mathematics. ISBN 978-0- 89871-624-5.
Tom Baden, Philipp Berens, Katrin Franke, Miroslav Román Rosón, Matthias Bethge, and Thomas Euler. The functional diversity of retinal ganglion cells in the mouse. Nature, 529(7586):355-- 350, January 2016. ISSN 0028-0836, 1476-4687. doi: 1038/nature16466 URL http:
//www. natura.trre..e. com/ aataaaiiilcesse att.aclls//tttreeeelelee 64
10

---

Published as a conference paper at ICLR 2024
J Alexander Bae, Shang Mu, Jinseop Kim, Nicholas L Turner, Ignacio Tartavull, Nico Kemnitz, Chris S Jordan, Alex Norton, William M Silversmith, Rachel Prentki, et al. Digital museum of retinal ganglion cells with dense anatomy and physiology. Cell, 173(5):1233-1366, 2018.
Pouya Bashivan, Kohitij Kar, and James J. DiCarlo. Neural population control via deep image synthesis. Science, 364(6439) May 2019. ISSN 0036-8075, 1095-9203, doi: 10.1126/ science.aav9436. URL https: /tsci://scccccee sciencemag orgg/ooneet////eeee /439/ eaav9436.
Eleanor Batty, Josh Merel, Nora Brackbill, Alexander Heitman, Alexander Sher, Alan Litke, EJ Chichilnisky, and Liam Paninski. Multilayer recurrent network models of primate retinal ganglion cell responses. In International Conference on Learning Representations, 2016.
Max F. Burg, Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Andreas S. Tolias, Matthias Bethge, and Alexander S. Fcker. Learning divisive normalization in primary visual cortex. PLOS Computational Biology, 17(6)ee009028, July 2021. ISSN 1553-7358. doi: 10.1371/journal. pebi.1009028. URL https:tttt.c :ttpp::///ouuaaaall//// plos plos. org /otpr//////aaa..//o pobi 1009028. Publisher: Public Library Science.
Santiago A. Cadena, Marissa A. Weis, Leon A. Gatys, Matthias Bethge, and Alexander S. Ecker. Diverse Feature Visualizations Reveal Invariances in Early Layers of Deep Neural Networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, volume 11216, pp. 225-240. Springer International Publishing, Cham, 2018. ISBN doi: 10.1007/978-3-0333 3-00-001258-- 8.14.
978-3-030-01257-1 978-3-030-01258-8 link. springer com/ 10tmm//...m 1007 978 - 3-030-0000---0 01000-8128888811 14.
URL http://
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network
learning by exponential linear units (clus). arXiv preprint arXiv:1511.07285 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 255. leee, 2009.
Zhiwei Ding, Dat Tran, Kayla Ponder, Erick Cobos, Zhuokun Ding, Paul G. Fahey, Eric Wang, Taliah Muhammad, Jiakun Fu, Santiago A. Cadena, Stelios Papadopoulos, Saumil Patel, Katrin Franke, Jacob Reimer, Fabian H. Sinz, Alexander Ecker, Xaq Pitkow, and Andreas Tolias. Bipartite invariance in mouse primary visual cortex. preprint, Neuroscience, March 2023. URL. wwwpxxiv.orr//ooo wwtpoorrg///ooo/ooooooooooo/ lookup/do /10. 1101 /20/22/222222222 03.. 532836.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
Karl Farrow and Richard H Masland. Physiological clustering of visual channels in the mouse retina. Journal of neurophysiology, 105(4):1516-1530 2011.
G.D. Field and E.J. Chichilnisky. Information processing in the primate retina: Circuitry and cod- ing. Annual Review of Neuroscience, 30(1);1-30, 2007. doi: 0011146/ammure..em.3000555555 094252. URL https: /ttes://doi..oe http//iioooog/.//////o/////o 1...1....... / /at/////nnvv...mm neuro. 30. 051 606 094252. PMID: 17335403.
Katrin Franke, Philipp Berens, Timm Schubert, Matthias Bethge, Thomas Euler, and Tom Baden. Inhibition decorrelates visual feature representations in the inner retina. Nature, 542(7642):439- 444, 2017.
Katrin Franke, Konstantin F. Willeke, Kayla Ponder, Mario Galdamez, Na Zhou, Taliah Muham- mad, Saumil Patel, Emmanouil Froudarakis, Jacob Reimer, Fabian H. Sinz, and Andreas S.
Tolias. State-dependent pupil dilation rapidly shifts visual feature selectivity. Nature, 610
(7930))128-133) October 2022. ISSN 1476-4687. doi: 10. 00.1038/44158--022-22 022-05270-3, URL
https: / www. nature. wwwm/a/tillee///ooooocooooao.eet s4158 022 05200-27720000 3. Number: 7930 Pub-
lisher: Nature Publishing Group

---

Published as a conference paper at ICLR 2024
Jiakun Fu, Suhas Shrinivasan, Kayla Ponder, Taliah Muhammad, Zhuokun Ding, Eric Wang, Zhiwei Ding, Dat T. Tran, Paul G. Fabey, Stelios Papadopoulos, Saumil Patel, Jacob Reimer, Alexan- der S. Ecker, Xaq Pitkow, Ralf M. Hacfner, Fabian H. Sinz, Katrin Franke, and Andreas S. Tolias. Pattern completion and disruption characterize contextual modulation in mouse visual cortex. preprint, Neuroscience, March 2023. URL http: htt////ioxxiiii org/ www../rrgglooooo/////ooooo lookup/do 10.1101/2023 03.13 13. 532473.
Jillian Goetz, Zachary F Jessen, Anne Jacobi, Adam Mani, Sam Cooler, Devon Greer, Sabah Kadri, Jeremy Segal, Karthik Shekhar, Joshua Sanes, al. Unified classification of mouse retinal ganglion cells using function, morphology, and gene expression. Cell reports, 40(2), 2022.
Tal Golan, Prashant C.
Raju, and Nikolaus Kriegeskorte. Controversial stimuli: Pitting neu- ral networks against each other as models of human cognition. Proceedings of the National Academy of Sciences, 11 117(47)29330-2933) November 2020. ISSN 0027-8424, 1091-6490. doi: 10. 1073//ass.. 1912334117. URL http: /www. pnas http://rrrgg.ooooo/o///o....... pnas. 1 91 212331117.
Kenneth D Harris, Hannah Hochgerner, Nathan G Skene, Lorenza Magno, Linda Katona, Carolina Bengtsson Gonzales, Peter Somogyi, Nicoletta Kessaris, Sten Linnarsson, and Jens Hjerling- Leffler. Classes and continua of hippocampal cal inhibitory neurons revealed by single-cell tran- scriptomics. PLoS biology, 16(6):c0003387, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp. 770-778, 2016.
Larissa Hoefling, Klaudia P Szatko, Christian Behrens, Yongrong Qiu, David Alexander Klindt, Zachary Jessen, Gregory S Schwartz, Matthias Bethge, Philipp Berens, Katrin Franke, Alexan- der Ecker, and Thomas Euler. A chromatic feature detector in the retina signals visual context changes. preprint, Neuroscience, December 2022. URL http: htttp://biorre http//iiooxiiiii//o orgg 1ookup/ doi /10//1... 11 111112222.2111111 11. 30. 5492...
Wanjing Huang, Qiang Xu, Jing Su, Lei Tang, Zhao-Zhe Hao, Chuan Xu, Ruifeng Liu, Yuhui Shen, Xuan Sang, Nana Xu, et al. Linking transcriptomes with morphological and functional phenotypes in mammalian retinal ganglion cells. Cell Reports, 40(1)) 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: // //arxiv.org/aaa
1412.41222 6980.
David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating what" and where". In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and Garnett (eds.), Advances in Neural Information Curran Associates, Inc., 2017.
Processing Systems 30, pp. 1506-35166
Steffen Krüppel, Mohammad H Khani, Dimokratis Karamanlis, Yunus C Erol, Sören J Zapp, Matthias Mietsch, Dario Protti, Fernando Rozenblit, and Tim Gollisch. Diversity of ganglion cell responses saccade-like image shifts in the primate retina. Journal of Neuroscience, 43(29): 5319-5339, 2023.
Jianwei Liu, Mengdi Wang, Le Sun, Na Clara Pan, Changjiang Zhang, Junjing Zhang, Zhentao Zuo, Sheng He, Qian Wu, and Xiaoqun Wang. Integrative analysis of in vivo recording with single-cell ma-seq data reveals molecular properties of light-sensitive neurons in mouse vl. Protein & cell, 11(6):417-332, 2020.
Konstantin- KonstantinKKlemens Lurz, Mohammad Bashiri, Konstantin Friedrich Willeke, Akshay Kumar Ja- gadish, Eric Wang. Edgar Y Walker, Santiago Cadena, Taliah Muhammad, Eric Cobos, Andreas Tolias, et al. Generalization in data-driven models of primary visual cortex. bioRxiv, 2020.
12

---

Published as a conference paper at ICLR 2024
Adam Mani and Gregory W Schwartz. Circuit mechanisms of retinal ganglion cell with stimulus- dependent response latency and activation beyond its dendrites. Current Biology, 27(4):471-49) 2017.
Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes, Advances in neural information pro- cessing systems, 29, 2016.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. till, 2017. doi: 10.23915/distil.000007 URL https://distiln /
Dis-
deature-visualizane
Pawel A. Pierzchlewiez, Konstantin F. Willeke, Ame F. Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas S. Tolias, and Fabian H. Sinz. Energy guided diffusion for generating neurally exciting images. bioRxiv, 2023. doi: 10.1 18. 18.51188111111 176. URL https://wwtt www.:wiioxiii.oom wwtpeeiwwwwwg//ooeooeeeeee wtt.://orgg/ooot//ttaaaeea 2023 /05 20/ 2023. 05. 111.111..1111...11 541176.
Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer, 2002.
Yongrong Qiu, David A. Klindt, Klaudia Szatko, Dominic Gonschorek, Larissa Hoefling, Timm Schubert, Laura Busse, Matthias Bethge, and Thomas Euler. Efficient coding of natural scenes im- proves neural system identification. PLOS Computational Biology, 101377, April 2023. ISSN 1553-7358. doi: 10. 10.177//juumal.ppb 101 URL https: pppo....................m.m....s wtt.rooowooooaatteeeeeeeeeeeeeee
pcbi.
wttppiii.i1..... 1011037.
Publisher:
Public Library of Science.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? ArXiv preprint arXiv:2007.08. 2020.
Joshua R Sanes and Richard Masland. The types of retinal ganglion cells: current status and implications for neuronal classification. Annual review of neuroscience, 38:221-246,, 2015.
Federico Scala, Dmitry Kobak, Matteo Bernabucci, Yves Bernaerts, Cathryn René Cadwell, Je- sus Ramon Castro, Leonard Hartmanis, Xiaolong Jiang, Sophie Laturnus, Elanine Miranda, Sha- laka Mulherkar, Zheng Huan Tan, Zizhen Yao, Hongkui Zeng. Rickard Sandberg, Philipp Berens, and Andreas S. Tolias. Phenotypic variation of transcriptomic cell types in mouse motor cortex. Nature, 598((779): 144-150, October 2021. ISSN 1476-4687 doi: 10. 110-88444586-000-2209900 24tlllee/e/18855
URL https:::////.. www. nature.
158 020- 02044441222---000 2907-3. 3.
Number: 7879
Publisher: Nature Publishing Group.
Fabian H. Sinz, Alexander S. Feker, Paul G. Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Xaq Pitkow, Jacob Reimer, and Andreas S. Tolias. Stimu- lus domain transfer in recurrent models for large scale cortical population prediction on video. preprint, Neuroscience, October 2018. URL htttp://biorxitte http://owww....oooooooooo/////// 11C1/452672.
Geoffrey Stanley, Ozgun Gokce, Robert C Malenka, Thomas Südhoff and Stephen R Quake. Continuous and discrete neuron types of the adult murine striatum. Neuron, 105(4):688-699, 2020.
Bosiljka Tasic, Vilas Menon, Thuc Nghi Nguyen, Tae Kyung Kim, Tim Jarsky, Zizhen Yao, Boaz Levi, Lucas T Gray, Staci A Sorensen, Tim Dolbeare, al. Adult mouse cortical cell taxonomy revealed by single cell transcriptomics. Nature neuroscience, 19(2):335-33) 346, 2016.
Bosiljka Tasic, Zizhen Yao, Lucas T Graybuck, Kimberly A Smith, Thuc Nghi Nguyen, Darren Bertagnolli, Jeff Goldy, Emma Garren, Michael Economo. Sarada Viswanathan, et al. Shared and distinct transcriptomic cell types across neocortical areas. Nature, 563(7729):72-78.. 2018.
Nai- Wen Tien, James T Pearson, Charles R Heller, Jay Demas, and Daniel Kerschensteiner. Genet- ically identified suppressed-by-cootrass by-contrast retinal ganglion cells reliably signal self-generated visual stimuli. Journal of Neuroscience, 35(30):10815 1-0800 2015.
13

---

Published as a conference paper at ICLR 2024
Nai-Wen Wen Tien, Carmela Vitale, Tudor C Badea, and Daniel Kerschensteiner. Layer-specific develop- mentally precise axon targeting of transient suppressed-by-conteeee retinal ganglion cells. Journal of Neuroscience, 42(38):7213-722) 2022.
Rudi Tong, Ronan da Silva, Dongyan Lin, Arna Ghosh, James Wilsenach, Erica Cianfarano, Pouya Bashivan, Blake Richards, and Stuart Trenholm. The feature landscape of visual cortex. bioRxiv, pp. 2023-11, 2023.
Nicholas M Tran, Karthik Shekhar, Irene E Whitney, Anne Jacobi, Inbal Benhar, Guosong Hong, Wenjun Yan, Xian Adiconis, McKinzie Arnold, Jung Min Lee, al. Single-cell profiles of retinal ganglion cells differing in resilience to injury reveal neuroprotective genes. Neuron, 104 (6):1039-1055, 2019.
Polina Turishcheva, Paul G Fahey, Laura Hansel, Rachel Froebe, Kayla Ponder, Michaela Vystrčilová, Konstantin F Willeke, Mohammad Bashiri, Eric Wang, Zhiwei Ding, et al. The dy- namic sensorium competition for predicting large-scale mouse visual cortex activity from videos. arXiv preprint arXiv:2305. /9654, 2023.
Ivan Ustyuzhaninov, Santiago A. Cadena, Emmanouil Froudarakis, Paul G. Fahey, Edgar Y. Walker, Erick Cobos, Jacob Reimer, Fabian H. Sinz, Andreas Tolias, Matthias Bethge, and Alexan- der S. Ecker. cotation-invariane clustering of neuronal responses in primary visual cortex. In International Conference on Learning Representations, 2020. URL https: net/ htte:/oormniid-rl13300
/ /ttp:////e.ereee..em
Ivan Ustyuzhaninov, Max F. Burg, Santiago A. Cadena, Jiakun Fu, Taliah Muhammad, Kayla Pon- der, Emmanouil Froudarakis, Zhiwei Ding, Matthias Bethge, Andreas S. Tolias, and Alexander S. Ecker. Digital twin reveals combinatorial code of non-linear computations in the mouse pri- mary visual cortex. preprint, Neuroscience, February 2022. URL httt::::/bboxxxi..rooo
lookup ht////i//10/ 10 111100222222 02 479884.
Anna L Vlasits, Thomas Euler, and Katrin Franke. Function first: classifying cell types and circuits of the retina. Current opinion in neurobiology, 56:8-15, 2019.
Edgar Y. Walker, Fabian H. Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G. Fahey, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience, 22(12): doi: 10. 1500088/41533--9-0000 019-993--1----111 -19-0517----
2060-2065, December 2019. ISSN 1097-6256, 1546-1726.
URL http: / www. natuuaruueen....... com/
Eric Y Wang. Paul G Fahey, Kayla Ponder, Zhuokun Ding, Andersen Change, Taliah Muhammad, Saumil Patel, Zhiwei Ding, Dat T Tran, Jiakun Fu, et al. Towards foundation model of the mouse visual cortex. bioRxiv, pp. 2023 03, 2023.
Konstantin F. Willeke, Paul G. Fahey, Mohammad Bashiri, Laura Pede, Max F. Burg, Christoph Blessing, Santiago A. Cadena, Zhiwei Ding, Konstantin- Lurz, Kayla Ponder, Taliah Muhammad, Saumil S. Patel, Alexander S. Ecker, Andreas S. Tolias, and Fabian H. Sinz. The Sensorium competition on predicting large-scale mouse primary visual cortex activity, June 2022. URL httl:t///// http///axvvvv/v/ org/ abs/ 62006 68666. arXiv:2206.0886 08666 [es, q-bio].
Konstantin F. Willeke, Kelli Restivo, Katrin Franke, Ame F. Nix, Santiago A. Cadena, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Alexander Ecker, Fabian H. Sinz, and Andreas S. Tolias. Deep learning-driven characterization of single cell tuning in primate visual area V4 unveils topological organization. preprint, Neuroscience, May 2023. URL http:// httt::/bbioxiv... wwt.:/..rglooooooeooooooo do1 /10. 10.1101100 20/2///2.22222222 05. . 540591.
D. L. K. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. DiCarlo. Performance- optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, (11(()::619--88) 8624, June 2014. ISSN 0027-8424, 1091- 1401112111.
6490. doi: 20.1073/mas.1403111 1073/pnas. 140311222
URL http:/ wwtpwwpwpp........ooowwotawt...p wtt.a/oooogg////////22/
14

---

Published as a conference paper at ICLR 2024
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland. September 6-12, 2014, Proceedings, Part 1 13, pp. 818-833. Springer, 2014.
Zhijian Zhao, David Klindt, André Maia Chagas, Klaudia P Szatko, Luke Rogerson, Dario A Protti, Christian Behrens, Deniz Dalkara, Timm Schubert, Matthias Bethge, et al. The temporal structure of the inner retina single glance. Scientific reports, 10(1))4399, 2020.
15

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 SUB-CLUSTER OPTIMIZATION DETAILS
To optimize MDS for sub-cluster candidates, needed avoid the trivial case where all neurons are in one cluster with the same MDS as before splitting. Thus, we first optimized MDS for the sub- clusters only and reassigned neurons after each stimulus optimization step. Hence, in the beginning of this optimization, neuron reassignments were done on noisy stimuli to break symmetry. After iter- ating this procedure for 50 steps, we performed one global optimization step considering all original clusters and the newly created sub-cluster candidates. To do so, we kept neurons' assignments to the clusters and randomly initialized the according MDS. We then optimized the MDS for all clusters and reassigned neurons based on them. Now, the neuron assignments and MDS accurately reflect the new optimal clustcring state. Then, we evaluated the mean objective across all clusters ((Je)es Eq. (1)) and kept the new sub-clusters if they improved overall clustering.
A.2 MOUSE RETINA DETAILS
Robustness check. As a simple robustness check of MDS clustering, we ran clustering with 30 instead of 5 initial clusters. The resulting clusters and average responses are visualized in Fig. 6. This run had a rather high adjusted rand index of 0.87, thus, we asked how similar the result shown in Fig. 2 would be compared to a typical run. Therefore, out of eleven comparison runs varying in initial number of clusters, initial neuron assignment, and MDS initialization, we compared to the run with median adjusted rand index (ARI: 0.67) and found overall good agreement (compare Fig. to Fig. 2).
Response traces. To analyze individual clusters it might be useful to inspect response traces of neurons. Fig. 8 shows mean responses for each MDS and cluster combination, for which response traces are grouped by Baden et al. (2016) RGC types.
Digital twin. The digital twin of the mouse retina was an ensemble of five identically structured convolutional neural networks (CNN) trained to predict inferred firing rates (from calcium imaging) of RGCs in response to dichromatic natural movies previously published by Hoefling et al. (2022). In this ensemble, each member consisted of a core module, that was shared between all neurons, and neuron specific readout module Klindt et al. (2017). The core implemented as CNN with two convolutional layers, each with 16 features. Both convolutional layers consisted of space-time separable 3D convolutional kernels, batch normalization layer and an ELU nonlinearity (Clevert et al., 2015). In the first layer, sixteen 2 X 11 X 11 x 21 (channels x height x width X frames) kernels were applied as valid convolution; in the second layer, sixteen 16 x 5 applied with zero padding along the spatial dimensions. The temporal kernels were parameterized as Fourier series. To account for inter- experimeetal variability affecting the speed of retinal processing, the model included a time stretching parameter trained for each recording separately Zhao et al. (2020). The readout module modeled the spatial receptive field (RF) of each neuron as a 2D isotropic Gaussian, parameterized as N(nz; My; o). The model's output, i.e. the predicted RGC responses, were implemented as an affine function of the feature maps of the core module, weighted by the spatial RF from the readout module, followed by softplus nonlinearity. During inference, predicted responses were averaged over all five members of the ensemble.
11 kernels were
Digital twin training. Each member was initialized with different seed and trained independently using the Adam optimizer (Kingma & Ba, 2015) minimizing a Poisson loss
N Lroisson Σ (f(m)) (n)) ((t)))))
where N is the number of neurons, and (r)) and ((1) are the measured and predicted firing rate of a neuron n for an input of duration t = 50 frames, respectively. The batch size was set to 32 and chunk size (number of frames for each element in the batch) to 50.
The training schedule was based on Lurz et al. (2020) and used early stopping (Prechelt, 2002) based on the validation set, consisting of 15 out of the 108 movie clips. If the mean correlation between predicted and measured neuronal responses failed to increase on the validation set during any five consecutive passes through the entire training set (epochs), the training was stopped and the model
16

---

Published as a conference paper at ICLR 2024
100
67
40
23
25
13
24
82
100
26
31
Most discriminative stimulus
49
81
100
70
67
61
40
4
46
54
75
100
76
64
41
5
15
18
30
42
100
44
82
6
29
42
69
60
81
100
76
7
11
14
22
44
100
1
2
5
6
7
[62]
[98]
[92]
[117]
[38]
[183]
[21]
Cluster mean response
[# neurons]
Figure 6: Mouse retina results of clustering run with 30 instead of initial clusters with another random initialization of stimulus and neuron assignments. The final number of clusters is lower than 30, as empty clusters are removed while running MDS clustering.
17

---

Published as a conference paper at ICLR 2024
86
13
MDS Clusters in compus artomn run
100
12
84
100
100
1.0
81
200
MDS-Clusters Fig.
(a)
(b)
[14]
show [21]
ON-OFF Tocal [31]
77
30
OK-OFF DE 1 [40]
70
12
ION IOFF locll 05 (16)
ON local trnn, [110]
Types from Bordon at. (2016) matureee nel
fre
aust
aipht [39]
ON DS sust 1 [14]
ON 05 sust 2 [12]
27
ON contrast suppr [14]
ON I5 [33]
OFF
OFF
Cluster [e nourons]
Figure 7: Results of typical clustering run (run with median ARI compared to results in Fig. 2). Results on held-out test neurons and empirical comparison to Baden et al. (2016) types shown. A. Cluster averaged digital twin response the optimized MDS. Elements within column normalized to highest value. Confusion matrix between neuron's 's MDS cluster assignment in Fig. 2 and the most typical comparison run. C. Confusion matrix between MDS clusters and Baden al. (2016) types. Elements within rows were normalized to sum to 1, annotated numbers display the number of neurons in percent.
18

---

Published as a conference paper at ICLR 2024
Each cluster's group
maximally discriminative sttrudtt
OFF slow
ON-OFF loc
DS
(ON)OFF loc 05
ON trans
ON high frea
ON low freq
DN sust
ON
ON DS sust
Saim
ON contr
ON DS sust
OFF
Stmm.
OFF suppr
5tm.
[60)
[88)
[104] Cluster
4
6[169]
[66]
neurons]
Figure 8: Mean traces of held-out test neurons in response to the most discriminative stimuli (left) grouped by the Baden al. (2016) RGC type. Group MEIs for each cluster are visualized on top of the plot. The extend of the x- and y-axis is shared across all plots with response traces.
19

---

Published as a conference paper at ICLR 2024
checkpoint performing best the validation set restored. This process of early stopping and weight restoring was repeated four times, each time reducing the initial learning rate of 0.01 by learning rate decay factor of 0.3.
A.3 MARMOSET RETINA DETAILS
Robustness check. As a simple robustness check MDS clustering, we compared to a typical comparison (the one with mean ARI compared to Fig. 4 selected across nine comparison runs varying in the number initial clusters, intial cluster assignment, and MDS initialization). The resulting clusters and average responses visualized in Fig. 9.
Most discriminative stimulus
100
90
60
25
2
7
100
40
12
3
100
47
3
21
82
100
1
2
3
4
Cluster mean response
Figure 9: Marmoset retina results of typical clustering run with five instead of ten initial clusters with a different random initialization of stimulus and neuron assignments. Responses normalized by mean of digital twin predictions across the twin's 's training set.
CNN with and readout structure similar to Hoefling et al. (2022). The core consisted of 4 layers with space-time separated kernels of shape 21 21 pixels (height width) for the spatial kernel and 27 frames for the temporal kernel in the first layer. In the following layers, the spatial kernel was of x 5 pixels size and the temporal kernel covered 5 frames. The number of channels increased from 8 the first layer to 16, 32 and 64 in subsequent layers. The kernels of the first layer were smoothed adding 2D Laplace filter in the spatial dimensions and ID Laplace in the temporal dimensions as a prior. In contrast to Hoefling et al. (2022) temporal kernels were not parameterized as Fourier series. In the readout, each cell's 's receptive field (RF) was modeled as an isotropic Gaussian. The response function was then modeled as affine function of the core's weighted feature maps at the RF positions, followed by a softplus. The feature map weight vector was regularized using Ll-norm to enforce sparsity.
Digital twin. The model trained predict marmoset RGC responses
Digital twin training. We trained the CNN using the Adam optimizer (Kingma & Ba, 2015) op- timizing the Poisson loss (Eq. (2)) 16 out of 20 available 5 minute trials. The batch size was 16 and we used early stopping on the correlation between predicted and recorded responses on the validation set which consisted of the 4 remaining trials. We reduced the learning rate by factor
20

---

Published as a conference paper at ICLR 2024
Confusion with STA-based clusters
1 [45]
2 [86]
95
0
0
STA-based clusters
[3]
67
# neurons]
[9]
09
11
0
[
[13]
233
0
[5]
20
[1]
0
100
[2]
[1351[7]
[4] [3]
MDS clusters
[# neurons]
Figure 10: Confusion matrix to STA-based clustering provided with the marmoset data. Elements within rows normalized to sum to 1, annotated number of RGCs in percent.
of 0.1 each time the validation correlation did not improve for 5 epochs. The initial learning rate was 0.008 and the minimal learning rate allowed was 8 10-5, We stopped training early if the validation correlation did not improve for 50 epochs.
MDS optimization. To optimize the MDS this dataset used ensemble of of the above described models varying in their initialization before model training. We also tailored a few of the parameters of MDS optimization this dataset, optimizing 39 frames (80 90) pixels one- channel MDS videos (85 Hz frame-rate) with a learning rate of 4 and chose ten initial clusters. Unlike for the mouse retina data. here we fed the predicted activity of only the last time-bin to the discriminative objective function (Eq. (1)). The MDS input was constraint with an I2-norm of 3.5 and the pixel values were clipped to stay within the range of and 1. We optimized the stimulus for a maximum of 10,000 optimization steps, and for 10 steps during splitting, or until an exponential moving average of the objective (Eq. (I)) did not improve for 100 optimization steps. During splitting, created two new sub-cluster candidates and terminated splitting after 100 iterations. As neuron's 's responses are not normalized in this data, we post-hoc normalized digital twin predictions such that for each neuron, predictions were standard-normin distributed across the digital twin's ss training stimuli, ensuring single cell responses do not dominate the within-cluster mean.
STA-based cell clustering. To be able to compare our MDS clustering of marmoset RGC's to an alternative clustering approach, we determined each cell's RF size the area within the 1.5 standard deviation contour of a Gaussian fitted to the cell's spatial component of the spike triggered average (STA) measured under spatio-temporal white-noise stimulation. The STA's temporal component and the RF size were taken as feature vector for each cell. The feature vectors were normalized and their dimensionality was reduced using Principal Component Analysis (PCA). We selected as many principal components necessary to explain 90% of the variance. We then used KMeans++ (Arthur & Vassilvitskii, 2007) on the reduced-ding dimeeeddddimms feature vectors to determine 4-6 initial clusters. These clusters were then manually curated ensure each cluster had similarly-sizz cells and tiled the retinal surface, resulting a total of nine clusters. However, the taxonomy of functional types in the marmoset is less understood compared to the mouse and concise interpretation of these clusters is not yet established, so they cannot be treated as ground truth. Comparing these clusters to our MDS approach yielded overall good agreement (Fig. 10).
Data recording. All marmoset data were obtained by recording RGC spikes extracellularly from isolated retina placed MEA, as explained in Krüppel al. (2023). Marmoset retinal tis-
21

---

Published as a conference paper at ICLR 2024
obtained immediately after euthanasia from animals used by other researchers, in cordance with national and institutional guidelines and as approved by the institutional animal committee of the German Primate Center and by the responsible regional government office (Niedersächsisches Landesamt für Verbraucherschutz und Lebeessmitellicherhhii,, permit number $3.19-42502-04-203455
A.4 MACAQUE VISUAL AREA V4 DETAILS
Robustness check. As a simple robustness check of MDS clustering, we ran clustering with 10 instead of initial clusters. The resulting clusters and average responses are visualized in Fig. 11.
100
48
25
27
21
6
17
29
22
3
15
36
100
36
12
43
26
6
25
18
15
6
13
46
62
100
29
50
38
11
42
33
37
6
18
12
21
26
100
40
37
13
19
20
29
8
38
Most discriminative stimulus
16
34
23
100
49
39
20
28
20
22
13
16
19
16
47
100
28
24
31
30
14
40
16
21
22
54
61
100
16
25
20
24
11
31
24
49
18
9
100
40
59
8
16
10
10
11
18
9
23
100
62
27
10
6
11
15
32
54
7
55
71
100
14
21
7
15
30
36
100
12
10
10
13
26
28
31
14
11
26
16
17
100
1
2
3
5
7
8
9
10
11
12
[6]
[7]
[45]
[6]
[9]
[8]
[9]
[13]
[6]
[13]
[6]
[11]
Cluster mean response
[# neurons]
Figure 11: Macaque V4 results of clustering run with 10 instead of 5 initial clusters with another random initialization of stimulus and neuron assignments. Responses normalized by mean of digital twin predictions across the twin's training set.
Digital twin. The digital twin of macaque V4 neurons, published by Willeke al. (2023), again followed the core-readout structure. The core the model consisted of a ResNet50 (He et al., 2016) which was adversarially trained ImageNet (Deng al., 2009) have robust visual representa- tions (Salman et al., 2020). In the digital twin's 's core, the first residual block layer 3 (layer-3.0) of the ResNet trained with image perturbation constraint 0.1 was selected to read out from, be-
22

---

Published as a conference paper at ICLR 2024
cause it yielded the highest predictive performance, compared to all other ResNet models and layers. The corresponding size of the output feature map was 1024. The model used (loffe & Szegedy, 2015). Lastly, the resulting tensor was rectified with ReLU unit to obtain the final nonlinear feature space. To predict the response of a single neuron from the feature space, Gaussian readout was used (Lurz al., 2020). For each neuron n, this readout learned spatial coordinates of the position of the receptive field on the output tensor and extracted a feature vector of length channels at this location. This extracted feature vector was then used in linear-nonlinear model to predict the neuronal response. To this end, an affine function of the resulting feature vector the chosen location was computed, followed by a rectifying nonlinearity, chosen to be an ELU (Clevert et al., 2015) offset by one (ELU(x)+ + 1) to make responses positive. The weight vector was L regulariized during training.
Digital twin training. The model was trained to minimize the Poisson loss (Eq. (2)) summed across neurons and computed between observed spike counts and the models' predicted spike counts with an additional L1 regularization of the readout parameters. Models were trained on the full dataset of n = 100 recording sessions with n = 1244 neurons and an image size of 100 by 100 pixels. After each epoch, the Poisson loss computed on the entire validation set. Early stopping was used as follows: If the validation loss did not improve for five consecutive epochs, we restored the weights with the lowest Poisson loss the validation set and down-scaled the learning rate by factor of 0.3 before training continued. After four early stopping steps were completed, the training was stopped.
MDS optimization. We tailored a few MDS optimization parameters the V4 dataset, optimizing single gray-scale (100 x 100) pixel images with a learning rate of 20, before each optimization step smoothed image gradients by convolving a Gaussian filter with standard deviation 2 and after each optimization step we normalized the image to a Ly-norm of 35. We optimized MDS for 666 steps, and for 1 step during splitting. We stopped MDS optimization early if the exponential moving average of the objective (Eq. (1)) did not improve for 100 optimization steps. Furthermore, during splitting we created two new sub-cluster candidates, and terminated splitting after 2 iterations when initializing with 10 clusters and 3 iterations when initializing with 5 clusters (theoretically allowing for the creation of 40 clusters each). As neurons' responses are not normalized in this data, we post- hoc normalized digital twin predictions such that for each neuron predictions were standard-normal distributed across the digital twin's 's training stimuli, ensuring single cell responses do not dominate the within-cluste mean.
A.5 EXTENSION TO TWO MDS PER CLUSTER
To investigate if our method could be extended to increase cluster granularity by optimizing multiple stimuli, performed proof-of-concepp experiment: optimized two MDS per cluster. The responses to both stimuli need to be aggregated before feeding them to the objective (Eq. (1)). Here, we chose the product of the responses across stimuli, soft and differentiable implementation of the "logical and" operation that requires neurons to respond to both stimuli. Initializing with five randomly assigned clusters for the mouse retina, we found similar result as when optimizing only one stimulus, with a crucial difference: cluster three now showed two different MDS with a sign-fiipped temporal UV dependence (Fig, 12A). Only ON-OFF cells would respond both of these stimuli, and consequently the neurons of this new cluster were mainly associated with ON- OFF Baden al. (2016) types (Fig. 12B). Interestingly, the MDS also showed orientation-selective features, and hence the according cluster also contained orientation-selectiv Baden et al. (2016) types.
While this proof-of-concept is very promising, the extension to multiple stimuli raises number of new questions. For instance, there are several ways of aggregating responses from multiple stimuli (for example taking the mean, the minimum, the maximum, etc.). Further, optimizing more than two stimuli could be beneficial, and adding a diversity term between stimuli (Cadena et al., 2018; Ding et al., 2023) could improve clustering. Such extensive expansions are beyond the scope of the current paper, and we propose them as a fruitful avenue to further improve optimization- based functional clustering.
23

---

Published as a conference paper at ICLR 2024
MDS
OFF local, 0S [14]
OFF slow [21]
ON-OFF local [31]
ON-OFF (48)
Types from Baden et al. (2016) [# neurons]
local, [16]
100
67
41
25
26
20
local trans, OS [118]
ON trans (28)
100
30
28
ON high freq [14] -
Mest discrininative stmakun
ON low freq [21] -
sust [43]
alpha [39]
39
48
85
100
69
82
[14)
DS sust 2 [12]
ON contrast [14]
000
40
70
67
100
73
ON DS sust 3 [13]
OFF suppr. 1 (83)
16
31
33
14
19
28
70
100
suppr. 2 [92]
3
6
[82] (68)11400(67) (P710125]156)
[82]
[68]
[1401
[471
1771
[125]
[561
response
[M neurons]
[# neurons]
(a)
(b)
Figure 12: Proof of concept of clustering based two MDS on mouse RGCs. Results on held-out test neurons shown. A. Cluster averaged digital twin response the optimized MDS. To obtain the response for cell, the product of the responses to each individual MDS is computed. Elements within column normalized to highest value. B. Confusion matrix between MDS clusters and Baden et al. (2016) types. Elements within rows were normalized to sum to 1, annotated numbers display the number of neurons in percent.
24

---

Published as a conference paper at ICLR 2024
AOPPOOO DOSTA
Teues RNo Rodben 12EAN
high
100
13
25
24
22
220
54
58
11
50
OFT supe 1000
220
2000
1791
uatter respanss
(a)
(b)
comparison
MDS Fig.
(c)
Figure 13: Most discriminative stimuli clustering on mouse RGCs for another digital twin model architecture. Results held-out test neurons and empirical comparison Baden al. (2016) types shown. A. Cluster averaged digital twin response to the optimized MDS. Elements within a column normalized to highest value. B. Confusion matrix between MDS clusters and Baden al. (2016) types. Elements within rows were normalized to sum to 1, annotated numbers display the number of neurons in percent. C. Confusion matrix between the MDS cluster assignments for the modified digital twin architecture and MDS cluster assignments reported for the original digital twin used for Fig. 2. Elements within rows normalized to 1, annotated numbers display the number of neurons in percent.
25

---

Published as a conference paper at ICLR 2024
Modian
1.0
lised
Highnsllowwd value
clurters
Final obyee Coee
0.0
0.7-
0.0250.05
16
32
6.4
128
25.6 512210.4
Figure 14: Mouse retina clustering results objective temperature. For each temperature value, we optimized three runs varying in stimulus and cluster assignment initialization.
A.6 RESULTS FOR ANOTHER MOUSE RETINA DIGITAL TWIN ARCHITECTURE
To investigate how the choice of digital twin architecture affects the clustering results, modified the mouse retina digital twin model (Appendix A.2) by adding a third layer and varying the size of the convolutional kernels. Specifically, we changed the kernels of the second layer to sixteen x 3 x x (channels x height width x frames) kernels, and introduced a third layer of sixteen 16 3
kernels, both applied with zero padding along the spatial dimensions. We trained five of these digital twins initialized with different random seeds and used the same training schedule Hoefling et al. (2022) (see Appendix A.2 for details). We applied clustering to this ensemble of digital twins. The resulting clusters and MDS looked similar for both digital twin neural network architectures (compare Fig. 2 to Fig. 13), suggesting that while details may differ across digital twin models, the overall result remains the same.
MDS
A.7 ANALYSIS OF TEMPERATURE HYPER-PARAMEN PARAMETER
We asked how the choice specific temperature would affect MDS clustering. For the mouse retina, systematically swept temperatures logarithmically and MDS clustering for three dif- ferent initializations. We found a broad, flat objective optimum for temperature values between 0.8 and 12.8 with consistent number of five to seven resulting clusters (Fig. 14). In general, all cluster- ing algorithms will slightly over- or under-cluster, and to determine precise number of cell types, functional clusters need to be verified by comparison with morphological, genetic, transcriptomic data. The analyses in our paper are based on temperature 1.6 with the highest objective value, re- sulting in seven clusters that we successfully matched an existing hierarchy of cell types (Baden et al., 2016).
For lower or higher temperatures, the objective value and the number of clusters we found decreased, indicating worse clustering performance. This results directly from the structure of our softmax objective function (Eq. (I)). For high temperatures, T >> Fx(*x)) objective Fx (x:)/T are dominated by the temperature, Hence, the objective becomes insensitive to stimulus- dependden inputs FA (xe), and clustering performance decreases. For small temperatures, < F& (xz), the inputs will be strongly amplified and the response of single cluster Fk- (zc) will dominate the objective, making insensitive to the inputs of all other clusters Fkyee (æz). The resulting MDS will then not suppress all clusters. In conclusion, good temperature balances these two undesirable regimes.
1..,...,................ C.... the inputs
26


Reviews:
[{"review_id": "6KM80G8ky7", "reviewer": "Reviewer_69wH", "summary": "In this paper, the authors developed a framework to jointly cluster cells into functional cell types and obtain Maximally Discriminative Stimuli (MDS) for each functional cluster, based on an EM-type iteration. The MDS aims to stimulate one cell type while suppressing all others, as an extension to previously developed Maximally Exciting Inputs (MEIs). The authors carried out several real data analysis to demonstrate the capability of the proposed framework.", "questions": "1. For the sub-cluster split, the authors used an \"evaluation MDS\" for determining if clusters should be kept. This seems to be a newly initialized, separately trained MDS which does not necessarily tell whether the sub-clusters are bringing in better cell type identification. I hope the authors can explain more about the logic of this.\n2. Can the authors elaborate more on Figure 2B? They referenced figure 2B when stating in the text that “none of the MDS exhibited direction selectivity”, but I didn’t quite understand why Figure 2B is referenced here.\n3. For the experiments in Figure 2, the identified MDS seems to be largely combination of several functional cell types from the published Baden et al (2016) paper. I wonder if further tweaking with the sub-clustering procedure can better recover the published functional cell types.\n4. How will the result change if the number of gradient ascent steps is changed during each M-step? Trying to understand if there is a delicate balance needed between how well the M-step is optimized and the E-step, or if the procedure is relatively robust and almost guaranteed to converge to the same set of results.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is relatively easy to follow and well-structured. The method seems to be straightforward and intuitive. The authors provided multiple experiments to demonstrate the performance of the proposed framework.", "weaknesses": "My primary concern is how useful and interpretable the method will be for actual practice. The experiments essentially treated a carefully-examined existing publication as the ground truth to compare with. I assume in a lot of real scenarios, we might already have this kind of biological baselines. For cases where they are not available, the interpretability of the identified MDS might be important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YeKlbmK7DE", "reviewer": "Reviewer_FrCb", "summary": "Identifying and understanding cell types and their functional properties is crucial for understanding perception and cognition. Traditional methods have limitations, such as bias and lack of knowledge about functional cell types. A new approach, Maximally Discriminative Stimuli (MDS), uses optimization-based clustering with deep predictive models to identify functional cell types. This approach is successful across species and recording techniques and provides real-time cell type assignment, making experiments more efficient. MDS is interpretable and visualizes distinctive stimulus patterns for each neuron type.", "questions": "While the authors express their belief in the usefulness of the algorithm and stimuli, there is no mention of empirical results or validation studies that demonstrate the actual effectiveness and applicability of the proposed approach. \n\nOne potential weakness of the clustering algorithm is its limitation in effectively distinguishing certain cell types with complex responses. For instance, for ON-OFF RGCs that respond to both light increments and decrements, optimizing a single, short MDS to maximize their response may not adequately capture their unique properties. Similarly, for cell types like \"OFF suppressed 2,\" which exhibit a high baseline firing rate mostly suppressed by stimulation, the algorithm may not effectively identify them without specific adaptations, potentially requiring additional stimulus optimization strategies for better classification.\n\nThe absence of a benchmark is a significant weakness in this paper. It would be helpful if the author can add any performance to determine its effectiveness or limitations.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* MDS provides a time-efficient on-the-fly cell type assignment by using a concise stimulus.\n* MDS outperforms conventional approaches in identifying the correct cell type cluster, saving 20% of experimental time compared to traditional methods.", "weaknesses": "A potential weakness in the presented approach is that it assumes that the most informative stimuli for classifying cell types can be automatically chosen without requiring domain knowledge or expert input. While this is presented as an advantage, it may also be a limitation, as there could be cases where domain-specific insights are necessary for more accurate and nuanced cell type classification. Additionally, the success of the approach relies on the availability of a \"digital twin\" dataset, which may not always be readily available for all experimental studies, potentially limiting its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Y1zvYBIPn", "reviewer": "Reviewer_KEew", "summary": "This paper proposes maximize discriminative stimuli (MDS) that maximizely activates each functional cell types, and not activate the other types, and it is an extension of MEI (maximize excitation inputs). The algorithm follows the similar way as expectation maximization (EM), start from random assignment of groups, and maximize the activation objective in M step, and then reassign groups in E step. The total number of cell type groups is not defined as a prior, depends on the coverage and threshold of the EM algorithms. The work is evaluated on multiple real datasets and different animals (mouse retina, marmoset retina, and macaque V4). The proposed method is also helpful to provide time-efficient cell type identification.", "questions": "1. How does functional cell types here related to neuron classes of excitatory and inhibitory?\n2. How to explain and interpret some off-diagonal entries in Fig 2D and Fig 4A?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty and Importance:\n1. The paper focus on a novel and important concept that finding inputs that maximize the difference of different functional cell types, which extend from single neuron MEI to group level. \n\nEvaluations:\n1. The work has been comprehensively evaluated on multiple experiments across multiple animals to demonstrate the soundness of the proposed approach. Sensitivity analysis including different initialization, number of clusters are included.\n\nWriting:\n1. The paper is well organized and presentation is good.", "weaknesses": "Method:\n1. The clustering approach is based on the discreteness hypothesis of functional cell types, while ignoring the neurons might be close to the boundary or shows patterns that belongs to two functional cell types in different trials. Therefore, there might be limitation of separability.\n2. It remains unclear how does the choice of neural networks or back-propagation algorithms affect the solutions. \n3. Qualitatively, it is difficult to compare and interpret different MDS, and understand how they diverge from each other.\n\nBaselines:\n1. Simple baselines like searching nearest neighbor in original image space, or simple image interpolation could be introduced and compared. \n\nEvaluation:\n1. It seems that the optimized MDS has not been tested on real mouse yet, only evaluated on holdout neurons and trials.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cvsbsLFcXs", "reviewer": "Reviewer_NLuU", "summary": "An optimisation-based clustering approach is proposed obtain functional clusters of neurons using deep predictive models and Maximally Discriminative Stimuli (MDS). The stimulus optimization and cluster reassignment are alternately conducted until the convergence is achieved. Empirical studies demonstrate the proposed algorithm recovers functional clusters in mouse retina, marmoset retina and macaque visual area V4.", "questions": "1. Where does the digital twin (i.e. the neural network) come from? Is it a pre-trained neural network?\n2. How are the results/performance sensitive to the hyper-parameters and initializations?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "++ The paper attempts to address an important problem of cell type identification.\n\n++ The experimental results seem to be promising and useful in practice.", "weaknesses": "-- Essential technical details of the digital twin (i.e. the neural network used to predict the neuron responses) are missing. From the text, it is not clear how the digital twin is obtained and how it will affect the optimisation of MDS.\n\n-- It is stated in the paper \"We optimized the stimulus by gradient-ascent, using the SGD optimizer with a learning rate of 100\". What are the learnable parameters during this optmization? It is unusual to use such a large value of learning rate. \n\n-- The statement is ambiguous \"After each step we renormalize the video snippet to a L2 norm of 30 and clamp ... into ranges of [-0.913, 6.269] and [-0.654, 6.269]\". What does it mean by \"a L2 norm of 30\"? What are the ranges from?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)