Rotation-invariant clustering of neuronal responses in primary visual cortex
Authors: Ivan Ustyuzhaninov
Santiago A. Cadena
Emmanouil Froudarakis
Paul G. Fahey
Edgar Y. Walker
Erick Cobos
Jacob Reimer
Fabian H. Sinz
Andreas S. Tolias
Matthias Bethge
Alexander S. Ecker
Venue:  (2020)
Keywords: computational neuroscience
neural system identification
functional cell types
deep learning
rotational equivariance

Abstract: Similar to a convolutional neural network (CNN), the mammalian retina encodes visual information into several dozen nonlinear feature maps, each formed by one ganglion cell type that tiles the visual space in an approximately shift-equivariant manner. Whether such organization into distinct cell types is maintained at the level of cortical image processing is an open question. Predictive models building upon convolutional features have been shown to provide state-of-the-art performance, and have recently been extended to include rotation equivariance in order to account for the orientation selectivity of V1 neurons. However, generally no direct correspondence between CNN feature maps and groups of individual neurons emerges in these models, thus rendering it an open question whether V1 neurons form distinct functional clusters. Here we build upon the rotation-equivariant representation of a CNN-based V1 model and propose a methodology for clustering the representations of neurons in this model to find functional cell types independent of preferred orientations of the neurons. We apply this method to a dataset of 6000 neurons and visualize the preferred stimuli of the resulting clusters. Our results highlight the range of non-linear computations in mouse V1.

Published as a conference paper at ICLR 2020
ROTATION-INVARIANT CLUSTERING OF NEURONAL RESPONSES IN PRIMARY VISUAL CORTEX
Ivan Ustyuzhaninovy 1-3 Santiago A. Cadena, 1-3 Emmanouil Froudarakis," 45 Paul G. Fahey, Edgar Y. Walker, Erick Cobos, 4,5 Jacob Reimer, Fabian H. Sinz, 4,5 Andreas S. Tolias, 1,46 Matthias Bethge, 1-3,5,f Alexander S. Ecker
1 Centre for Integrative Neuroscience, University of Tübingen, Germany
2 Bernstein Center for Computational Neurosience, University of Tübingen, Germany
Institute for Theoretical Physics, University of Tübingen, Germany
4 Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA
5 Center for Neuroscience and Artificial Intelligence, BCM, Houston, TX, USA
6 Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA
I Authors contributed equally
$ Present address: Department of Computer Science, University of Göttingen, Germany
ecker www.::/sss.iigottttnnnn.e...ee
ABSTRACT
Similar to convolutional neural network (CNN), the mammalian retina encodes visual information into several dozen nonlinear feature maps, each formed by ganglion cell type that tiles the visual space in an approximately shift-equivariant manner. Whether such organization into distinct cell types is maintained the level of cortical image processing is an open question. Predictive models building upon convolutional features have been shown provide kute-of-the-ar perfor- mance, and have recently been extended to include rotation equivariance in order to account for the orientation selectivity of neurons. However, generally no di- rect correspondence between CNN feature maps and groups of individual neurons emerges in these models, thus rendering it an open question whether neurons form distinct functional clusters. Here we build upon the rotation-equivarti resentation a CNN-based V1 model and propose a methodology for clustering the representations of neurons in this model to find functional cell types indepen- dent of preferred orientations of the neurons. We apply this method to a dataset of 6000 neurons and visualize the preferred stimuli of the resulting clusters. Our results highlight the range of non-linear computations in mouse V1.
rep-
Fit rotation-equivariant CNN
2
Align readouts
3 Cluster aligned readouts
Feature weights - clusters
Aligned readouts
fra
Raw readout
())))
CNN
((=)
(.(
Orbit under eyclic
8N
Pa
(m)
tions
Figure I: An overview our approach. ! Fit rotation rouutiin-equivaaiiatt CNN predict neural responses and use readout vectors r; as proxies for neural computations. different preferred orientations. Cluster the aligned readouts.
② Align readouts to account for

---

Published as a conference paper at ICLR 2020
1 INTRODUCTION
A compact description of the nonlinear computations in primary visual cortex (V1) is still elusive. Like in the retina (Baden et al. 2016 Sanes & Masland, 2015), such understanding could come from a functional classification of neurons. However, is currently unknown if excitatory neurons in VI are organized into functionally distinct cell types.
It has recently been proposed that predictive models of neural responses based on convolutional neu- ral networks could help answer this question (Klindt et al. 2017 Ecker et al., 2019). These models are based on a simple principle (Fig. [ ): learn a core (e.g. a convolutional network) that is shared among all neurons and provides non features (().. which are turned into predictions of neural responses by a linear readout for each neuron (Antolík et al.) (016). Models based on this basic ar- chitecture exploit aspects of our current understanding of VI processing. First, convolutional weight sharing allows us to characterize neurons performing the same computation but with differently lo- cated receptive fields by the same feature map (Klindt et al.] 2017; McIntosh et al. 2016 Kindel jet al.] 2219 Cadena et al.) 2019). Second. V1 neurons can extract local oriented features such as edges at different orientations, and most low-level image features can appear at arbitrary orienta- tions. Therefore, [Ecker et al.] (2019) proposed a rotation-equivariant convolutional neural network model of VI that extends the convolutional weight sharing to the orientations domain.
The basic idea of previous work (Klindt et al., 2017 Ecker et al. 2019) is that each convolutional feature map could correspond to one cell type. While this idea is conceptually appealing, it hinges on the assumption that V1 neurons are described well by individual units in the shared feature space. However, existing models do not tend to converge to such solutions. Instead, VI neurons are better described by linearly combining units from the same spatial location in multiple different feature maps (Ecker et al. 2019). Whether or not there are distinct functional cell types in V1 is therefore still an open question.
Here, we address this question by introducing clustering method on rotation -aattion-quuvariantt spaces. We treat the feature weights (Fig. [+() that map convolutional features to neural responses as an approximate low-dimensional vector representation of this neuron's 's input-output function. We then split neurons into functional types using two-stage procedure: first, because these feature weights have a ootatio--quivariinn
structure, we find an alignment that rotates them into a canonical orientation (Fig. 22) in a second step, cluster them using standard approaches such k-means or Gaussian mixture models (Fig. [3) ). We apply our method to the published model and data of Ecker et al. (2019) that contains recordings of around 6000 neurons in mouse V1 under stimulation with natural images. Our results suggest that VI neurons might indeed be organized into functional clusters. The dataset is best described by GMM with around 100 clusters, which are to some extent redundant but can be grouped into a smaller number of 10-20 groups We analyse the resulting clusters via their maximally exciting inputs (MEIs) (Walker et al. 2018) to show that many of these functional clusters do indeed correspond to distinct computations.
2 RELATED WORK
Unsupervised functional clustering via system identification As outlined in the introduction, our work builds directly upon the methods developed by Klindt et al. (2017) and Ecker et al. (2019). While these works view the feature weights as indicators assigning each neuron to its 'cell type' (feature map), we here take different view on the same model: rather than focusing on the convo- lutional features and viewing them as cell types, we treat the feature weights as a low-dimensional representation of the input-output function of each neuron and perform clustering in this space. This view on the problem has the advantage that there is no one-to-one correspondence between the number of feature maps and the number of cell types and disentangle model fitting from its inter- pretation. On the other hand, our approach comes with an addition complexity: because the feature weights obey rotational equivariance and we would like our clustering be invariant to rotations, we require a clustering algorithm that is invariant with respect to a class of (linear) transformations.
Invariant clustering A number of authors have developed clustering methods that are invariant to linear (Tarpey 2007), affine (Brubaker & Vempala 2008) image transformations by rotations,
2

---

Published as a conference paper at ICLR 2020
scalings and translations (Frey & Jojie 2002). Ju al. (2019) cluster natural images by using CNN to represent the space of invariant transformations rather than specifying it explicitly.
Alignments Instead of using custom clustering algorithms that are invariant under certain trans-
formations, we take simpler approach: we first transform our features such that they are maximally aligned using the class of transformations the clustering should be invariant to. This approach has been used in other contexts before, usually by minimizing the distances between the transformed observations. Examples include alignment of shapes in R using rigid motions (Gower 1975 Dry- den & Mardia 1998), alignment of temporal signals by finding monotonic input warps ZZoou & De la Torre 20122 y or alignment of manifolds with the distance between the observations being defined
according the metric on the manifold (Wang & Mahadevan 2008 Cui et al., 2014). There is
also work on alignment objectives beyond minimizing distances between transformed observations, examples of which include objectives based on generative models of observations (Kurtek et al.) 2011 Duncker & Sahani 2018) or probabilistic ones which are particularly suited for alignment with multiple groups of underlying observations (Kazlauskaite et al. 2019).
3 ROTATION-EQUTVARIAN CLUSTERING
Our goal to cluster neurons in the dataset into groups performing similar computations. To do so, we use their low lowdiimeniiona representations obtained from the published rotation- equivariant CNN of Bcker et al. (2019), which predicts neural activity as a function of an external image stimulus. We briefly review this model before describing our approach to rotation-invarial clustering.
CNN model architecture The model consists of two parts (Fig. [O)::
1. A convolutional core that is shared by neurons and computes feature representations T (x: E @W*H*X where is the input image, W x H is the spatial dimensionality and is the number feature maps.
2. A separate linear readout Wn, 8n @ rn E W for each neuron 1,..., N, factorized into a spatial mask 8,, and a vector of feature weights Pn.
The predicted activity of neuron n for image a is
a, (x) f(w, ((x)) = f(r 8n ((x)) ER
where f()) is non-linear activation function. Such a CNN therefore provides K-dimensional fea- ture weights rn characterising linear combinations of spatially weighted image features 8,, ((x) that are predictive of neural activity. We treat these feature weights as finite-dimensional proxies of actual computations implemented by the neurons. Because masks 8, (another component of read- outs w,, defined above) are irrelevant for our analysis. we will often refer to Tn simply as readout vectors. We will be referring to the matrix having ry as its rows as the readout matrix R E IR Nx
Rotation-equivarian core Feature representations (xx) are rotation-equivarian,, meaning that weight sharing is not only applied across space but also across rotations: for each convolutional filter there exist O rotated copies, each rotated by 2r/0. Feature vectors (x)
(œ) at position (i.j)
therefore consist of F different features, each computed in O senearly-spaced orientations (such that F x O = K). We can think of (1y(z) as being reshaped into an array of size F x O. Having computed dx (ar), , we can compute dhy (xx) with x' being stimulus æ rotated around (i. by 2x 2x/O by eyclically shifting the last axis of (iy((r) by one step (this mechanism is illustrated in Fig. 7
Rotation-equivalent computations The linear readout adheres to the same eqqiiiin-qquivariant structure as the core. As our goal is cluster the neurons by the patterns of features they pool while being invariant to orientation, we need to account for the set of weight transformations that correspond to a rotation of the stimulus when clustering neurons. We illustrate this issue with a small toy example consisting of six neurons that fall into two cell types (Fig- 3). Within each cell type (columms in Fig. B the individual neurons differ only in their orientation. More formally, define the computations performed by two neurons n, and ny to be rotation -qqivven-quivalent if there exists rotation Vij such that for any input stimulus x we have an, (x) = any (Mir(x)), We will refer such neurons rotation-equivalent well.
3

---

Published as a conference paper at ICLR 2020
1
0.4
2.6
2.6
1
0.4
0.4
2.6
8is"
0.5
2
0.9
8n
4
0.9
0.5
2
;
8n 4
2 0.9 0.5
B
Type
1x
Type 2: 1x
= Σ
0.5 2
2.6
0.9
8
1 0
2 0
= 1.5
Σ
0.5
2.6
0.9
0 2
an
Σ
[ 2.6
0.5 2
0.9
0 1
0 2
Σ
0.5
0.4 2.6
0.9
0
0 0
2n/3
0.3 2 0.9
0.4 2.6
0
3.5
24/3 %
Σ
0.5
1
0.4 2.6
2
0.9
[0
0 0
3.6
feature vectors (reshaped)
feature vectors (reshaped)
Figure 2: Toy example illustrating the computations in rotation ceaatioo--quiaiiannt CNN with two features (red and blue; cartoon feature representations are shown on top of corresponding values of ((x)) in three orientations (0, /33, 2T/ /3). A: Output of the rotation-equivariant CNN for an input image rotated by π r/3 (base orientation) can be computed by cyclic shift. B: Example of two distinct types of neurons (columms) in three orientations (rows). Computations for both types consist of linear combinations of the two features computed by the CNN with the same weights, but in different relative orientations. Readouts of neurons of the same type in different orientations are cyclic shifts of each other, since they produce the same outputs correspondingly rotated inputs.
Readouts of votation-equivalent neurons Directly clustering the readout matrix R does not re- spect the rotation equivalence, because readout vectors of neurons implementing rotation-equivalent computations are not identical (Fig. 2). To address that, we first modify R to obtain matrix R with the rows corresponding to rotation-equivallnt neurons aligned to a canonical form, and then cluster Ñł to obtain functional cell types. Rotating an input æ by multiple of 2 2r/O corresponds to eyclically shifting ((x), so the readout vectors of two potation-eqivallen neurons whose orienta- tion difference Viy is a multiple of 2 are also cyclic shifts cach other (Fig. 2}. For arbitrary rotations that not necessarily multiple of 2TT/O. /O, assume the readout Tn, of neuron nj to be a linear interpolation of cyclic shifts of Tn, corresponding to the two nearest rotations which are multiples of 2nT O0 Formally, define cyelic rotation matrix by an angle [0,2n] follows (matrix has shape O x O; column indices are shown above the matrix):
1
i
i+1 1
i 2
i+3
0
0
0
1-7
7
0
a0
0
0
0
1
1
i
2T
mod O
Sa
(2)
a0
7
i
1 1-7
7
0
2π
Given a readout vector Fn E K we can think of it as a matrix E OxF with columns corre- sponding to readout coefficients for different orientations readout T. by angle E [0, 2T) can be expressed as matrix multiplication r. (a) = S.rx. Note, by writing (a) as a function of a we refer to cyclically rotated (transformed) readouts, while Tn the fixed ones coming from pre- trained CNN and (0) Tu-
single feature. The cyclic rotation of
For two ootation-equivalle rotation of rn, by a, which is the rotation angle of vhej. If a is multiple of 2n/O otherwise it is only an approximation which becomes increasingly accurate as O increases.
neurons n, and nj. the readout vector Tn, can be computed as a cyclic
Aligning the readouts Assuming VI neurons form discrete functional cell types, all neurons in the dataset (and hence the readouts characterising them) be partitioned into non-overlapping classes w.r.t. the rotation equivalence relation we introduced above. Choosing one representative of each class and replacing the rows of with their class representatives, we can obtain R from R. Next, we discuss algorithm for finding such representatives of each class.
4

---

Published as a conference paper at ICLR 2020
We claim that by minimising the sum of pairwise distances between the eyclically rotated readouts
N Σ {as) 1Hl
{aī} - arg min
Σ
- (ay))l,
(3)
transform each readout into representative of a corresponding equivalence class (same for all readouts of a class), i.e. r;(aç) = ry(a}) if neurons and are rotation-equivalee indeed the case because the readouts of the same equivalence class lie the orbit obtained by cycli- cally rotating any representative of that class. Such angles {a;] } that neurons of the same class end up on the same point on the orbit (i.e. aligned to the same class representative) clearly minimise Eq. ( and since different orbits do not intersect (they are different classes of equivalence), read- outs of different equivalence classes cannot end up at the same point. Note that the resulting class representatives are not arbitrary, but those with the smallest sum of distances between each other in the Euclidean space. This mechanism is illustrated in Fig. 2
This is
Clustering aligned readouts Having obtained R with rows (a()), we can cluster the rows of this matrix using any standard clustering method (e.g. K-Means, GMM, etc.) obtain groups of neurons (cell types) performing similar computations independent of their preferred orientations.
Continuous relaxation of cyclic rotations The rotation- invariant clustering described above is based on solving the opti-
22.5
misation problem in Eq. (3). To do so, we would typically use
20.0
gradient-basee based optimisation, which prone to local minima be- cause of the way we define cyclic rotations in Eq. (2). According to that definition, a rotated readout is linear combination of two nearest base rotations, rather linear combination of all such rotations with only two coefficients being non-zero. That means that gradients of all but two coefficients w.r.t. the angle a are zero, and the optimisation would converge to a local minimum corresponding to the best linear combination of the two base ro- lations initialised with non-zero coefficients (Fig. 3
17.5
15.0
7.5
5.0
Angle
Figure 3: Distance between two vec- tors (top left corner) with first fixed and second cyclically shifted by an angle on the x-axis. Contin- uous relaxation (shades of blue) of linearly interpolated (black) cyclic shifts smooths gradients and helps overcome local minima.
To address this issue, propose approximate Eq. ( such that rn, (a) is a linear combination of all base rotations with non- zero coefficients, with the coefficients for the two nearest base rotations being the largest. Specifically we compute the coeffi- cients by sampling the von Mises density at fixed orientations to ensure cyclic boundary conditions and define Fr, (a), continu- ous relaxation of rn, (ax), as (a) Tn, where
7/1
72
...
70
7O
71
70-1
with 7 exp(T cos(a - (i - 1) 2n /O))
S,
(4)
exp(I cos(a
1) 2 2T/O))
7/2 73
The parameter T > O controls the sparseness of coefficients (1:). For small T, many of the co- efficients are significantly greater than zero, allowing the optimiser to propagate the gradients and reduce the effect of initialisation. As T increases, Fn, (a) becomes more similar to Tn,, (a), and the rotations by multiples of 2T/0 are recovered. In the limit, i., (2rk/O) + To, (2zk/O) as T + co (Fig. 3). Instead of fixing T, we learn it by optimising the regularised alignment objective with additional reconstruction loss preventing trivial solutions (e.g. all coordinates of r; (a being the same for T = 0):
N {a}} arg min Σ
Σ
(a,)) (a((ss) (())))-(((j)))) 3 Σ (((0) r:ll-
(5)
5

---

Published as a conference paper at ICLR 2020
A
B =
B = 0.01 I((a") R())
B
@ 0.479m
R
R(a") R(O)
Paireite distances
10'
Noise SD = 0.01
Noisc SD = 1.07
0.6
10-1
0.2
10
10
0.2
10-1
Raw
Aligned
Raw
Aligned
0.283
1.5
0.7
0
2
Figure 4: Synthetic data set: generation, alignment and dependence on noise. A: Panel R shows the unaligned synthetic data set as well as the corresponding shifted GP samples for each of the two groups of neurons (see details in the main text). Colored boxes in correspond the colors of corresponding GP samples. Panels R(a*) and R(0) show aligned readouts and readouts rotated by using Eq. respectively. R((0) should be similar to R for an adequate choice of (and consequently optimised value of T'). B: Effect of observation noise. Mcans of pairwise distances for each of the two groups shown in matrix R for two levels of Gaussian noise added to the dataset. Black dashed line: expected pairwise distance due to noise only (i.e. for perfectly aligned data with added noise). Raw and aligned matrices for each of the two groups are shown below the bar plots.
EXPERIMENTS
Synthetic dataset We generate a small toy dataset consisting of 16 hypothetical neurons (readouts) of two cell types to illustrate the proposed alignment method. Fach readout consists of just one feature eight base orientations (linearly spaced between 0 and 7 /4) and generated by one of the two underlying types of readouts cyclically shifted by random angle such a dataset we draw two independent noiseless functions from a Gaussian process (GP) with periodic kernel (with period 2T), then for each readout we randomly choose one of the two GP samples, shift it by an angle & and evaluate the shifted function at the base orientations to obtain an 8- 8-dimensional vector modelling the observed readout values. This process is illustrated in Fig. AA.
[0, 2Tr). To generate
Neural data We use the same dataset as in Bekkr et al. 2019), consisting of simultaneous record- ings of responses of 6005 excitatory neurons in mouse primary visual cortex (layers 2/3 and 4).
three-layer core with 16 features in 8 orientations in each layer (kernel sizes 13, 5. 5) and 128-dimensional readouts (F 16, O ==8) 8). We use the pre- pre-trained model provided by Ecker et al. (2019). We align the readout matrix R by minimising Eq. (5) w.r.t. the rotation angles a; and temperature T. We fit models for 20 log-spaced values of /3 in [0.001, 10), and choose for analysis the one with the smallest alignment loss (Eq. (3)) among the models with optimised temperature T 5. We use Adam (Kingma & Ba 2015 with early stopping and initial learning rate of 0.01 decreased three times.
Model details We analyse rotation-equivarian CNN consisting
Clustering aligned readouts We use the Gaussian mixture model implemented seikit-learn [Pedregosa et al. 2011) for clustering the aligned readouts R. We use spherical covariances to reduce the number of optimised parameters. To obtain quantitative estimate of the number of clusters in R, randomly split the dataset of 6005 neurons into training (4000 neurons) and test (2005 neurons) sets, fit GMMs with different numbers clusters on the training set, and then evaluate the likelihood of the fitted model the test set.

---

Published as a conference paper at ICLR 2020
5 RESULTS
5.1 SYNTHETIC DATA SET ALIGNMENT
We start by demonstrating on a synthetic dataset (Sec. 4 that optimising Eq. (5) can successfully align the readouts (Fig. [A), assuming 3 has been chosen appropriately. Note that readouts have been shifted by arbitrary angles (not multiples of π as demonstrated for readouts in colored boxes in Fig. [ and they are aligned precisely via interpolation Eq. ④ We can also see the effect of the parameter 3. controlling the relative weight of the reconstruction term (i.e. similarity of readouts rotated by 0 degrees to the observations). Small values of 3 incur small cost for poor reconstructions resulting in small optimised values of I and over-smoothed aligned readouts.
We next ask whether the alignment procedure still works in the presence of observational noise (4). For small to moderate noise levels (Fig. [ left), alignment reduces the pairwise distances to the level expected from the observation noise (shown at the top), confirming the visual impression (shown at the bottom) that alignment works well. For high noise levels (Fig. 4 right), alignment breaks down expected, and observe overfitting to the noise patterns (shown by the pairwise distances after alignment dropping below the level expected from observation noise alone).
5.2 MOUSE VI DATASET
Clustering We evaluate the GMM used to cluster R for different numbers of clusters. The test likelihood starts to plateau at around 100 clusters (Fig. 5), so we use 100 clusters in the following.
Test likelihood
220
Visualization of clusters To visualize the clustering result, we com- pute two-dimensional t-SNE embedding (van der Maaten & Hinton 2008) of the matrix of aligned readouts R, which is coloured accord- ing to the GMM clustering of R with 100 clusters (Fig. 6). Note that we use the embedding only for visualization, but cluster T28D aligned readouts in R directly. In addition to the embeddings, we also visualize the computations performed by some of the clusters by showing maxi- mally exciting inputs (MEls). We compute MEIs via activity maximi- sation (Erhan et al.] 2009: Walker et al.) 2018) and show the stimuli that maximally drive the 16 best- pest-predicted neurons of each cluster. We ob- serve that MEIs corresponding to neurons of the same cluster are generally consistent up to rotation and receptive field location, suggesting that the proposed clustering method captures the similarities in the neural computations while ignoring the nuisances as desired.
210
1 100 200 300 400 500
Number of clusters
Figure 5: Test set like-
lihood of GMMs applied to R as a function of the number of clusters.
Network learned redundant features We noticed a number of clusters with similar MEIs (e.g. Block 9 and Block 13 in Fig. 6). There could be two reasons for this observation: (a) the neural com- putations corresponding to these clusters could be different in some other aspect, which we cannot tell by inspecting MEIs as they represent only the maximum of complex function, (b) the fea- tures learned by the CNN could be redundant, i.e. the hidden layers could learn to approximate the same function in multiple different ways. To answer this question, we compute cluster confusion matrix (Fig. 7 left), which quantifies how similar the response predictions of different clusters are across images. The element (p. q) corresponds the correlation coefficient between the predicted responses on the entire training set of hypothetical neurons with cluster means for clusters and used as readouts. accounting for potential differences in canonical orientation across clusters. By greedily re-arranging clusters in the matrix into blocks based on their correlations, we show that the 100 clusters in the model can be grouped into much smaller number of functionally distinct clusters. Using correlation threshold of 0.5 in this re- ce-arrangement procedure, we obtain an exam- ple arrangement into 17 blocks (Fig. 7). Thus, the network has learned an internal representation that allows constructing very similar functions in multiple ways, suggesting that further pruning the learned network before clustering could lead to a more compact feature space for V1.
Finally, to quantify how consistent the resulting 17 groups of clusters are, we compute an MEI confusion matrix (Fig. 7 right panel). Its (i. element is the predicted activity of neuron j for the MEI of neuron i, after accounting for orientation and receptive field location (i.e. (y(y;), where

---

Published as a conference paper at ICLR 2020
Block
Block 12
Block 14
Block
Block 5
Block
Block
Block
Block
Block 2
Block 11
Riock
Block
Block 3
Block 10
Block 1
Block 16
Block 17
Figure 6: 2D i-SNE embedding of the aligned readouts R. colored according the GMM clustering with 100 components. Black stars show the locations of cluster centers. For some of the clusters, the MEIs of 16 best predicted neurons of that cluster are shown. The titles in the MEI subfigures show which matrix block in Fig. (left) that cluster belongs to.
yi is the MEI of neuron i moved and rotated such that it optimally drives neuron j). We show this matrix using the same grouping as for the cluster confusion matrix above and restrict (out of 6005) best predicted neurons (with test set correlation > 0.7). Note that some of the blocks from the cluster confusion matrix do not appear here, indicating that those clusters include poorly predicted neurons (e.g. block 17). The MEI confusion matrix exhibits block-diagonal structure, with most MEIS driving neurons within the same blocks most strongly, albeit with different degrees of within-bloc' similarity.
the 542
8

---

Published as a conference paper at ICLR 2020
0.8
0.6
0.2
13
12
9:1011
Figure Left: Cluster confusion matrix (100 100) for 100 clusters shown in Fig. 6 Rows and columns have been arranged into 17 groups (blocks). Right: MEI confusion matrix for well- predicted neurons (test correlation 0.7) arranged into the same blocks as on the left.
6 CONCLUSIONS AND FUTURE WORK
We have presented an approach to clustering neurons into putative functional cell types invariant location and orientation of their receptive field. We find around 10-20 functional clusters. the boundaries of some of which are not very clear-cut. To systematically classify the VI functional cell types, these proposals need be subsequently examined based variety of biological criteria reflecting the different properties of the neurons and the prior knowledge about the experiment.

---

Published as a conference paper at ICLR 2020
REFERENCES
Ján Antolik, Sonja B. Hofer, James A. Bednar, and Thomas D. Mrsic-flogel. Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses Natural Scenes. PLoS Comput Biol, 2016.
Tom Baden, Philipp Berens, Katrin Franke, Miroslav Román Rosón, Matthias Bethge, and Thomas Euler. The functional diversity of retinal ganglion cells the mouse. Nature, 529, 2016.
Spencer Ch. Brubaker and Santosh Vempala. Isotropic pea and affine-invariant clustering. In Pro- ceedings of the 2008 49th Anmual IEEE Symposium on Foundations of Computer Science, FOCS 2008, 2008
Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS computational biology. 15(4), 2019.
Zhen Cui, Hong Chang, Shiguang Shan, and Xilin Chen. Generalized unsupervised manifold align- ment. In Advances in Neural Information Processing Systems (NIPS). 2014.
lan L.. Dryden and Kanti V. Mardia. Statistical Shape Analysis. Wiley, Chichester, 1998.
Lea Duncker and Maneesh Sahani. Temporal alignment and latent gaussian process factor inference in population spike trains. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- cesa-Bianchi, and R. Garnett (cds.), Advances Neural Information Processing Systems (NIPS). 2018
Alexander S. Ecker, Fabian Sinz, Emmanouil Froudarakis, Paul G. Fahey, Santiago Cadena, Edgar Y. Walker, Erick Cobos, Jacob Reimer, Andreas S. Tolias, and Matthias Bethge. A rotation- equivariant convolutional neural network model of primary visual cortex. In International Con- ference Learning Representations, 2019.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical report, University of Montreal, 2009. Also presented at the ICML 2009 Workshop Learning Feature Hierarchies, Montréal, Canada.
Brendan Frey and Nebojsa Jojie. Fast, large-scale transformation-innariie clustering. In Advances in Neural Information Processing Systems (NIPS). MIT Press, 2002.
James C. Gower. Generalized procrustes analysis. Psychometrika, 40(1), 1975.
Xu Ju. João F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In The IEEE International Conference on Computer Vision (ICCV), 2019.
leva Kazlauskaite, Carl Henrik Ek, and Neill Campbell. Gaussian process latent variable alignment learning. AISTATS, volume 89. PMLR, 2019.
William F. Kindel, Elijah D. Christensen, and Joel Zylberberg. Using deep learning to probe the neural code for images in primary visual cortex. Journal of Vision, 19(4), 2019.
Diederik Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. In 3rd Interna- tional Conference on Learning Representations (ICLR), 2015.
David Klindt, Alexander S. Ecker, Thomas Euler, and Matthias Bethge. Neural system identifica- tion for large populations separating "what" and "where". In Advances in Neural Information Processing Systems (NIPS), 2017.
Sebastian A. Kurtek, Anuj Srivastava, and Wei Wu. Signal estimation under random tarp-ngs and nonlinear signal alignment. In Shawe-Taylor, R. 7emel, P. Bartlett, Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems (NIPS). 2011.
Lane McIntosh, Niru Maheswaranathan. Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes. In Advances in Neural Information Processing Systems (NIPS). 2016.
10

---

Published as a conference paper at ICLR 2020
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duch- esnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 2011.
Joshua R. Sanes and Richard H. Masland. The types of retinal ganglion cells: current status and implications for neuronal classification.. Annual review of neuroscience, 38, 2015.
Thaddeus Tarpey. Linear Transformations and the k- k-Means Clustering Algorithm: Applications to Clustering Curves. The American Statistician, 61, 2007.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9, 2008.
Edgar Y. Walker, Fabian H. Sinz, Emmanouil Froudarakis, Paul G. Fahey, Taliah Muhammad, Alexander S. Ecker, Erick Cobos, Jacob Reimer, Xaq Pitkow, and Andreas S. Tolias. Inception in visual cortex: in vivo-silico loops reveal most exciting images. bioRxiv, 2018.
Chang Wang and Sridhar Mahadevan. Manifold alignment using proerustes analysis. In Proceedings of the 25th International Conference on Machine Learning (ICML), 2008.
Feng Zhou and Fernando De la Torre. Generalized time warping for multi-modal alignment of human motion. 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012.

---

Published as a conference paper at ICLR 2020
A RANDOM PERMUTATIONS OF FEATURES
Figure A1: 2D t-SNE embedding of the aligned readouts R with feature weights randomly permuted for each of the neurons. The colors correspond to the GMM clustering with 100 components. Black stars show the locations of cluster centers. For some of the clusters, the MEIs of 16 best predicted neurons of that cluster are shown.
12

---

Published as a conference paper at ICLR 2020
Figure A2: 2D t-SNE embedding of the aligned readouts R with feature weights randomly permuted across the neurons. The colors correspond to the GMM clustering with 100 components. Black stars show the locations of cluster centers. For some of the clusters, the MEIs of 16 best predicted neurons of that cluster are shown.
Control (random permutations for each neuron)
Control
Aligned readouts
(random permutations across neurons)
Figure A3: t-SNE embeddings for the aligned readouts (Fig. 6). and the controls with randomly permuted features for each neuron (Fig. AT) and across the neurons (Fig. A2
13

---

Published as a conference paper at ICLR 2020
B SYNTHETIC DATASET: DEPENDENCE ON NOISE
Noise SD 0.00 Algred
Noise
0.10
kaw d8t3
:SKE 9am d01))
LSKE salgnedl
haw daxa
Algred
1-Ski yrww data)
tSk Jaligndi
Noise SD
0.20 SNE daits)
Noise SD
0.50 1-SNE fraw
uned)
Raw
Isligndd
0.75 Noise SD
1.00 Noise SD
Algridd
data))
gnoll
haw
Algred
jalgmodi
2.00 Noise SD
Noise 5D
5.00 1-kE Praw dama
Kaw dita
Algred
1:8E 93w dats)
LIKE galgnedi
Raw dara
Algred
SNE (alinedi
Figure B1: Alignment of synthetic dataset of 100 observations generated using the procedure described in Sec. for different amount of i.i.d. Gaussian noise added to the observations. The panels for each noise level show the 16 (out of 100) examples of the raw and aligned data well the t-SNE embeddings of raw and aligned data coloured according to the GMM clustering with two components.
14

---

Published as a conference paper at ICLR 2020
C MERGES AND SPLITS OF CLUSTER CONFUSION MATRIX BLOCKS
Sandis
Figure CI: Sequential merges of the three pairs blocks with the highest correlations in the cluster confusion matrix (Fig. 7 left). The merged blocks and the correlation values are shown in the titles of panels.
Origntl morrs (Pg 7)
Sale 12
BeDD
Figure C2: Sequential splits of the three pairs of blocks in the cluster confusion matrix (Fig left). The merged blocks, the correlation values, and the examples MEIs of one the GMM clusters in each of the splitted blocks are shown for each splitting step.
15


Reviews:
[{"review_id": "r1gCIKb0KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a rotation-invariant representation of a CNN modeling the V1 neurons and a pipeline to cluster these neurons to find cell types that are rotation-invariant. Experimental validation is performed on a 6K neuron dataset with promising results. \nThe paper is well postulated.\n\nBelow are comments about the work:\n\n1. In Figure 2, what does 1 x feature + 2 x another_feature mean?\n2. In Equation 3, why was the ‘square’ of error differences not used? \n3. In the clustering approach, how is the number of mixtures set for the GMM? How stable is the model to different number of mixtures?\n4. In Figure 6: are Blocks 5 and 13 the same clusters (since they are of the same color) or is it that the colourmap use did not have 100 colors? \n5. In the ‘network learned redundant features’, Sentence 1: why do the authors say ‘similar MEIs’. The 16 neurons rendered in both blocks look different. \n6. It will be informative to know how the number of clusters vary based on the correlation threshold used to collapse 100 clusters to a lower number. Are the clusters still functionally distinct for varying thresholds? Further why is MEI confusion matrix only shown for 13 groups?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eKwBtIcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an original approach to predict the function of groups of neurons in the V1 cortex based on their invariance to well designed rotation invariant CNN filters. The design of these features is funded by the observation that specific ganglion cell types have rotation and scale invariant responses to visual stimuli. \nThe method is very clearly explained and the evaluation on an publicly available dataset looks promising. The clustering Figure 6 in particular is very insightful. \nThe paper could have been more impactful if a comparison with a ground truth was built. The issue is clearly that ground truth is hard to establish for this type of problems but biological observations and annotations of cell types can be available (unfortunately not public as far as I know). \nI would also be curious to know how such a method can be applied to a blind patient whose retina does not react to visual stimuli. Is there a biological function that will still preserve such invariance properties which allow to find structure in the data?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lNHjJJ5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this study, the authors develop a method to cluster cells in primary visual cortex (V1) based on the cells' responses to natural images. The method consists in three steps: \n- fit a rotation-equivariant convolutional neural network model to V1 cells (previously described in Ecker et al. 2019)\n- align all cells by choosing the rotation for each cell that minimizes overall distance between cells in feature space, so that the clustering is mostly blind to the orientation of the filters.\n- cluster the cells using a Gaussian mixture model (GMM).\n\nAlthough I find this article mostly well-written and the topic important, I cannot recommend acceptance because (1) the study does not make a significant contribution to our understanding of V1, (2) the main innovation in ML presented (alignment method) is quite specific and will thus not likely be of interest for the general audience of ICLR:\n\n(1) An important question in visual neuroscience is whether V1 cells form discrete functional clusters as opposed to a continuum. Another related question is whether these functional clusters correspond to distinct cell types characterized by specific wiring patterns, gene expression and/or morphology.\nThe analyses performed do not answer any of these two questions:\n- the clustering model (GMM model) is not compared statistically to other models that would assume a continuous structure in the data (e.g. cells form a sparse continuous manifold in feature space). Although clusters do appear in the t-SNE visualization, this visualization does not provide statistical evidence that cells indeed form distinct clusters.\n- The correspondence of the proposed clusters to cell types with specific wiring patterns, gene expression and/or morphology is not established. To establish this correspondence would require further experiments, as acknowledged by the authors: \"To systematically classify the V1 functional cell types, these proposals need to be subsequently examined based on a variety of biological criteria reflecting the different properties of the neurons and the prior knowledge about the experiment\".\n\n(2) The alignment method, which consists in rotating the cells in feature space so that orientation is not a factor for subsequent clustering, is quite specific to the problem studied and likely not of interest for the general ICLR audience.\n\n\nAdditional feedback:\n\n- Title: ROTATION-INVARIANT CLUSTERING OF FUNCTIONAL CELL TYPES IN PRIMARY VISUAL CORTEX \n=> \"functional cell types\" is not adequate here, since the article does not establish the existence of functional cell types. Could be replaced with \"cell responses\".\n\n- Abstract: We apply this method to a dataset of 6000 neurons and provide evidence that discrete functional cell types may exist in V1.\n=> this sentence is misleading, since no evidence for functional clusters is provided.\n\n- \"Thus, the network has learned an internal representation that allows constructing very similar functions in multiple ways\"\n=> To avoid the caveat of redundant features, the authors could try to add a dimensionality bottleneck on feature space before readout.\n\n- \"Small values of β incur a small cost for poor reconstructions resulting in small optimised values of T and over-smoothed aligned readouts.\"\n=> A simulated annealing procedure (progressive increase of T during learning) could potentially allow the use of larger β values here (i.e. less distortion of the filter).\n\n- The alignment procedure could lead to the emergence of spurious structure in the clustering. It would be important to control for this potential artifact by running the procedure on an unstructured synthetic dataset.\n\n- It is possible that the MEIs within clusters look more similar than they actually are, since the cells are fitted from the same common bank of features. It would be useful but maybe difficult to control for this.\n\n- It would be interesting to test the clustering procedure on a shuffled version of the readout weights (shuffle across features and V1 cells), so as to keep sparsity but not any other structure. Does the t-SNE map look less clustered? Is the GMM fit qualitatively different?\n\n- Fig1(2): add legend/caption. what are the ellipses?\n\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is enthusiastically supported by all three reviewers. Thus an accept is recommended.
Decision: Accept (Talk)