Learning and aligning single-neuron invariance manifolds in visual cortex
Authors: Mohammad Bashiri
Luca Baroni
Ján Antolík
Fabian H. Sinz
Venue: ICLR 2025 Oral (2025)
Keywords: neural invariances
invariance manifold
MEI
implicit neural representations
contrastive learning
invariance alignment
clustering
visual cortex
macaque V1
primary visual cortex
Primary area: applications to neuroscience & cognitive science
TL;DR: Our method learns single-neuron invariances and aligns them, enabling population-level exploration of neural invariances.

Abstract: Understanding how sensory neurons exhibit selectivity to certain features and invariance to others is central to uncovering the computational principles underlying robustness and generalization in visual perception. Most existing methods for characterizing selectivity and invariance identify single or finite discrete sets of stimuli. Since these are only isolated measurements from an underlying continuous manifold, characterizing invariance properties accurately and comparing them across neurons with varying receptive field size, position, and orientation, becomes challenging. Consequently, a systematic analysis of invariance types at the population level remains under-explored. Building on recent advances in learning continuous invariance manifolds, we introduce a novel method to accurately identify and align invariance manifolds of visual sensory neurons, overcoming these challenges. Our approach first learns the continuous invariance manifold of stimuli that maximally excite a neuron modeled by a response-predicting deep neural network. It then learns an affine transformation on the pixel coordinates such that the same manifold activates another neuron as strongly as possible, effectively aligning their invariance manifolds spatially. This alignment provides a principled way to quantify and compare neuronal invariances irrespective of receptive field differences. Using simulated neurons, we demonstrate that our method accurately learns and aligns known invariance manifolds, robustly identifying functional clusters. When applied to macaque V1 neurons, it reveals functional clusters of neurons, including simple and complex cells. Overall, our method enables systematic, quantitative exploration of the neural invariance landscape, to gain new insights into the functional properties of visual sensory neurons.

Published as a conference paper at ICLR 2025
LEARNING AND ALIGNING SINGLE-NEURON INVARIANCE MANIFOLDS IN VISUAL CORTEX
Mohammad Bashiri Luca B Ján Antolik", Fabian H. Sinz
1 Noselab GmbH, Munich, Germany
"Department of Computer Science, University of Göttingen, Germany
Faculty of Mathematics and Physics, Charles University, Prague, Czechia
*Campus Institute Data Science (CIDAS), University of Götringen, Germany "Equal contribution, 'mohamaddba 1 com
ABSTRACT
Understanding how sensory neurons exhibit selectivity certain features and in- variance to others is central to uncovering the computational principles underlying robustness and generalization in visual perception. Most existing methods for char- acterizing selectivity and invariance identify single or finite discrete sets of stimuli. Since these only isolated measurements from underlying continuous mani- fold, characterizing invariance properties accurately and comparing them across neurons with varying receptive field size, position, and orientation, becomes chal- lenging. Consequently, systematic analysis of invariance types at the population level remains exper-explored. Building on recent advances in learning continuous invariance manifolds, we introduce a novel method to accurately identify and align invariance manifolds of visual sensory neurons, overcoming these challenges. Our approach first learns the continuous invariance manifold of stimuli that maximally excite neuron modeled by response- learns affine transformation response-prcdicting predicting deep neural network. It then the pixel coordinates such that the same manifold activates another neuron as strongly as possible, effectively aligning their invari- ance manifolds spatially. This alignment provides a principled way to quantify and compare neuronal invariances irrespective of receptive field differences. Using simulated neurons, demonstrate that our method accurately learns and aligns known invariance manifolds, robustly identifying functional clusters. When applied to macaque V1 neurons, reveals functional clusters of neurons, including simple and complex cells. Overall, our method enables systematic, quantitative explo- ration of the neural invariance landscape, to gain new insights into the functional properties of visual sensory neurons.
INTRODUCTION
To understand visual sensory processing, it is crucial to examine not only the coding properties of individual neurons but also how these coding properties are organized across the entire neural population. Two fundamental principles of sensory coding are selectivity and invariance: visual neurons exhibit selectivity to specific features such as orientation, while maintaining invariance to other transformations such as translation (Hubel & Wiesel 1962, Riesenhuber & Poggio, 1999). For example. in the primary visual cortex (VI), simple cells are selective to the precise phase of stimulus, whereas complex cells exhibit phase invariance, responding consistently regardless of phase shifts (Hubel & Wiesel 1962). These computational properties enable neurons to maintain informative yet stable responses across diverse inputs, thus supporting reliable object recognition and visual processing.
Recently, data-driven approaches using Deep Neural Networks (DNNs), acting digital twins of sensory neurons, have dramatically advanced our ability to model neural responses YYamins & DiCarlo (016 Cadena et al., 2019; Lurz et al.) 2020 Klindt et al. 2017). These approaches enable the identification of a single maximally exciting input (MEI) [Walker et al. 2019; Bashivan et al. 2019) diverse set of nearly maximally exciting stimuli that outline the invariance space of the

---

Published as a conference paper at ICLR 2025
neuron (Ding al. 2023 Cadena et al. 2018). In this context, an invariance manifold can be defined as the set of stimuli that elicit near- maximal responses from a neuron. These stimuli are related by transformations to which the neuron is invariant, meaning that the neuron's 's response remains constant (or nearly so) across these variations. For instance, the invariance manifold of a complex cell exhibiting phase invariance is the set of Gabor patterns that differ only in phase and maximally excite it. While invariances can also be explored at suboptimal response levels, focusing on peak response levels highlights the stimuli to which the neuron most selective. Most existing methods, however, do not readily enable analyses to compare the invariance manifolds of neurons across populations and to identify clusters of neurons sharing common invariances. Ideally, such a method should meet the following requirements:
It should accurately capture the invariance manifold of individual neurons, ensuring that subse- quent analyses faithfully reflect neural coding.
It should enable the identification of common invariance properties while disregarding differ- ences in selected 'nuisance' receptive field (RF) attributes such as position, size, and orientation.
It should allow for a meaningful measure of the similarity of invariance properties across neurons.
Current approaches do not satisfy all of these requirements simultancously.
In this paper, we propose novel approach for aligning and comparing invariance manifolds across population of neurons, fulfilling the above-mentioned requirements and allowing clustering of neurons based on their invariance properties. Our method first identifies the invariance manifold of individual neurons (Fig. top panel) using the same approach as Baroni al. (2023). It then learns an affine transformation on the pixel coordinates such that the same manifold activates another neuron as strongly as possible, effectively aligning their invariance manifolds spatially (Fig. middle panel). Neurons whose invariance manifolds strongly activate each other likely share similar computational roles (Fig. bottoo panel). By performing this pairwise alignment across a population of neurons, our method systematically uncovers shared invariance properties, irrespective of differences in their receptive field attributes, allowing for clustering of neurons based on their invariances.
We tested our method on simulated neurons and macaque VI neurons modeled by response- predicting DNN. The simulated neurons were carefully designed to display range of invariance properties, and our method successfully learned and aligned their corresponding invariance manifolds, enabling accurate identification of the ground truth functional types. When applied to a response- predicting model of macaque VI neurons, our approach captured and aligned the invariance manifolds of these neurons, uncovering multiple functional clusters, including, but not limited to, the expected clusters of canonical simple and complex cells.
Overall, our method generalizes effectively from synthetic to macaque V1 neurons, offering a robust framework for systematically analyzing invariance properties across sensory neurons, to deepen understanding of the computational roles of individual neurons and gain new insights into the broader organizational principles governing neural populations.
RELATED WORK
Our method is uniquely situated at the intersection of characterizing neuronal invariances and clustering neurons based on shared functional properties. The works discussed below are closely related to one or more aspects of our approach, either addressing neuronal invariances or functional clustering of neurons.
Neuronal invariances Recent pioneering work demonstrated the use of deep learning to charac- terize neural selectivity by identifying maximally exciting inputs (MEIs), which were subsequently validated through in-vivo experiments (Bashivan et al. (2019] Walker et al. 2019 Ponce et al. 2019). Using cesponse-predictinn DNNs, Cadena et al. (2018) and Ding et al. (2023) characterized neural invariances by generating multiple such MEls for a neuron by directly optimizing the pixel values while encouraging diversity across the MEIs using an additional regularization term.
Limitations of discrete approximations While these methods marked a step forward in characteriz- ing invariances, they approximate the underlying continuous invariance manifold through finite set
2

---

Published as a conference paper at ICLR 2025
of points. This discretized approach risks leaving gaps in the manifold, potentially resulting in a less precise representation of the invariance, or requires a large number of samples, making it computa- tionally challenging to capture its full complexity. This could result in challenges for single-neuron invariances that involve more complex transformations along the underlying manifold, as simply interpolating between discrete points may not reliably produce
samples the manifold.
Learning continuous invariance manifolds Recently, [Baroni et al. (023) proposed an alternative approach that combines contrastive learning with implicit neural representations (INRs) map a continuous latent input onto the invariance manifold. This method captures the manifold as continuous entity, addressing the discretization issue from previous methods. Similarly, Wang & Ponce (2022a b) introduced a framework for modeling neural invariances at different activation levels, providing theoretical insights into their geometry and examining variations tuning landscapes across the ventral stream. In contrast to Baroni et al. (2023), which learned continuous invariance manifolds using INRs, Wang & Ponce (2022a (b) used a pretrained generative model to explore invariances in its latent space, capturing diverse patterns within the naturalistic image space learned by the generative model. Although these methods advance our understanding of invariance manifolds, they do not provide mechanism to compare invariances across neurons at the population level, gap we aim to address in this study.
Functional clustering of neurons Reliable comparison of neurons based on encoding properties requires separating shared nonlinear computations from variations in nuisance RF properties, such as location, size, and orientation. For example, two simple cells with the same RF pattern but different locations and orientations should be grouped in the same cluster. While multiple methods have been developed for clustering neurons based on their encoding properties, they typically do not fully isolate the nonlinear computations from the nuisance RF properties or use clustering metrics that are often difficult interpret.
Model-based clustering Ustyuzhaninov et al. 2019) employed a model-basee approach to cluster neurons based on feature weights in the readout layer of caaationally-qquivrtie response-predeeeee predicting model. This method effectively clusters neurons with similar encoding properties, regardless of differences the orientation and position of their RFs. However, this approach does not account for RF size variations, property known to vary with eccentricity in higher mammals' visual systems (Wilson & Sherman 1976 Cavanaugh et al. 2002] Harvey & Dumoulin (01T). Additionally, this approach risks incorrectly separating neurons with similar encoding properties, as the same neuronal response function can be realized by different combinations features. Finally, a metric on feature weights does not readily translate into interpretable metric of neuronal similarity.
Stimulus-based clustering Burg et al. (2024) introduced a technique that clusters neurons by identifying the most discriminative stimuli across groups of neurons, but does not account for variations in RF orientation or size. Similarly, Willeke et al. (2023) and Tong et al. (2023) employed contrastive learning cluster neurons with similar maximally exciting inputs (MEIs) macaque V4 and mouse visual cortex, respectively. However, their method relies on dimensionality reduction of single-neuron neuron MEIs, which may fail cluster neurons exhibiting invariance properties reliably. For example, two images from the invariance manifold of the same neuron could end up in different clusters, leading to inconsistent cluster assignment depending on the image used.
METHOD
Our approach consists of two stages: Template Learning and Template Matching. The first stage identifies the continuous invariance manifold of single neuron, and the second stage aligns this learned manifold with another neuron to maximize its activity, evaluating their similarity based on shared invariance properties (Fig. [..
3.1 TEMPLATE LEARNING
Implicit Neural Representation of the invariance manifold In the template learning step, following Baroni et al. (202)). we use Implicit Neural Representations (INRs) to parameterize the continuous invariance manifold of a neuron modeled by response- prediccing DNN, generating diverse images that elicit maximal activity. INRs are neural networks that represent continuous signals, such as images 3D shapes, by mapping coordinates to signal values without relying on discrete grid-based
3

---

Published as a conference paper at ICLR 2025
Step 1: Template Learning
Response-predicting model
Latent input
INR trained learn invariance manifold template neuron
INR
Image space
Representation
Σ fo (e (x,,,,, Response Response-predictin predicting Model
Grid of pixel coordinates
Template neuron
Step 2: Template Matching
Target
Learnable affine
transfomattin aligns the template invariance manifold to target neuron
b
Template neuron's invariance manifold (template manifold)
Template manifold matched to target neuron
Responses to template manifold
Activation
Template neuron
Target neuron (before matching)
Target neuron (after matching)
Latent input
Figure I: Overview of the method. Top panel) Template learning. An Implicit Neural Representation (INR), mapping from pixel coordinates and latent space, generates manifold of images that maximally, and equally, excite a neuron (referred to as the template neuron). Middle panel) Template matching. The same INR is used to generate the invariance manifold of the template neuron. affine transformation on the pixel coordinates (x, y) is Iearned to align the template invariance manifold to a target neuron such that it activates the target neuron as strongly as possible. Bottom panel) Left: Template neuron's 's invariance manifold (referred to as the template manifold). Center: Template manifold after the template matching step, i.e., aligning to target neuron. Right: Schematic representation of the neurons' responses to different images of the template manifold. The template manifold maximally activates the template neuron but does not activate the target neuron. However, after matching, if the template neuron and target neuron present similar invariance properties, maximally activates the target neuron.
representations (Genova et al. 2019 Atzmon & Lipman 2020 Sitzmann et al. 2019 Mildenhall
al. 2021). In the context of image generation, an INR maps 2D pixel coordinates (ar, y) to pixel values Io (z, y). producing continuous, resolution-independenn image from the input grid of pixel coordinates. Similar to Baroni et al. (2023),
introduce a periodic ID latent variable E [0, 2z) as an additional input to the INR, enabling it to generate multiple images from the same pixel grid and produce family of images (g y, xx)x,) >)) by varying 2. For brevity, we might represent (x y, 2) as g(2) in the following equations, omitting the dependence x, and the parameters di which remain implicit.
While we generally follow the approach of Baroni et al. (2023), we introduced several modifications, including changes to the architecture and the use of positional encoding. Specifically, we implemented the INR as fully connected neural network with 4 hidden layers, each containing 50 hidden units with a Tanh nonlinearity. Since we aim to generate grayscale images, the output layer consists of used
single unit with Tanh activation, producing values between -1 and 1. Additionally, positional encoding with random Fourier features (Tancik et al. 2020). as used by [Bronii et al. (202)), the pixel coordinates (r.y) and extended it to the latent variable When applied the
4

---

Published as a conference paper at ICLR 2025
pixel coordinates, this encoding enables control over the spatial frequency of the generated images, whereas its application to the latent variable allows the INR to capture more complex variations along the manifold, even with small changes in 2.
Objective function The INR parameters è are trained to generate a diverse set of images that collectively span the invariance manifold of target neuron, ensuring all images elicit maximal activation. The images generated by the INR are fed to a response- response-predicting DNN [ 2 with fixed parameters O. which evaluates the activation of the target neuron in response to the generated images (Fig. top panel). Before being fed to the DNN, the generated images are linearly scaled satisfy specific norm constraint. Combined with the activity maximization objective, this ensures that the pixel variability in the generated images represents only features relevant to the neuron and prevents saturation of pixel values. To ensure diversity of patterns across the generated images, spanning the underlying invariance manifold, we use a contrastive objective (Chen et al. 2020p that encourages variations across the images. Combined with the objective of maximizing the neuron's s response, the complete objective function (to maximize) for template learning is:
N. Σz,ez exp (sim ( (z.) ((2
04 + log QMEI
CTL
(I)
N ExiEz! exp (sim g (g()) g(x)))(77))
where N is the total number of generated images corresponding the number of latent points 2. is the activation a; = fo (x, of the target neuron, normalized by its response to its MEI, obtained by standard pixel optimization (Appendix Aj. Although different latent points were chosen to be equally spaced on a grid, the grid was jittered at each training step to continuously cover the full range of the latent input. The strength of the contrastive objective is controlled by A. The contrastive term uses cosine similarity sim sim(g(z)), ((z))) between images generated from latent points 2, and 2j, scaled by temperature r. Positive examples Z come from latent points near 2s while negative examples 2 come from distant points, with N+ and N_ being their respective counts. This formulation encourages similarity between images corresponding to nearby latent points while making images corresponding to distant latent points different from each other, promoting a smooth and diverse learned manifold. Additional details the template learning step can be found in Appendix B and an analysis of the method's robustness to varying definitions of "near" and "distant" latent points is provided in Appendix C
OME
3.2 TEMPLATE MATCHING
Aligning the learned invariance manifold After the template learning step, where we identify the invariance manifold of the template neuron, the second stage focuses on aligning this template manifold with another neuron (referred to as the largel neuron). We refer to this step as Template Matching (Fig. D middle panel). In this step. the template manifold is fixed by freezing the INR parameters o の. To spatially transform the template, we learn affine transformation Ag applicd to the pixel coordinates (xr, y) such that images sampled from the template maximize the response of the target neuron fo y),,
Objective function The objective function for optimizing the parameters y of the affine transforma- tion matrix is similar to Eq. but without the contrastive learning term since the INR parameters are not being optimized in this step and the invariance manifold fixed:
LTM = N
ā =o (io(go-(As xx,,
OME
where N is the total number of images generated from the manifold, âq is the activation of the target neuron when presented with the transformed image 96- (Ag (x, 2), and âMи is the response of the target neuron to its MEL. By learning the affine transformation, we align the spatial structure of the template neuron's 's manifold to that of the target neuron. This alignment allows us to probe the degree of shared invariance between the two neurons, based on how well the transformed manifold activates the target neuron (Fig. [ bottom panel). The activation metric used for alignment is immediately interpretable because it tells us how much the template manifold can drive another neuron. Additional details on the template matching step can be found in Appendix D
5

---

Published as a conference paper at ICLR 2025
Importantly, while we focus affine transformations, our formulation can be extended to nonlinear (invertible) distortions of space or subsets of affine transformations, allowing control over how flexible or constrained the invariance alignment should be. Additionally, the continuous nature of INRs prevents artifacts typically introduced by discrete pixel coordinate transformations.
3.3 CLUSTERING BASED ON INVARIANCE, ALIGNMENT
By applying the template learning and template matching steps to all pairs of neurons in the population, we construct an activation matrix, where each row corresponds one neuron (the template neuron) and each column to another neuron (the target neuron). Each entry (i,j) in this matrix represents the average response of neuron j to the images of the invariance manifold of neuron after matching. To make sure that activations are comparable across neurons despite differences in their intrinsic excitability, we used relative activations as opposed absolute responses. The relative response for any given neuron was computed as: (a ao) (UMEI (x)) where or is the absolute response, oo is the baseline response (i.e, response to a gray image), and OMEE is the MEI response. This yields relative activation values where 0 corresponds to the baseline response and to the MEI response.
The activation matrix reflects functional relationships between neurons by capturing how their invariance manifolds activate each other. Importantly, because neurons exhibit nonlinear response characteristics, clustering based on the activation matrix offers a better understanding of functional similarity than clustering directly using invariance manifolds MEIs, distances in image space do not necessarily have a one-to-one correspondence with functional differences between neurons.
Additionally, notice that the matrix is not symmetrie. Specifically, if Mj and M2 are the invariance manifolds of the template and target neuron, respectively, a high activation means that M1 S. M2, but not necessarily vice versa. For example, if we consider simple and complex cell, after the alignment of their invariance manifold, the Gabor pattern of simple cell would maximally activate a complex cell, while the invariance manifold of a complex cell would not maximally activate simple cell. Therefore, this asymmetry reflects a meaningful difference between neurons' computational roles and the relationship between their invariance manifolds.
Once the activation matrix is constructed, we cluster neurons by applying a standard agglomerative hierarchical clustering on the rows of the activation matrix. Each row corresponds to single neuron and represents how strongly its invariance manifold activates other neurons after matching. We use cosine dissimilarity as the distance metric for clustering, which helps focus on the relative activation patterns between neurons rather than the magnitude of the activation, ensuring that functional relationships are properly captured.
4 EXPERIMENTS AND RESULTS
4.1 VALIDATION ON SIMULATED NEURONS
As a first step, we validated our approach using a set of Gabor-based simulated neurons to verify and quantify its ability accurately capture and align neural invariances across different neuron types. The main advantage of this simulated environment is that we have access to the ground truth invariances for each neuron type, enabling precise benchmarking of the accuracy of the method.
The simulated neurons consisted of several types, including simple and complex cells, as well as additional neurons exhibiting non-trivial, arbitrary invariance properties (A A top row). Simple cells were modeled as linear-nonlinear systems using single Gabor filter and complex cells were modeled using set of Gabor filters with shared parameters but varying phases, achieving phase invariance by selecting the highest response across filters. Neurons with more complex invariances were similarly modeled using filters obtained as combinations of Gabors with varying arbitrary parameters and selecting the highest response across filters to achieve the desired invariance. For each neuron type, we created six instances, resulting in a total of 36 neurons. Neurons of the same type differed from one another by an affine transformation of their receptive fields, including variations in position, orientation, scale, and shear (Fig (Fig bottom row). To standardize the response of the neurons and facilitate the comparison of results, we normalized the Gabor filters to have a maximum activation of 1 in response to stimuli of unitary norn. Further details on the simulation can be found in Appendix E

---

Published as a conference paper at ICLR 2025
A
11
II]
4
H
14
B
Before clustering
After clustering
E
Model vs GT
F Consistency across seeds
1.0
1.0
0.8
Dna Cen
0.8
Sinilarity
Similanity
arthastin
0.6
0.6-
ponden
0.4
DIf Cladeer
0.4
0.2-
0.2
Neurens (mntial order)
Neuraes Jorder by cluscer ot mg)
0.0
Activation Manifold
0.0
Activation Manifo
Figure 2: Results on simulated neurons. A) Types of Gabor-based simulated neurons, each associated with distinet color. Top row: invariance manifold of a representative neuron for each type. Left to right: even simple cells, odd simple cells, complex cells, and three types of neurons with selectivity and invariance more complex patterns. Note that images for simple cells are identical, indicating their lack of invariance. Bottom row: For each neuron type we model 6 neurons that differ by an arbitrary affine transformation illustrated by dotted lines around each neuron's 's receptive field. B) Learned invariance manifolds for representative neurons shown in Panel A top row. C) Activation matrix resulting from the pairwise alignment of neural invariances. Each row represents how invariance manifold of one neuron activates others post-alignment, while each column shows how a specific neuron is activated by the manifolds of other neurons. D) Activation matrix ordered by applying hierarchical clustering algorithm. The block diagonal pattern corresponds to identified clusters reflecting different neuron types. Dendrogram colors correspond to neuron types in Panel ground truth. Left barplot: activations from learned manifolds
A. E) Comparison of predictions (red), aligned manifolds of neurons within the same cluster (green), and aligned manifolds of neurons from different clusters (gray). A similarity of I means the invariance manifold activates the neuron as high as its MEI. Right barplot: similarity scores between the ground truth and the learned manifolds (red), aligned manifold of neurons within the same cluster (green), and aligned manifold of neurons from different clusters (gray). F) Consistency of results across 5 seeds, i.e., = 10 unique pairwise combinations. Left barplot: similarity scores of activation matrices across seeds. Similarity of diagonal entries is shown in red, off-diagonal same cluster is shown in green, and off-diagonal different cluster shown in gray. Right barplot: similarity scores of learned manifolds (red), aligned manifolds of neurons within the same cluster (green) and from different clusters (gray), across seeds.
We applied the template learning and template matching steps to all pairs of simulated neurons resulting in an activation matrix where row shows how strongly the invariance manifold of neuron i activated all other neurons (Fig. PC). While the neurons' order is randomized in this matrix, the high activation in the diagonal entries shows that our method successfully recovered the known invariance transformations for each neuron, generating invariance manifolds that activated the neurons close to their maximal activation (see Fig. 2 for examples of the learned invariance manifold). We then applied hierarchical clustering. using cosine dissimilarity as a distance metric, to the rows of the activation matrix, accurately recovering the predefined neuron types (Fig. 2D) and showing that method successfully aligned the invariance manifolds of simulated neurons with similar type, despite differences in their nuisance RF properties.
7

---

Published as a conference paper at ICLR 2025
We further quantified deviations from the ground truth by analyzing the learned and aligned manifolds as well as the corresponding activations. For a given neuron, we computed the activation similarity between the ground truth actor and the learned manifold activation act as 1 - lacter act), Similarly, we computed manifold similarity by measuring how well the images on one manifold match the closest images on the other manifold, averaged bidirectionally. This captures the degree to which the two manifolds overlap in terms of their most similar images (for more details refer to Appendix D. Our method effectively yielded activation values near the maximum not only in response to the learned invariance manifold of each neuron but also in responses to invariance manifolds of other neurons within the same type, after being matched (Fig. 2 left, red and green bar). Notably, when this alignment was performed between neurons of different types, the resulting activation was consistently lower, as expected (Fig. E left, gray bar). When comparing the learned and aligned manifolds with ground truth manifolds, we observed high similarity scores for manifolds aligned within the same cluster, indicating that invariances were successfully aligned (Fig. 2E right, red and green bar). Alignments between different clusters yielded lower similarity scores, highlighting differences in invariance properties across distinct neuron types (Fig. 2 right, gray bar).
Additionally, assessed the consistency of our method across different initializations (seeds), at both the activation and manifold levels. For comparing the activations across seeds, we defined an activation similarity metric
|acl; - actz), J. which measures the closeness of activations between seed and j. averaged across entries in the activation matrix. Similarly, we computed a similarity score of the corresponding learned and aligned manifolds across seeds to evaluate consistency in manifold learning and alignment, directly using images sampled from the manifolds. As depicted in Fig. F our analyses show that our results on the synthetic data are robust, with high consistency observed both at the level of activation and in the alignment of invariance manifolds.
Finally, in Appendix G we present detailed comparison of our approach with two model-based clustering methods introduced by Klindt et al. (2017) and Ustyuzhaninov et al. (2019). Specifically, we show that while these methods effectively account for shifts (Klindt et al. al.[ 7) or both shifts and rotations (Ustyuzhaninov et al. 2019), they do not extend to the broader class of affine transformations. Notably, the comparison also highlights that our method is model- -agnosig,, meaning that it can be applied to any robust response-predictin model of biological neurons, further demonstrating its flexibility.
4.2 APPLICATION ON MACAQUE V1 NEURONS
We then applied our method response-pres pereonni-prediitingg model of a population of biological neurons recorded from the primary visual cortex of two macaques, using the response- response-predicting model from Baroni et al. (2023) trained on the publicly available dataset from Cadena et al. (2024). Briefly, this dataset contains neural responses from 458 VI neurons recorded across 32 sessions as the macaques viewed up to 15,000 unique grayscale ImageNet images per session, while fixating. Images were shown for 120ms. and neuronal responses were measured by counting spikes from 40 to 160ms post- image onset. The model from Baroni et al. (2023) is an ensemble of three DNN-based models which were trained predict these neural responses, approaching state-of-the with correlation of 0.73 between predicted and observed trial trial-averaged responses. We selected the 100 best-predicted neurons (53 neurons from Macaque and 47 from Macaque 2) to ensure accurate modeling of their encoding properties (Fig. (3).
predictive performance
Applying the template learning step, our method learned invariance manifolds (Fig. BA) that max- imally activate the neurons (fgg.B.. diagonal entries). We then performed the template matching step align the invariance manifolds across neurons, resulting in 100-by-100 activation matrix. As with the simulated data, we applied hierarchical clustering to this matrix to group neurons based on how their invariance manifold activated other neurons after matching (Fig. 3))
To identify meaningful clusters, we selected dendrogram cut-off threshold that minimized the number of clusters while ensuring that neurons within each cluster shared similar invariance properties, as verified through visual inspection of their invariance manifolds. This resulted in five clusters containing multiple neurons (Fig. A). We identified two large clusters corresponding to simple and complex cells (Fig. A-B, Clusters 1 and 2). Neurons in Cluster 1 displayed selectivity to Gabor patterns with no invariances, whereas neurons in Cluster 2 exhibited phase invariance. The remaining smaller clusters did not show uniform orientation tuning across their receptive fields, but
8

---

Published as a conference paper at ICLR 2025
A
2
Naurons (order by clustaringt
Consistency
Distribution neurons across animals
Genera lization across models
across seeds
Proportion rauurrn
to Mode Inrattunn merntuuneemaa mantedd
1.
Relelive www.iieeeiiiieeee activelion
2.8
Similarity
Tetron DFf cluster
0.6
04-
2.4
9,90
02-
Mociiqu Macique
02
Neurons (order by clustering)
0.-
3.0
0.0
Resporse to Model2
0.5
10
Cluster 1D
inverisnce manito
Figure 3: Results on macaque VI neurons. A) Examples of learned invariance manifolds, organized by clusters identified in panel B. Each cluster is assigned numerical ID and color. The top row shows invariance transformations (or their absence) for representative neurons from different clusters. The bottom row displays an example image within the invariance manifold for multiple neurons for each cluster. B) Activation matrix ordered by hierarchical clustering applied to rows, revealing clusters of neurons with shared invariance properties. C) Consistency of results across seeds. Left barplot: similarity scores of activation matrices across seeds. Similarity of diagonal entries is shown in red, off-diagonal same cluster shown green, and off off-diagonal different cluster shown in gray. Right barplot: similarity scores of learned manifolds (red), aligned manifolds of neurons within same cluster (green) and from different clusters (gray), across seeds. D) Distribution of neurons within each cluster across different animals. E) Generalization across models. Invariance manifolds from model 1 (which was used for the previous analyses) were shown to model 2 (a model with different architecture predicting the same neurons). The y-axis shows the response of model to model I's invariance manifolds, and the x-axis shows the response of model 2 to its own invariance manifolds.
instead exhibited selectivity to more complex patterns, diverging from the traditional classification of VI neurons into simple and complex cells. For example, Cluster 5 contained neurons selective to checkerboard-like texture patterns and showed invariance to local alterations of these patterns. All identified clusters contained neurons recorded from both macaques (Fig. 3)), confirming that analysis did not detect inter-subject differences, reflecting consistent sensory coding across the two animals. As we did with the simulated neurons, we tested the robustness of our method applied the macaque VI neurons across different seeds. Our method produced consistent results in terms of the learned and aligned manifolds (Fig. BC) and cluster consistency (Adjusted Rand Index 0.75).
Finally, to rule out the possibility that our results were due to idiosyncrasies of the response- predicting model. we trained a second model with different architecture (details in Appendix H) on the same set of neurons (Fig. B). We refer to the initial model as Model and the newly introduced model as Model We presented the invariance manifolds obtained from Model to their corresponding neurons in Model 2. Then, we learned the invariance manifolds of the same neurons directly from Model 2 and compared the activations resulting from these two procedures. On average, the invariance manifold from Model 1 activated the corresponding neuron in Model 2 89% of the strength of its own manifold, confirming that the invariance manifolds from Model strongly activate the same neurons in Model 2 and generalize well (Fig. 33
Together, these results demonstrate that our method can robustly identify common invariance proper- ties of visual coding in biological neurons.

---

Published as a conference paper at ICLR 2025
5 DISCUSSION
We introduced a novel method to identify and align the continuous invariance manifolds of visual sensory neurons, enabling systematic characterization of visual coding across neural populations. By combining DNN-based response-predicting models of biological neurons, an INR, and affine transformations learn and align single-neuron invariance manifolds, we provide a principled approach to quantify and compare invariances irrespective of nuisance receptive field differences. Analyzing the activation matrix resulting from invariance alignment provides insights into the functional differences among neurons, enabling grouping into distinct functional clusters based on shared invariance properties.
When applied to Gabor-based simulated neurons, our method accurately learned the true underlying invariance manifolds and successfully aligned them across neurons, identifying clusters corresponding to the ground truth neuron types. Applying our method to 100 well-predicted macaque VI neurons, we identified five clusters: two corresponding to traditional simple and complex cells, and smaller clusters with invariances to more intricate patterns. For example, one class responded to checkerboard- likee textures and was invariant to phase variations in multiple directions, exhibiting complexity beyond the simple/complex dichotomy.
We currently cannot fully exclude the possibility that these clusters result from recording artifacts or data preprocessing steps, such spike sorting. Ultimately, dedicated closed-loop neurophysiological experiments would need to be conducted to empirically validate the clusters. However, there is some evidence that these clusters could reflect biological reality. First, all selected neurons exhibited high predictive performance on a test set of images that were not used for training the response- predicting model, suggesting that the model accurately captured the encoding properties of these neurons. Second, all clusters contained cells from both monkeys, indicating that the RF patterns and invariances generalize across animals. Third, the invariance manifolds learned by one model highly activated the same neurons of another model with a different architecture, indicating that the clusters, which are based on the learned invariances, are likely not an artifact of the network used to fit the cells, but reflect property of the underlying data. Thus, our method could have potentially found novel types of invariances in macaque V1 neurons, similar to the more complex invariances found in mouse visual cortex (Ding al. 2023).
While our approach has merits, it is not without limitations. First, the computational complexity of the method scales quadratically with the number of neurons, making it relatively expensive to apply on larger neural populations. Second, aligning complex invariance manifolds spatially is not necessarily convex problem, which could complicate the alignment process for neurons with more intricate invariances. Finally, in this work we focused on one-dimensional invariance manifolds. While this appears be sufficient for macaque V1 data, other cortical areas might require dimensioaa invariance manifolds. Nonetheless, Baronn et al. (2023) showed that INRs can be used to learn higher-dimensionnl invariances.
Looking ahead, several promising directions can be explored to extend this work. Improving the scalability of the approach facilitates its application to larger neural populations. The flexibility of our framework allows exploring transformations beyond affine, enabling alignment of invariances up to more complex transformations. Additionally, our method can be easily applied to other species, potentially identifying novel invariance properties and enabling cross-species comparisons to reveal conserved or divergent functional principles across different neural systems.
In summary, the method we presented is novel tool for exploring neural invariances in biological data at the population level, enabling the identification of neuron clusters with shared invariance properties, with the potential to uncover new principles of neural computations in the visual cortex. These principles could also inform the design artificial neural networks, helping to bridge the discrepancies between biological and artificial systems highlighted by Feather et al. 2019 2023
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
The complete implementation of our method, including a Docker container for easy setup, can be found in https:: /ttp::://gttuiie com/ sinzlab/ laminn All relevant training configurations, hyperparameters, and implementation details are provided the appendix for full transparency. The experiments were conducted using Python 3.9, PyTorch 1.13.1, and CUDA 1 11.7. However, our method is also compatible with the latest stable versions: PyTorch 2.6.0 and CUDA 12.4. Running the complete pipeline on simulated neurons took approximately hours single V100 GPU (one seed, total number 36 simulated neurons). For macaque V1 neurons (total of 100 neurons), each neuron's ss template learning and matching experiment was separately and took approximately 15-20 minutes on a single A100 GPU (one seed).
ACKNOWLEDGMENTS
We would like to thank Noselab GmbH, particular Thomas Heydler and Arno Schäpe, for supporting MB. LB and JA were supported by the EU Horizon 2020 Maria SklodowskaCurie grant agreement No 861423. JA was supported by the ERDF-Project Brain dynamies, No. CZ.02.01.01000022 0000002240000044433333 and by the grant no. 25- of the Czech Science Foundation (GACR). FHS was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms Project-ID 276693517 17, and SFB 1456, Mathematics of Experiment Project-ID 432680300. Computing time was made available on the high-performance computers HLRN-IV at GWDG at the NHR Center NHR @ Göttingen. The center is jointly supported by the Federal Ministry of Education and Research and the state govermments participating in the NHR (www.nhr-verein.der wtt.rrreeineeeeeseeerrrrrrreee

---

Published as a conference paper at ICLR 2025
REFERENCES
Matan Atzmon and Yaron Lipman. Sal:
Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2565- 2574, 2020.
Luca Baroni, Mohammad Bashiri, Konstantin F Willeke, Ján Antolik, and Fabian H Sinz. Learning invariance manifolds of visual sensory neurons. In NeurIPS Workshop on Symmetry and Geometry in Neural Representations, pp. 301-326. PMLR, 2023.
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 364(64439) 2019.
Max F Burg. Thomas Zenkel, Michaela Vystrčilová, Jonathan Oesterle, Larissa Höfling, Konstantin Willeke, Jan Lause, Sarah Müller, Paul G Fahey, Zhiwei Ding, et al. Most discriminative stimuli for functional cell type clustering. ArXiv, 2024.
Santiago A Cadena, Marissa A Weis, Leon A Gatys, Matthias Bethge, and Alexander S Ecker. Diverse feature visualizations reveal invariances in early layers of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 217-232, 2018.
Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS computational biology, 15(4):e1006897 2019.
Santiago A Cadena, Konstantin F Willeke, Kelli Restivo, George Denfield, Fabian H Sinz, Matthias Bethge, Andreas S Tolias, and Alexander S Ecker. Diverse task-driven modeling of macaque v4 reveals functional specialization towards semantic tasks. PLOS Computational Biology, 20(5): e1012056, 2024.
James R Cavanaugh, Wyeth Bair, and J Anthony Movshon. Nature and interaction of signals from the receptive field center and surround in macaque vI neurons. Journal neurophysiology: 88(5): 2530-2546, 2002.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
P Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014.
Zhiwei Ding, Dat T Tran, Kayla Ponder, Erick Cobos, Zhuokun Ding, Paul G Fahey, Eric Wang, Taliah Muhammad, Jiakun Fu, Santiago A Cadena, al. Bipartite invariance mouse primary visual cortex. bioRxiv, 2023.
Alexander S Ecker, Fabian H Sinz, Emmanouil Froudarakis, Paul G Fahey, Santiago A Cadena, Edgar Y Walker, Erick Cobos, Jacob Reimer, Andreas Tolias, and Matthias Bethge. A rotation-equivaarian convolutional neural network model of primary visual cortex. arXiv preprint arXiv: 10504, 2018
Jenelle Feather, Alex Durango, Ray Gonzalez, and Josh McDermott. Metamers neural networks reveal divergence from human perceptual systems. Advances in Neural Information Processing Systems, 32, 2019.
Jenelle Feather, Guillaume Leclere, Aleksander Mądry, and Josh H McDermott. Model metamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience, 26(11):2017-2034 2023.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7154-7164, 2019.
Ben M Harvey and Serge O Dumoulin. The relationship between cortical magnification factor and population receptive field size human visual cortex: constancies cortical architecture. Journal of Neuroscience, (38): (1338):3360 33604-13612 2011.
12

---

Published as a conference paper at ICLR 2025
David H. Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architec- ture in the cat's 's visual cortex. The Journal of physiology, 160(1):106, 1962.
David Klindt, Alexander S Ecker, Thomas Fuler, and Mathhis Bethge. Neural system identification for large populations separating "what"and and "where". In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Konstantin Klemmnns Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay K Jagadish, Eric Wang. Edgar Y Walker, Santiago A Cadena, Taliah Muhammad, Erick Cobos, Andreas S Tolias, et al. Generalization in data-driven models of primary visual cortex. BioRxiv, pp. 2020-10, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4)9999-1009, 2019.
Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(11):101--102; 1999.
Fabian Sinz, Alexander Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. Advances in neural information processing systems, 31, 2018.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-awarc neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
Rudi Tong, Ronan da Silva, Dongyan Lin, Arna Ghosh, James Wilsenach, Erica Cianfarano, Pouya Bashivan, Blake Richards, and Stuart Trenholm. The feature landscape visual cortex. bioRxiv, pp. 2023-11, 2023.
Ivan Ustyuzhaninov, Santiago A Cadena, Emmanouil Froudarakis, Paul G Fahey, Edgar Y Walker, Erick Cobos, Jacob Reimer, Fabian H Sinz, Andreas S Tolias, Matthias Bethge, et al. Rotation- invariant clustering of neuronal responses in primary visual cortex. In International Conference on Learning Representations, 2019.
Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12): 2060-2065, 2019.
Binxu Wang and Carlos R Ponce. On the level sets and invariance of neural tuning landscapes. arXiv preprint arXiv:2212.13285 2022a.
Binxu Wang and Carlos R Ponce. Tuning landscapes of the ventral stream. Cell Reports, 41(6). 2022b.
Konstantin Willeke, Kelli Restivo, Katrin Franke, Arne F Nix, Santiago A Cadena, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Alexander S Ecker, et al. Deep learning-driven characterization of single cell tuning in primate visual area unveils topological organization. bioRxiv, pp. 2023-05, 2023.
JR Wilson and Murray Sherman. Receptive-field characteristics neurons in cat striate cortex: changes with visual field eccentricity. Journal of neurophysiology, 39(3):512-533, 1976.
Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3)3356-36, 2016.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
TABLE OF CONTENTS
MEI generation
15
Template learning details
16
Robustness to "near" and "distant" latent point confi oonfiguration
17
Template matching details
19
Gabor-based simulated neuron implementation
20
Similarity score between manifolds
21
Comparison with model-based clustering algorithms
22
Macaque response- models details
25
Supplementary figures
26
14

---

Published as a conference paper at ICLR 2025
A MEI GENERATION
For a given neuron, We obtained the Maximum Exciting Image (MEI) by directly optimizing image pixel values to maximize its response over 1000 steps with a step size of 10. Typically, the optimization converged well before reaching the maximum steps.
Generating single MEIs prior learning invariance manifolds critical for reasons. First, provides the maximal neuron response OMEE, which is used to normalize the balance between the activation term and the contrastive term in the loss function, ensuring consistent importance across different neurons. Second, from an MEI, it is possible to compute an MEI mask that isolates the receptive field (RF) of biological neurons, as described in (Ding al., 2023).
MEI masks were computed following Ding et al. (2223 by identifying the convex region where pixel values exceed 0.5 standard deviations. During template learning, applying these masks while calculating the contrastive regularization term of the objective ensures that variations in pixel values are encouraged only within the neuron's RF.
15

---

Published as a conference paper at ICLR 2025
B TEMPLATE LEARNING DETAILS
The initialization weights for the fully connected layer in the Implicit Neural Representation (INR) were sampled from a Gaussian distribution with o 0.1. For positional encoding of the pixel coordinates, we used Fourier features with 50 dimensions with a projection scale of 10. For the positional encoding of the latent input, we used Fourier features with 50 dimensions with a projection scale of 0.1.
During template learning, to train the INR, we employed jittering grid technique. A grid of N 20 uniformly distributed points E 2 along 1D) periodic latent dimension was passed to the INR each step, generating images {g(=))zEZ. This grid was jittered at every step of the training procedure to allow the latent input to span the entire range. To define 2 and 2t for each point 2i, corresponding to near (positive) and distant (negative) points the latent space, we chose a neighboring radius that spanned 10% of the latent space in each direction, resulting in two near (positive) grid points in each direction for every point 2,. All other points were designated as distant (negative). Contrastive regularization was applied solely within the points of the mask obtained from the MEI. The temperature parameter was set to T = 0.3.
The parameters of the INR were optimized using Adam optimizer (Diederik) 2014) with learning rate of 0.001, focusing on maximizing the activity and contrastive learning objectives as defined in Eq. 1 Training continued for minimum of 500 steps and concluded once specific activity requirements were met by the images generated from the jittered grid sampled in the latent space (activity was monitored every 50 training steps). For experiments involving simulated neurons, the average required activity was set to a = 0.99a and the minimum to a 0.98a For experiments involving macaque neurons, the average required activity was M. and the minimum was = 0.850ME7. accommodating the expectation that the invariance manifold of biological neurons might not exhibit perfect iso-response properties. The strength of the contrastive regularization term, initially set to À = 2, was reduced by a factor of 0.8 via a scheduler whenever the activity term plateaued (patience parameter of 5). This adaptive schedule, coupled with the condition to satisfy activity requirements stop training, allows careful balance between activity maximization and contrastive regularization. As result, the INR can successfully capture neural invariances across neurons exhibiting varying degrees of invariance.
16

---

Published as a conference paper at ICLR 2025
C ROBUSTNESS TO "NEAR" AND "DISTANT" LATENT POINT CONFIGURATION
The contrastive term in the template learning objective (Eq. T relies on specifying positive and negative examples for given latent point zț. Positive examples images corresponding to latent points 2 near 2;, while negative examples are images corresponding to distant points 2 This formulation encourages similarity among nearby images and diversity among distant ones, promoting a smooth and diverse learned manifold. Here we assess the effect of "near" and "distant" latent point configuration on the smoothness of the learned manifolds, and their resulting activation. Specifically, we used complex cell (Fig. (STA) and applied our method with five different settings of "near" vs "distant" range (Fig. SIC): 5 %, 10%, 20%, 40%, and 45%. Each value represents the radius of the
A
Ground truth (GT) invariance manifold
B
Smoothness of GT inveriance manifold
2.0
Euclidean distance
maa idx
1.5
10
Imae
0.5
20
0.0
10
20
Image idx
C
Z+ range: 0.05
0.10
0.20
0.40
0.45
Latent value idx
10
20
Latent value ldx
10
"near" point
"distant" point Z_
E
Image idk
10-
mage idx
F
Activation
10
20
Image idx
Figure S1: The effect of "near" and "distant" latent point configuration. A) Ground truth invariance manifold of a complex cell. B) Euclidean distance matrix showing pairwise distances between images sampled from the GT manifold at equidistant phases. C) Neighboring mask illustrating "near" (red) and "distant" (blue) points based on varying definitions of proximity (Z, range: 0.05-0.45). Each row corresponds to different 2, threshold. D) Learned invariance manifolds across different proximity definitions. E) Euclidean distance matrices between image pairs sampled from the learned manifolds at equidistant points in the latent space. F) Neural activation curves elicited by images sampled from the learned manifolds.
17

---

Published as a conference paper at ICLR 2025
neighboring percentage of the whole latent space. We sampled 20 points from ID latent space. Consequently, 5% represents "near" points (1 on the left and on the right) which the minimum number of "near" points possible. Similarly, 45% represents 18 points (9 on the left and 9 on the right), which the maximum number of "near" points possible. Our method successfully learns smooth parametrization of the underlying manifold for most of the "near" percentages considered (Fig. (SD,EE. Only in the extreme cases where "near" points represent the majority, smoothness degraded (Fig. STTD.E last two columns). This is expected, this solution improves the contrastive score by finding a balance between ensuring high similarity with (roughly half of) the "near" points and maximizing the cosine distance with the (few) "distant" points. Overall, these results suggest that our method presents robustness to the exact setting of "near" vs "distant" point, especially when "near" points represents small minority.
18

---

Published as a conference paper at ICLR 2025
D TEMPLATE MATCHING DETAILS
For each pair of template and target neurons, we performed template matching to maximize the target neuron's 's response to the template invariance manifold fo(go- (Aa (x), through an affine transformation y).
To minimize the risk of converging on local minima, we initialized the affine transformation by centering the template's invariance manifold on the receptive field of the target neuron, as determined by the centroids of their MEI masks. We then uniformly rescaled the coordinates based on the size ratio between the MEI masks of the target and template neurons. Finally, we rotated the coordinates to the angle that yielded the maximal neural response.
We optimized the parameters of the affine transformation using the Adam optimizer with learning rate of 0.001. Training was halted when the average activity of the target neurons ceased to increase, with patience parameter set to 15.
To accommodate GPU memory constraints during template matching for a large number of neurons, we batched the procedures into groups of or 50 neurons, depending on the available computational resources.
19

---

Published as a conference paper at ICLR 2025
E GABOR-BASED SIMULATED NEURON IMPLEMENTATION
We implemented simulated neuron models presenting various invariances using Gabor filters. We obtained simple cells linear-nonlinear systems using normalized Gabor filter and final non- linearity (ELU + 1)/2 which ensures non-negative responses and a unitary maximal response for images with unit norm. To model complex cells, we used a set of normalized Gabor filters with shared parameters but different phases. To achieve phase invariance, the model selects the filter that produces the highest output value, which then passed through the same final nonlinearity. Similarly, for neurons exhibiting more complex invariances, we obtained sets of filters by combining multiple Gabor filters with parameters that were obtained by varying an underlying 1D continuous variable, and normalizing them. To achieve the desired invariance, analogous to the process used for complex cells, the model selects the highest output value among those generated by the Gabor filters, which is then passed through the same final nonlinearity.
20

---

Published as a conference paper at ICLR 2025
F SIMILARITY SCORE BETWEEN MANIFOLDS
We define the similarity score between two manifolds as
score = 2
1 = T1
max (sim(( (2)())))))))))) (z)). +
Σ m ==1
max (sim(gi(4)) (2
where gı (2t) and g2(2,) represent the images on the two manifolds, parametrized by the mappings of two INRs, gi and 82. This measure computes the highest cosine similarity between each image one manifold and the images on the other manifold and vice-versa, then averages these quantities within manifolds and across manifolds to get measure of similarity between two manifolds.
21

---

Published as a conference paper at ICLR 2025
G COMPARISON WITH MODEL-BASED CLUSTERING ALGORITTMM
In this section, we compare our method to the model- clustering approaches proposed by Klindt (et al. (2017) and Ustyuzhaninov et al. (2019). Briefly, both methods rely on DNN-based models composed of two main parts: core and readout. The core is implemented 2D convolution layers, extracting nonlinear features across spatial locations. These features are shared across all neurons but are linearly combined by the readout at specific spatial locations to predict each neuron's 's response. While the core captures shared computations across the neuronal population, the readout's neuron-specific weights, which serve as proxies for neuron-specific computations, can be used to cluster neurons.
neural network with
Both methods can cluster neurons independently of differences in RF position, as the convolution operation is conslation-equivariann extends this by incorporating rotation-equivarian convolution layers. This allows the core to learn nonlinear functions not only at different spatial locations but also across different orientations, enabling clustering invariant to both translation and rotation using the feature weights of the readout. While these approaches have proven effective for identifying functional clusters invariant to shifts (Klindt et al. 2017) and rotations (Ustyuzhaninov et al., 2019), our method generalizes these invariances to include all affine transformations, and can straightíorwardly be generalized to more powerful transformations as well. Here, we provide an empirical comparison of the clustering results obtained with these different methods.
However, the model proposed by Usttuzhaninov et al. (2019)
Simulated neurons To keep this experiment focused on assessing how well our approach accounts for transformations beyond translation (Klindt et al. 2017) and rotation (Ustyuzhaninov et al. 2019). we considered two types of neurons: odd and even simple Gabor cells. For each type, we considered two different RF sizes, two different orientations, and five randomly sampled locations, giving rise to 20 simulated neurons per cell type, and 40 in total (Fig. S2AA). To keep the demonstration and its results clear, we focus on our method's s ability to cluster neurons in way that is invariant against transformations of pixel coordinates and did not demonstrate another ability of our model: to cluster by invariances.
Implementation of model-basee methods We implemented the methods proposed by Klindt et al. (017) and Ustyuzhaninov et al. (2019) using convolutional and rotation -uution-equivariaat convolutional models (using steerable filters as in Ecker et al. (2018)) respectively. Given the linear- nonlinear nature of the cells considered (implementation details in AppendixE), we used a single-layer core followed by Gaussian readout as in Lurz et al. (2020) and (ELU 1)/2 final nonlinearity. The Gaussian readout is special case of the factorized readout which was used by Klindt al. (2017). Specifically, while the factorized readout lcarns spatial mask for each neuron learn relevant spatial locations (i.e., neuron's ' RF) for predicting neural responses, the Gaussian readout learns a single location in the spatial dimensions for each neuron and linearly combines features across channels at that specific location. Thus the Gaussian readout a factorized readout where the spatial mask is reduced single pixel on the output tensor of the core. This results in a significant reduction in the number of readout parameters and has been shown to be more data-efficient (Lurz et al.) 2020).
To minimize redundancies in features learned by the core, we enforced readout feature weights to be positive and adjusted the model capacity by fixing the number of channels to eight for the convolutional model (corresponding to cell type x number of sizes x number of orientations) and four for the rotation- -equiir--quvvaiiant convolutional model (corresponding to cell type x number of sizes). For the rotation rotation-equivarian equivariant model, we uniformly discretized the full 2x range of orientations into four possible values E {0, π x/2, n, 3 3r/2), matching the distance between the orientations considered in our simulated population (i.e.. 0 and T/ 2/2). To further avoid redundancies and simplify training, we fixed the readout position for each neuron to its correct location, to focus the training of these models on learning the convolutional kernels and neuron-spest -percffc feature weights.
For the remainder of this section, and in Fig. $2 refer to the convolutional (Klindt et al. 2017) and rotation- eequvirriiiuvvaiiattt convolutional (Ustyuzhaninov et al. 2019) models as CNN-based and RotEgCNN- based, respectively.
Training of model-based methods presented once, grouped in batches of size 10, minimizing the mean squared error between the ground truth responses of our simulated simple cell models and the predictions of the convolutional architectures. Similar to Klindt et al. (2017), to encourage the models learn distinct kernels and
Both models were trained on 150,000 white noise images
22

---

Published as a conference paper at ICLR 2025
Even simple cells
Odd simple cells
A
rototion
rotation
size
rotation
B
CNN based model
RotEqCNN based model
RotEgCNN-base mode feature weight similarity after ignment
Feature weight Va
020
n=5
nwS
S
Feature weiight uc
0.015
0.015
6.010
Neurons
similantyy
0,005
0.005
Neurons
Leamed
Learned
E
Predictive performance (correlation)
method app ied CNN-based model
method applied
RoLEqCNN- -based model
RotEqCNN- based model
1 0.000-
0.995
Neurons (order custeringg
Neurons (order by dusseringg
975
0.996
950
0.950
0.975
1.000
INN-based model
Neurons (order by clustering)
Neurons (order by clustering)
Figure S2: Comparison with model-based clustering algorithms. A) Simulated neuron population, containing odd and even simple cells with different sizes, 2 different orientations, and different randomly sampled positions. B) Feature weights of CNN-based model (Klindt et al. 2017) (017).Thr The x-axis represents the learned kernels, while y-axis shows the corresponding feature weights for each neuron. Colors correspond to neurons colors in panel A. Neurons with the same neuron type, rotation, and size can be clustered together as they present similar feature weights. C) Feature weights of RotEgCNN-based model Ustyuzhaninov et al. (2019), while disregarding the specific orientation, by marginalizing (i.e., summing) the weights across different rotations of the same kernel. The x-axis represents the learned kernels, while the y-axis shows the corresponding marginalized feature weights for each neuron. Colors correspond to neurons colors panel A. D) Cosine similarity between feature weights after aligning the weights regardless of their orientation. Neurons belonging to the same type and sharing size have high cosine similarity. Colors on the top and on the left are the same as panel A. E) Scatter plot showing predictive performance (in terms of correlation to target responses) of CNN-based model vs RotEg based model. F) Resulting clustered activation matrix from the application of our method on the CNN-based model. Dendogram colors the same as in panel A. G) Resulting clustered activation matrix from the application of our method on the RotEqCNN- model. Dendogram colors are the same in panel A.
allow clustering of neurons based on distinct functional properties, we applied an LI regularization to the readout feature weights with strength coefficient of 0.01. This setup enabled the models to effectively capture the functional diversity of the neuron population while maintaining both interpretability and efficiency. For training, used the Adam optimizer with learning rate of 0.01.
23

---

Published as a conference paper at ICLR 2025
Clustering via model-based methods Both models achieved high predictive performance, computed as the correlation between the ground truth responses of our simulated simple cell models and the predictions of the trained models to newly sampled white noise images (see Fig. (S2E), and learned to associate different neurons to different channels, grouping them according to the design of their architecture. Specifically [S2] shows how, the end of our training of the CNN-based model, neurons of the same type, size, and orientation learn similar feature weight vectors independently of their position. Consequently, while clustering based on these feature weights is invariant to differences in location, it does not disregard differences in size and orientation.
Similarly, Fig. |S2C and D show how, at the end of our training of the RotEqCNN- based model, neurons can be clustered using their feature weights according to differences in type (odd vs even) and size, disregarding differences in orientation and position. Specifically, neurons performing the same computation but with different orientations picked the same kernel (i.e., the same computation), which in the RotECCNN-banl -asee model present with different rotations. In this case, a simple marginalization based summing feature weights across the different rotations is sufficient to cluster neurons with unique type and RF size, disregarding RF location and orientation (Fig. S2C).
Additionally, we performed an alignment procedure similar to Ustyuzhaninov et al. (0119) compare feature weights irrespective of rotational differences. By finding the optimal rotation that aligns the feature weights of two neurons, orientation. The results, shown in Fig. S2D, highlight that neurons of the same type and size exhibit high cosine similarity in their aligned feature vectors, where the optimal rotation by minimizing the Euclidean distance between weights. This confirms the ability of the method introduced by Ustyuzhaninov al. (2019) in clustering neurons despite variations in receptive field (RF) location and orientation. However, similar to Klindt et (2017). this method remains limited in its ability to cluster neurons irrespective of other nuisance RF properties such as size, which is known to vary with eccentricity in higher mammals' visual systems (Wilson & Sherman 1976
assess their computational similarity while disregarding
found
Cavanaugh
et al. 2002 Harvey& & Dumoulin 201T).
Comparison with our method Lastly, we applied our method, activation based clustering via template learning and template matching, both models. The clustering results are consistent across both architectures (Fig. G2 and group neurons with the same type (odd vs even) together, successfully disregarding differences in RF position, orientation, and size (further confirming the results shown in Fig. Importantly, the consistency of our results across both architectures highlights the model-agnostic nature of our method. That is, as long as an accurate response- peennnstepeiiiinngggg model is used our method yields consistent clusters, which is crucial advantage in not constraining (or biasing) such findings or analyses to specific architectures or their implementation. An additional important observation is that the activation matrix, which contains the activations after template matching, presents relatively high values between neurons of different types. This reflects the fact that the neuron types considered, odd and even simple cells, differing only in phase, perform quite similar computations, as when properly shifted odd Gabor patterns can activate an even simple cell relatively high, and vice versa. In contrast, this computational similarity is not visible in the feature weights of the model-based methods.
24

---

Published as a conference paper at ICLR 2025
H MACAQUE RESPONSE-PREDICTIIN MODELS DETAILS
In our experiments on macaque V1 neurons, we employed the model used by Baroni et al. (2023) to predict neural responses 93x93 pixel grayscale images, representing 2.65 degrees of the visual field. This model comprises an ensemble of three DNN-based models, each trained separately to minimize the Poisson loss between predicted and recorded neural responses. The final predictions from the ensemble model are obtained by averaging the outputs of individual models.
Each DNN-based model composed of two main parts: a core and a readout. The core is implemented as neural network with 2D convolution layers, extracting nonlinear features across spatial locations. These features are shared across all neurons but are linearly combined by the readout neuron- specific spatial locations, with the resulting values passed through a final ELU + 1 nonlinearity, to predict each neuron's 's response, More specifically, the core consists of three layers of depthwise- separable convolutions, each with kernel size of 24, 9, and 9, respectively, 32 feature channels per layer, followed by batch normalization, ELU nonlinearity, and, for every layer except the first one, a squeeze-and -ndeecctioonniiiii block. To predict single- neuron responses, each oNN-based model employs a pyramidal readout (Sinz et al. 2018) followed by an ELU + nonlinearity.
To assess the generalization of method across different models, trained second ensemble model. Similar to the first ensemble model, this ensemble model includes three separately trained DNN-based models. However, in contrast, for each DNN-based model used the Gaussian readout introduced byrz et al. (2020). We also did not use the squeeze squeeze-and-excitation blocks, while the other aspects of the models remained the same.
Following the training approach of Baroni et al. 2023). we 7-scored the stimuli before training each model in the ensemble using the Adam optimizer (Diederik] (014) with an initial learning rate of 0.004, a batch size of 128, and a learning rate reduction by factor of 0.3 up to three times when there was no improvement validation loss for three consecutive epochs. This second ensemble architecture achieved strong predictive performance, with correlation of 0.71 between predicted and observed trial-averaged responses, comparable the predictive performance of the first ensemble, which achieved a correlation of 0.73.
25

---

Published as a conference paper at ICLR 2025
I SUPPLEMENTARY FIGURES
A
Predictive performance
B
Predictive performance across models (correlation)
across clusters
0.95
Correlation
0.90
Model
Cluster ID
Macaque1
Macaque2
0.80
0.80
0.95
Cluster
Model 1
Figure S3: Predictive performance of esspone-prediectoe models. A) Predictive performance of Model (which was used for the main results) in terms of correlation to trial-averaged responses on left out images (i.e., test set). B) Predictive performance of Model vs Model 2 in terms of correlation to trial-averaged responses on left out images (i.e., test set). Model 2 has different architecture was trained on the same set of neurons as Model
26


Reviews:
[{"review_id": "bi73EPgqeb", "reviewer": "Reviewer_oPYM", "summary": "This paper adapts and builds on work that optimizes for an “invariance manifold” of maximally exciting inputs to neurons. The authors introduce an affine transformation that can be applied to learn a minimal similarity between one neuron’s invariance manifold and another’s. They show that this allows efficient clustering of neuron properties in simulations of macaque V1 responses and in neural predictions obtained from “digital twin” models of macaque V1 responses.", "questions": "A few easy-to-address concerns would move this paper into a clear acceptance for me: \n1) The paper states that it tests the method on “biological neurons from macaque V1” (line 72). This is not true. The paper tests the method on *models* (i.e. additional simulations) trained to predict the response of V1 neurons (digital twins). There is no biological neural data in the paper to validate the method, and thus the findings may just be properties of the models and not properties of biological neurons. The wording in this section and elsewhere in the paper MUST be changed, as currently it is misleading to readers. \n2)\tUsing the phrase “trivial” to describe known properties of receptive field attributes like position, size, and orientation does not seem appropriate. These coding properties are thought to be *fundamental* to the organization of neurons in primary visual cortex. For the second bullet point on line 56, Perhaps the authors mean something like “The method should allow for flexibility to account for known variability of neurons, and find common structure given this variability”?  Please rephrase this bullet point as you see appropriate, and also change other locations in the paper where these properties are referred to as “trivial”. \n3)\tI found the explanation of INRs (Lines 136-158) and the associated figure 1 very difficult to follow. This whole section could be improved to concretely explain what was done for this paper, instead of generally explaining INRs and differences from previous work (as the reader might not be familiar with the details of Baroni et al, so the methods should be self-contained in this paper). For instance, in Figure 1, the depiction of two neural networks is confusing, as I believe that the parameters of the response-predicting model $\\theta$ are fixed, but this is not noted? Additionally, what images are used to train the INRs (is there a train/test set? I could not find details of this, or how these images were chosen for the two sets of experiments, in the main text or the supplement). It is also unclear how the modifications allow direct control of the spatial frequency (line 157), and if this is critical it should be better explained. These are some examples, but I would encourage the authors to generally revise this methods section and add appropriate details to the supplement so that the setup is clear to the readers. \n4)\tHow are the parameters for the contrastive objective for “near” and “distant” points determined, and are different results obtained when using a different range for near and distant points? Additionally, it is stated that the formulation encourages a smooth and diverse learned manifold (218-219). It would be good if this was quantified, for instance by showing lines for real points like that in Figure 1 bottom right, to show that there are no jumps etc. when varying the latent parameters. \n5)\tIn section 4.1, a few more details about the model neurons should be provided in the main text so that readers understand the simulations. Additionally, another sentence in the main text describing how the manifold alignment is measured (the right bars on the plots in Figure 2E,F & Fig 3 C) would be good to help  the reader understand the difference between the measured “Activation” and “Manifold” similarity. \n\nI also had a few more general questions about the work: \n\n6) How does this method hold up in the case of the underlying invariance manifold not being continuous in pixel space? For instance, take the simple example of $y=\\texttt{min}(x^2, 1)$. Two points at +/-1 have the same y output (which is the maximum value) but they are not connected. In this sense, there are two discrete invariance manifolds of the maximum responses. I’m not sure how a  case like this would fit into the proposed framework, and so more detail about why a continuous manifold is a reasonable assumption (or at least acknowledging limitations of it) would strengthen the paper.  \n7) In terms of invariances, the work by Feather et al. (Neurips 2019, and Nature Neuro 2023) come to mind. However, this work found that deep neural networks often have idiosyncratic invariances that are not shared across different models (and they also did not attempt to characterize the full manifold of invariances—they just draw samples from it). Do the authors add in regularization priors (such as smoothness etc) to the initial MEI generation to promote more smooth and interpretable MEIs, which could encourage them to be more shared across the different models? Or is there something about the fact that the models used are an ensemble of CNNs, rather than a single one, discouraging idiosyncrasies? It also might be worth mentioning this previous work on invariances in the related work section. \n8) Can some of the “affine transformations” for similarity of the manifolds be cast as a different similarity metric applied to the representations? The work by Williams et al. (NeurIPS 2021) on generalized shape metrics comes to mind. https://arxiv.org/pdf/2110.14739 \n\nTypos and other minor suggestions: \n* Line 53:  “and identifying” is the wrong tense (does not match “compare”) – it should be “and identify” \n* As currently written, the related work section is difficult to understand. It might be worth revisiting this section to unpack some of the jargon. Alternatively, some authors put the related work towards the end near the discussion, which in this case might help the reader understand why each of these points matters for the paper (for me, I had to return to this section after reading the rest of the paper to really understand what was going on).  \n* In Figure 1 bottom right panel, is this real data from one of the experiments or is it just a fake-data visualization? This should be noted. \n* How is noise taken into account for the neural data?  \n* Lines 269-272: I believe wording in this section is too strong – the Gabor pattern of the simple cell will not always maximize the activity of a complex cell? Maybe rephrase, or more concretely explain what is meant. \n* Line 429 – are there really “no invariances” in the Gabor patterns in Figure 3A? It seems a little strong to say this, as there are slight pixel changes.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, the idea behind the paper seems solid. It extends previous work of capturing invariances in an interesting direction of aligning the invariances between different neural populations. The paper is also fairly easy to read and understand, apart from the sections noted below.", "weaknesses": "As currently written, the paper muddles together previous work and what is new in this paper. For instance, in the abstract (lines 14, and followed up in line 21) it is noted that previous work does not model the invariances as a “continuous manifold”, and that this paper learns the continuous invariance manifold. However, I believe that the authors are directly using a method from Baroni et al 2023, which defines a continuous manifold so this isn't actually \"novel\" in this work. More details about weaknesses and ways to improve the paper are covered in the section section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yRmlX9Qimp", "reviewer": "Reviewer_5DFt", "summary": "The authors developed a method to learn the invariant manifold of a visual neuron / encoding model of a visual neuron. Specifically, they used implicit neural representation to parametrize images and uses a circular variable to condition the image parametrization, creating a 1d circular submanifold. Then they optimized the image parametrization via activation maximization and contrastive loss to find the invariance manifold for a given neuron. Additional template matching step through fitting affine transforms allowed the method to account for difference in receptive field size location / rotation allowing for comparison on the population level. They validated the method on synthetic neuronal data and applied it to real neuronal data from monkey V1, finding clustering across neurons.", "questions": "- One related line of work in this domain the authors seem to miss to compare / discuss is [1, 2]. There the authors used deep generative models $G:z→I$ to parametrize natural images. After finding the activation maximizing latent $z^*$, they sampled images from a 2d submanifold around $z^*$ from the latent space of $z$, and directly find the 1d invariance / iso-response manifold (S1) from this 2d slice. They also measured this invariance manifold on various target levels, where the authors seem to miss. \nSo it seems like a comparable method to find invariance transformations for image computable neuron model, that is also highly activating. \nThough targeting slightly different questions, their work seems related and worth mentioning.\n\n[2]: Wang, Ponce, Tuning landscapes of the ventral stream, 2022, Cell Rep.\n\n- Intuitively, the contrastive loss seems a little redundant, do we really need the numerator positive pair part? e.g. it feels like purely by the smoothness of image parametrization, the near by latent should give rise to the similar images. We just need to make distant latent to give distant images. \n( Is it there for numerical stability purpose? )\n- Necessity of “Invariance alignment matrix”\nFor Fig 2CD and 3B, alignment of invariant manifolds, how much of it comes from the similarity of the MEI themselves (applying the fitted affine transformed), how much comes from the invariance manifold? \nIt seems the images from invariant manifold are all “variation on a theme”, i.e. all pretty close to the MEI, so I think MEI will give rise to similar plots as Fig2C,Fig3B. Can you compare the matrix for MEI and the matrix for invariance manifold? \nGiven so what’s the additional information in the “alignment of invariance manifold”?\n- The method is amazing, but what else can the proposed method contribute to visual neuroscience beyond defining cell type clusters?\n- In the end, the proposed method can only find 1 dim invariance manifold. Though we know the neurons can be invariant to much higher dim inv manifold (could be as high as d-1 dim, where d is stimuli dim). \nCan this method be used to figure out the true dimensionality of that manifold?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem of characterizing neuronal invariances systematically in a modern way is important and under-explored.\n- The method is well presented and principled, and innovative in combining INR, contrastive learning, affine fitting in an elegant way. Similar ideas can potentially impact many other fields.\n- Overall, the presentation of the results are stunning and visually pleasing. Kudos on your efforts for making these.\n- The method is very thoroughly evaluated on synthetic and real neural data, including consistency across noise seed, robustness across encoding models etc.", "weaknesses": "- Intuitively and empirically, the closer to the MEI / peak, the “smaller” the invariant manifold [1]. One weakness is the proposed objective is that it didn’t explicitly allow setting a “target level” of $\\alpha_i/\\alpha_{MEI}$ but seek to optimize the activation value (→ 1). This could result in the invariant manifold being too close to the MEI and not having enough variability, which seems to me be the case from Fig. 2. \nA potential simple improvement will be to allow a target level to be set like having $|\\alpha_i/\\alpha_{MEI}-\\lambda|$ in the objective. c.f. Eq.2 in [1].\n\n[1]: Wang, Ponce, On the Level Sets and Invariance of Neural Tuning Landscapes, 2022, PMLR, https://arxiv.org/pdf/2212.13285", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WcPR9rv6Sd", "reviewer": "Reviewer_Nobq", "summary": "This paper proposes a method for categorizing the receptive fields of visual neurons into classes that are invariant to affine transformations. For a given neuron (referred to as the template neuron), they first employ a previously developed method (Baroni et al. 2023) to identify a one-dimensional manifold of images that maximize the response of the neuron (referred to as the template manifold). Then, given another neuron (referred to as target neuron), they identify an affine transformation of the template manifold that maximizes the response of the target neuron. If the transformed template manifold produces strong responses in the target neuron (and the converse holds), then the template neuron and target neuron are identified as belonging to the same \"class\". They apply their method to synthetic neurons and biological neurons to identify classes of neurons with receptive fields that are \"equivalent\" up to affine transformation.", "questions": "- How are $\\mathcal{Z}_+$ and $\\mathcal{Z}_-$ defined? It seems like this could have a significant effect on the learned manifold of stimuli.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall I enjoyed reading this paper. It provides a systematic method for classifying neurons by their receptive fields. As the authors show, the method captures previously identified receptive fields in V1 (simple cells and complex cells) can be used to identify and characterize previously unidentified receptive fields in V1. The general procedure seems promising.", "weaknesses": "As someone who is not familiar with these types of work, it took me some time to appreciate what this paper is doing. For example, the authors state early on that they are seeking a method that \"should allow for a meaningful measure of similarity between two invariance manifolds\". I had no idea what this meant when I first read it. There was no mention of what this \"invariance manifold\" is prior to that statement (it's mentioned in the abstract, but not the introduction). I think that slowly walking through an example (maybe with a figure) with simple cells would be helpful.\n\nA few other comments: While invariance to affine transformation seems reasonable in V1, it's not clear how one would want to define invariance in high-order visual areas, though I don't see this as an issue that needs to be addressed in this work.\n\nThis paper relies on existing methods which are only briefly explained. (I'm not familiar with these works, hence my low confidence score.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "txf0ApfCh8", "reviewer": "Reviewer_aC4g", "summary": "This paper proposes a new method to map invariance manifolds of visual neurons. They combine prior methods to arrive at a new formulation that allows for continuous deformations of MEIs and clustering that marginalizes over affine spatial transformation. They validate their method on simulations and test it on Macaque V1 data.", "questions": "117-119: please clarify\n\nWhy do you use a periodic 1D latent variable? Please justify and explain\n\n152: why remove batchnorm?\n\n153: steps = typo?\n\n155: Are the Fourier features new compared to Baroni? They are pretty standard in Sirens / implicit function modelling. Why do they allow better controlling spatial frequency etc., do you have data to support this?\n\nWhy did you choose cosine similarity as image metric? Might this have shortcomings?\n\n253: neuron + s\n\nWhy don't you use the activation matrix directly as a similarity matrix input to agglomerative clustering, why first compute the cos.sim. between rows? [more data massaging -> more skeptical]", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, cites relevant prior literature and does a good job working through the logic and experiments. Figure 1 is very illustrative and of high quality. The result figures are also very clean and show convincing results. I like the focus on continuous transformations, allowed by implicit representations, rather than pointwise estimates such as Cadena et al. 2018.\n\nThe 'new cell types' on V1 data are exciting, looking forward to closed loop validation.", "weaknesses": "It would be instructive to discuss the concept of a nullspace, as being the input space that the neural response is invariant to. One can think of an MEI as a first order approximation to the response function. Thus, for such a linear function it is quite natural to think about its nullspace aka the invariance manifold. This actually leads to a question: there seems to be a way to game the objective function. One could simply keep the MEI in the RF identical while making large changes outside the RF (for instance, strongly changing backgrounds); How is this avoided?\n\n108: The method in Klindt et al. 2017 performance functional clustering that is invariant to RF position; The method in Ustyuzhaninov et al. 2019 extends this to clustering that is also invariant to orientation; Arguably, the main difference that the affine transformations in this paper add on top of those prior methods are: scaling, shearing and reflections of MEIs\n\nInvariance is a broader concept than just invariance around the MEI. One can have iso-response input manifolds showing different invariances at every activation level of the neuron. Please add this distinction and explain why you only study invariance at the MEI.\n\nIt would be great to add comparisons to prior methods. For instance, your method should be able to solve the functional clustering despite RF shifts like in Klindt et al. 2017; It should also be able to solve the functional clustering despite RF shifts and rotations like in Ustyuzhaninov et al. 2019. I am happy to raise my score if you can add those comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)