Parallel Backpropagation for Shared-Feature Visualization
Authors: Alexander Lappe
Anna Bognár
Ghazaleh Ghamkhari Nejad
Albert Mukovskiy
Lucas Martini
Martin A. Giese
Rufin Vogels
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Computer Vision
Neuroscience
Deep Learning
Feature Visualization
Primary area: neuroscience_and_cognitive_science
TL;DR: A method to visualize shared features between images that drive neural responses.

Abstract: High-level visual brain regions contain subareas in which neurons appear to respond more strongly to examples of a particular semantic category, like faces or bodies, rather than objects. However, recent work has shown that while this finding holds on average, some out-of-category stimuli also activate neurons in these regions. This may be due to visual features common among the preferred class also being present in other images. Here, we propose a deep-learning-based approach for visualizing these features. For each neuron, we identify relevant visual features driving its selectivity by modelling responses to images based on latent activations of a deep neural network. Given an out-of-category image which strongly activates the neuron, our method first identifies a reference image from the preferred category yielding a similar feature activation pattern. We then backpropagate latent activations of both images to the pixel level, while enhancing the identified shared dimensions and attenuating non-shared features. The procedure highlights image regions containing shared features driving responses of the model neuron. We apply the algorithm to novel recordings from body-selective regions in macaque IT cortex in order to understand why some images of objects excite these neurons. Visualizations reveal object parts which resemble parts of a macaque body, shedding light on neural preference of these objects.

Parallel Backpropagation for Shared-Feature Visualization
Alexander Lappe 1.2
Anna Bognár*
Ghazaleh Ghamkhari Nejad
Albert Mukovskiy!
Lucas Martini 1.2
Martin A. Giese!
Rufin Vogels KU Leuven
Hertie Institute, University Clinics Tübingen IMPRS-IS
alexander lappeduniiiin.cc weweeuunntttubiinnnn.eeee de
Abstract
High-level visual brain regions contain subareas in which neurons appear to spond more strongly to examples particular semantic category, like faces bodies, rather than objects. However, recent work has shown that while this finding holds on average, some out-of-category stimuli also activate neurons these regions. This may be due visual features common among the preferred class also being present in other images. Here, we propose a deep-learning based approach for visualizing these features. For each neuron, identify relevant visual features driving its selectivity by modelling responses to images based on latent activations of deep neural network. Given out-of-category image which strongly activates the neuron, our method first identifies a reference image from the preferred category yielding similar feature activation pattern. We then back- propagate latent activations of both images to the pixel level, while enhancing the identified shared dimensions and attenuating non-shared features. The procedure highlights image regions containing shared features driving responses of the model neuron. We apply the algorithm novel recordings from body- regions macaque IT cortex in order to understand why some images of objects excite these neurons. Visualizations reveal object parts which resemble parts of macaque body, shedding light neural preference of these objects.
vuthiiy drivers of neuron
CNN
CNN
image similarity
arg max
parallel backprop
out-of-category driver of neuron
Figure Why does this object image activate IT neurons that are selective for bodies? Our goal is to visually explain responses of category-selective neurons to outside-of-category (ooc.) stimuli. We start with an ooc. stimulus (object), that strongly activates neuron which primarily fires for specific category (bodies). We compute latent CNN activations for the image, as well as for set of within-category reference images. A neuron-specific similarity metric operating on the latent activations finds the reference image most similar to the ooc. stimulus. The proposed parallel backpropagation method then highlights the shared features driving the neural response.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Background. The primate visual system has evolved to process highly diverse set of tasks and stimuli. In higher visual areas, specialized subregions exist, in which neurons preferentially discharge in response to images stemming from a particular semantic class. These areas are often hypothesized to underlie specific computations that may only be relevant for specific semantic concept, but it not entirely clear why they emerge. The most prominent category-selectio
brain regions consist of 'face-cells' (" face patches"), which on average fire more strongly when stimulated with faces than objects I 2 as well as body-selective regions which have been studied to a lesser extent For face-selective cells, it has been shown that object images also elicit responses, albeit more sparsely than face images [6 77. Going further, [6] showed that models trained solely on non-face images could predict responses to face images in macaque face cells. Therefore, it has been argued that face-selective cells are not driven by the semantic concept of a face, but instead respond to visual features that are more common in face than object images. Hence, out-of-category (Ooc.) images may still activate otherwise category-selective cells, as long as relevant features are apparent in the image. As of yet, characterization of these features has remained relatively rough, largely confined to the finding that face cells tend to prefer round objects and body cells tend to prefer spiky objects [8). Other work suggests that only small fraction of response variance in face cells can be explained by simple shape features like roundness [6. While these global tuning properties are useful to understand average preferences of patches, therefore argue that more fine-grained feature characterizations are necessary to understand responses at the single-image and singlc-neuron level.
3 349 5
method for visualizing features of an ooc. stimulus, which are responsible for eliciting high responses from an otherwise category-selective neuron. Our approach relies analyzing the similarity of the image to selected within-category image which displays similar features, as shown in Fig. [] Specifically, after finding a cathin- image with similar, neuron-specific features, we use gradient methods to highlight features extracted from a vision model which are shared between the two images and drive the neural response. This helps to visually answer the question why feature detector that is preferably activated by within-class images, would also respond to the ooc. image in question. The procedure compatible with any backbone visualization method that returns saliency map for hidden units of a convolutional neural network. Further, it is entirely class- for any selective brain region, as well as for studying internal behaviour of artificial vision systems. In this work, we show results for novel set of multi-unit recordings from body-selective regions in macaque superior temporal sulcus. Body cells constitute particularly interesting problem, as different body poses produce vast variability among body images, suggesting rich set of features driving these neurons. We summarize our contributions as follows:
Contribution. To address this gap, we propose nerra----ll--
and can therefore be used
We present novel method for visualizing features driving responses of category neurons to out-of-category images, shedding light on why these neurons do not exclusively respond to within-categor images,
We present results from multi-unit recordings of body-selective neurons in macaque IT cortex, demonstrating that these neurons encode overlapping visual features for bodies and objects,
We apply the proposed visualization method to the data, discussing why some non-body objects activate these neurons.
2 Related work
Category-selective visual brain areas. The largest part of the body of work studying category selectivity in the brain targeted towards face patches (2] M. Several papers have come to the conclusion that face-selective neurons respond to visual features correlated with faces, rather than the semantic concept of a face. 7 generated maximally exciting images for face cells, which activated the neurons strongly but were not rated as like-like by human participants. [9] showed that face cells also respond selectively to pareidolia images, which are objects eliciting perception of face in humans. Neurons still responded after the images were scrambled, destroying perception of face, indicating that responses were driven by low-level features. Recently, (6] successfully predicted responses of face cells to face images, after training linear readout model using solely non- face

---

images. [8] proposed a unified view of IT cortex organization, arguing that selectivity was based on a small set of principal axes of image space. Reducing the dimension allows the authors to state that face cells prefer objects with high scores on principal components corresponding to 'stubby" and 'animate', whereas body-selective cells respond to 'spiky' and 'animate" objects. Recent work [10] challenges the hypothesis of shared coding principles between faces and non-faces in face cells, showing that computational mechanisms in these brain regions are far from being well understood. Our neurophysiological recordings add additional evidence to this debate. Highly relevant to work is that of [ЛЛ, which visualized body cell responses to bodies and some objects by showing fragments of highly activating image to determine most important image regions. This approach showed that a large proportion of cells respond to local body fragments. The main advantages of our computational approach are its high-throughput, allowing to visualize large number of cells and objects simultaneously, and enhancing interpretability of features by analyzing them in the context of the preferred semantic class.
Attribution methods. A substantial body of work exists on attributing behaviour of deep computer vision models to specific image regions. The most common goal is to understand why classification models output the observed class label, given input image, meaning that attribution methods are often applied to the very last network layer. Several families of approaches have been proposed for tackling this problem: The first relies image perturbations like occlusion [2 [3] the second is based on an analysis of latent activations [4] 15]. and the third computes gradients of class activations
w.r.t. pixel intensities [16 17 [8] More recent work [19 20 has shown that some of these methods rely too strongly on the input image itself, being independent of the network to varying degrees. Even though our reweighting scheme is compatible with any attribution method that is computable for latent activations, we therefore rely on vanilla backpropagation in this work, which has been shown to not be biased towards reconstructing the input image. We provide results for integrated gradients (18) in the appendix. Further, relevant to our work is that of [21] which computes saliency maps for image similarity. The approach is similar that of Grad-CAM, as relies on analyzing feature activations before and after a global pooling layer at the end of the network. For that reason, the latter method is mainly suitable for visualizing global similarity as judged by a network trained to do so, rather than more local features relevant for biological neurons along the visual hierarchy in the primate brain.
3 Methods
We propose approach for visual explanations of neural responses oft-of-category (DOc.) stimuli in category-selectiv visual brain areas. The categories currently of most interest in the high-level vision literature are faces and bodies, but the method can be applied to any semantic category. We leverage differentiable neuron models to explain high neural firing rates in response to an onc. image by analysing visual similarity an image from the preferred category which also drives the neuron. Formally, let Zout E Rxxhxx denote an ooc. image yielding high activity from a recorded category- selective neuron. The overall goal of this work is determine the visual features of Toot driving the neural response. We approach this problem in three steps:
1. We learn a linear readout vector w on top of a pretrained CNN S(.) to predict neural responses to within-category stimuli 1in,1:-... Vin.N- The learned vector then incorporates information about which features apparent in within within-category images are relevant for the neural response.
2. We employ a neuron-specific similarity metric based on the learned readout to find a within- category image xin with similar visual features as Tout- Visual inspection of this reference image on its own can yield insights on why Jour activates the neuron.
3. We backpropagate gradients of CNN activations to the pixels of both images and reweight them to highlight features that are
(a) present in Tout-
(b) present in Tin-
(c) highly relevant for the neural response.
Explicitly highlighting shared features helps identify specific image regions responsible for the images' neuron-specific similarity.
3

---

3.1 Modelling neural responses
In recent years, a pretrained CNN backbone combined with trainable linear readout module has become the gold standard in modelling the stimulus-driven variance in neural responses to images [22 23 In these models, an image i E R3xxxx is fed through CNN f(+) up to a predetermined layer to obtain latent representation E IRC. The predicted response for neuron is then computed as
9 = (a, w)
(1)
where w R is learned weight vector, and (+--) denotes the standard dot product. The vector w encapsulates information about which visual dimensions the neuron is tuned to, as large w(() implies that increased activity in feature will strongly increase the predicted neural response. Conversely, (() = 0 implies no change in predicted activity if feature i is manipulated and thus such features can be ignored when trying to understand a neuron's response pattern (assuming perfect model fit). Therefore, training model to predict neural responses to within-catee within-category category stimuli reveals which visual features drive responses within the category. If the model then generalizes to ooc. stimuli, we can infer that ooc. neural responses are driven by features present in both data distributions.
3.2 Neuron-specifie image similarity
If an image Tout strongly drives a neuron, we propose to interpret the relevant image features by studying visually similar image from the category that the neuron is selective to. To quantify similarity, we adapt the common method of computing the cosine similarity of latent activations computed using CNN [24]. Since we have additional information on which latent dimensions drive the neuron, we weight feature by the corresponding weight w(()) Formally, for images æ1, 22 and their corresponding latent activations a1 := f(x1), d2 == f(22), we define the neuron-specific image similarity as
8(11, 22) = (a W Ow)
|0 w 02
where @ denotes the Hadamard product. Since w is often sparse, this formulation will effectively ignore most dimensions and focus solely on those that are highly relevant for predicting the neural response. For the downstream procedure of generating a visual explanation for the neural response to the image Tout, we simply select the most similar image from our within-category dataset, i.c.
Tin = arg max s (Jon: x) : x E De).
(3)
where De denotes the set of within-category images.
3.3 Parallel backpropagation for shared-feature visualization
Even though the images Four and Fi ideally have high similarity metric, the similar features not always clearly identifiable. In some cases, the features might not be easily interpretable, or in other cases, several features might be shared upon visual inspection, such that it remains unclear whether all of them or subset contribute to neural activity.
In order to further highlight shared features contributing to model activity, we compute weighted gradients of the latent model activations w.r.t. to image pixels [16). This method was originally devised for classification models to highlight those image pixels that most strongly influence given class probability. We build on this work to visualize shared features between images by introducing a simple reweighting procedure. Specifically, we strengthen the influence of features that are shared, while attenuating features that are specific to only one of the images. For the sake of brevity, we illustrate the procedure for the gradients of Tou only, the process is analogous for Pin- First of all, note that the gradient of the predicted neural response Your is given by
Hoot
(Aout, w) = Σ (()
a
2xout
O.r
D.ost
Due to the sum rule of calculus, the gradient of the predicted response is given by a weighted sum of the gradients of the latent features, with w(() acting as weight for feature i. Hence, features that are highly relevant for model predictions will dominate the gradient. In turn, those pixels for which an

---

i
0x1
Weighted Sum
CNN
Hadamard Product and Norm
T2
W
CNN
Weighted
Sum
Dimensions
I
a2
Oxz
Figure 2: Sketch the parallel backpropagation method. pre-trained CNN cut off at predeter- mined layer computes latent feature activations. These are backpropagated to obtain the Jacobians of the two activation vectors w.r.t. to the respective images. We then calculate the normalized Hadamard product of the activation vectors and the element-wise square of the learned linear readout vector for the considered neuron. The pixel saliency map is then computed as the sum of gradients of each feature, weighted by the feature's entry in the Hadamard product.
infinitesimal increase would strongly enhance one of the highly relevant features will be assigned a high intensity when plotting the gradient over the image.
For purposes, this gradient map is not satisfactory. First of all, it does not take the other image into account at all, and thus fails to leverage the images' similarity structure. Further, since the gradient weights do carry information theoretically possible to assign high intensity to pixels that would strongly increase features with low activity. To remedy these deficiencies, we propose an adjusted pixel saliency map based on replacing the weights for the features in (4).
whether feature is present the image in the first place, is
Before reweighting the features, first smooth and normalize each gradient to have unit norm, i.c. Bersa -- ||2 1. Smoothing alleviates pixel-level noise and allows the user to determine regions of pixels with high contribution. has been shown that post-processing smoothing step substantially improves the quality of gradient-basee attributions [20]. Normalization ensures that the pixel saliency map is not determined solely by the gradient magnitude, as some features may be more sensitive to pixel perturbation than others. More importantly, it allows us to later bound the norm of the saliency map from above, which substantially improves interpretability, Subsequently, we weight each feature by its contribution the neuron-specift similarity metric s(...) given in (2). Formally, define the weight for feature as
Orout Wr (4)
800
(5)
|iin
Oout C wwlp
and finally the pixel saliency maps
[[ootti Pins w)= = Σ 3(4)
[(Zmn Fouts w) = Σ ((0)
Gout:
(6)
Dzout
axm
The revised weight vector & imposes high weights only on those features that are highly activated in the feature vectors of both images (captured by af and a and are also relevant for the neuron's activity (captured by (1)). Dividing by the product of norms allows to bound the intensity of
5

---

the saliency maps by writing
[| Tim, w) 122111 Σ g(
ooui l2 S Σ ||:30)
(8) Quul
Doout
Ioou
Σ ((3)
Guai 112
Σ g(t) s s(Fi, Fout).
Note that this inequality only holds if in 2 0 and Gout 2 O for all i= L,..., C, as this is needed to ensure ((0) > O. However, this constraint is usually satisfied as latent activation are commonly fed through ReLU layer before extraction. The inequality yields that the total intensity of the pixel saliency map (for either image) as given by its L2 norm is bounded by the neuron-specific similarity. Thus if the images are dissimilar, the saliency maps will have low intensity. Finally, note here that the reweighting procedure is agnostic to the way the salience map for each feature is generated. Any (backpropagation) pixel-attribution method may be used, as long as each latent feature is assigned one saliency map with unity norm.
()
Running these steps successively for one recorded neuron yields as output one ooc. image and one within-catee category image, along with one saliency map per image. To study shared features, we display the images side-by-side with the saliency maps overlayed, as seen in Figs and []]
Experimental setup
which was adversarially trained on ImageNet 226 27 as our CNN backbone. This architecture has shown good fits to neural data To predict the response to image, we feed it through the Resnet up to the last ReLU of layer 4.1. Subsequently, a Gaussian readout [31 reduces the dimension of the Resnet activations from e x ha X w, to e by selecting learned spatial location in the latent feature map from which to extract the final features. The receptive fields of the extracted features cover the entire foreground of the images. Finally, fully-connected layer maps these features to neural responses.
Model architecture. For experiments, we use Resnet-50 25
in several studies 28
29 30
Stimuli. For gathering neural data, we utilized two sets of images. The first consisted of 475
images of a macaque avatar on a gray background (see Fig. [1] 2. The images were subsampled from set of 720 images, in which the avatar appeared in 45 uniquee poses, extracted from nine different behavioural classes, each shown from 16 viewpoints [32]. The second set comprised 6,857 objects from varying categories shown on the same gray background and was combined from the Openlmages dataset (33] as well as several smaller ones [34 35 To test for body selectivity, we used an additional set of 2068 control body images including a variety of species. These images were only used to test for nategory-selectivve the text refer to the original image set showing the monkey avatar. All stimuli were centered with respect to the fixation point. They were shown to the monkey at resolution of 280x280, and were resized to 224 for the Resnet.
of the recorded cells. All other references to body images in
Neural data collection. We recorded multi-unit activity (MUA) from and surrounding two fMRI- defined body category-selective patches (5) in the macaque superior temporal sulcus (STS), using 16-channel Plexon V probes, while the subjects performed fixation task. Since body patch neurons typically discharge sparsely for non-body objects, we employed online stimulus selection: In a first phase, we recorded responses to the set of 475 monkey avatar images. We then trained a model for each channel to predict neural responses to these images. Subsequently, we predicted neural responses our set of object images, and for each neuron we selected the highest and lowest predicted activator, as well as the object most similar to the top-activating body image, according to s(,,)) .). We followed the same procedure for the control body images. Finally, in a second experimental phase we recorded responses to these novel images as well as a subset of 75 of the original body images, to test recording stability. We included recording channels in the analysis if the test/retest stability was higher than .60, as measured by the correlation between responses to body images before and after model fitting. Further, we tested each channel for body wwwtearr--eeeeettieeeeee the selected objects and the selected control bodies using a Mann- Whitney U-test. Both the animal
by comparing the median response to

---

a)
Model Performance
b) Responses to visualized objects
MSB
ASB
Spike Rate (1/s)
Monkey
Monkey
Monkey G
Monkey T
1.0
Correlation
0.8
200
0.6
0.4
0.2
0.0
Recording Channel
-0.3
100 Monkey body test set Objects
25
50
75
Selected object
Recording Channel Rank
Body mean
Recording stability
Object mean
Figure 3: Correlation between predicted and recorded neural responses for held-out monkey body images (blue) and object images (orange). Significant positive correlation for object images demonstrates that the visual features predictive of responses to body images are also predictive of responses to objects. Brown dots show the correlation between responses to the same stimuli in the first and second recording phase. Dashed lines show the average across channels (colored) and the .05 confidence interval for the correlation coefficient under the null hypothesis that p = 0 (black). b) Neural responses the objects for which features are visualized in Figs. I and Responses to visualized objects are higher than mean responses to objects and bodies in the vast majority of cases.
care and experimental procedures adhere to regional (Flanders) and European guidelines and have been approved by the Animal Ethical Committee of Leuven under the protocol number P182/2019. Further experimental details are given in the appendix.
Fitting the neuron model. We split the monkey image set into a training-validationnnes split consisting of 400/50/25 images. The parameters of the Gaussian readout location, as well as the linear weights, were trained simultaneously using the Adam optimizer [36]. to minimize the mean squared error between recorded and predicted responses. We augmented the training data by incorporating silhouettes of the monkey, for which the tuning of body patch cells in mid-STS has been shown to be largely preserved (37]. Further, penalized the seadout-weight using L1/2 regression to sparsify the vector while still allowing for some large weights. We set the learning rate 10-1 and the weight of the regularization to 0.1 After training for 2500 epochs, we selected the model with lowest loss on the validation set. Importantly, the model was not trained to predict responses to non-body images, meaning that it must utilize the same visual features to predict bodies and objects. Models and training runs, as well as the visualization procedure were implemented in PyTorch [38]. To compute the Jacobian of latent features for visualization, we utilized the corresponding functionality of PyTorch's 's autograd. All experiments were run on a single Nvidia RTX 2080Ti.
Results
5.1 Model generalizes from bodies to objects
After training the models on body images, we first test how well they generalize to images of objects. Model performance in terms of correlation between predicted and recorded neural response is shown in Fig. 3 a). 93.7/95.3 % of channels the posterior/anterior region exhibit a significant positive correlation, suggesting an at least partially class-agnostic feature preference. Interestingly, strong performance on the held-out monkey images does not seem to be a necessary condition for strong performance objects. This effect may be caused by body images being highly similar due to fine-grained sampling of poses and viewpoints, compared to high feature variance among objects.
5.2 Feature visualization
Shared features between objects and bodies. Having found that there exists a set of visual features driving neural responses to both objects and bodies, we apply the proposed visualization method to characterize these features. Results are displayed in Figs. (posterior region MSB) and (anterior region ASB). We show results from multi-unit sites for which model performance on the out-of-category images was high (r > 0.4), selecting a subset representing wide range of
5
7

---

Monkey G
Monkey
Figure 4: Results obtained by applying the proposed method to multi-unit recordings from body- selective cells in macaque STS (posterior region). Each subplot corresponds one recording channel. The object on the left is among the most highly activating objects for the channel. The image on the right corresponds to the most similar preferred body.
highlighted features. For neighbouring recording sites, visualizations are often similar, likely due to similar feature preferences. For each channel, we display the image pair with highest neuron-specific similarity among the top-5 activating objects and top- 15 activating bodies. Fig. 3)) demonstrates that the visualized objects activate the corresponding channels more strongly than the average body/object image in most cases.
The method discovers variety of shared features between highly activating bodies and objects. Most of them correspond to parts the body rather than the entire body, which is aligned with previous findings for neurons from MSB. [II]. In fact, specific object parts seems bear resemblance specific body parts in the model's latent space. For example, extended structures appear to activate the same latent dimensions as arms/shoulders. responses to body images also predicts strong responses to other extended structures. We observe objects driving the model due similarity to limbs, tails, heads, torsos well as more diffuse features which are more difficult to interpret. Interestingly, while lot of the observed features could be characterized as 'spiky", we also find neurons which activated by stubby objects. The corresponding bodies show crouched poses without protruding extremities. This demonstrates that the -ingle-maaee and multi-unit level, tuning properties are more ine-grained than previously suggested [8] In some cases, we observe the same object with different highlighted features in different recording channels, indicating that objects may have multiple features that activate different neurons. An example of this can be seen in Fig. 5 where the highlighted features of the same image correspond to a leg and torso of the effective body images of two different channels. Additional results are given in the appendix.
model that relies arms/shoulders explain

---

Monkey G
Monkey T
FLK
GO MALLE
Figure 5: a)/b) Results for ASB region. c) Example results for objects lacking a highly similar body image among the channel's top drivers, and for weakly activating objects. Top row: strongly driving objects. Bottom row: weakly driving objects. Low similarity is properly reflected by low intensity of the saliency map.
Objects without shared features. We find that for some object images eliciting high neural response. the visualization method yields no shared features with any of the top-activating body images (Fig. ()). This could be either due to neurons preferring features not present in the original body image set, or due to tuiig properties not captured by the model. These cases demonstrate the utility of the bound for the norm of the saliency maps given in (7), as the lack of image similarity is clearly reflected by the visualization.
Discussion
Limitations. The proposed approach is made possible through the of differentiable neuron model, which means that the visualization quality depends on the ability of the model to capture neural tuning properties. As models of visual processing further improve in the future, predict that visualization quality will improve accordingly. Further, the visualizations will reflect idiosyncrasies of the underlying attribution method used for generating saliency maps for latent features. Since this backbone can be chosen freely, advances on the topic of attribution methods will also improve our visualizations.
Conclusion. We presented a method for visualizing selectivity of class-selective visual feature detectors when confronted with out-of-class images. Further, we showed that body-selective neurons encode bodies and objects using an at least partially shared feature set. We visualized these features, providing an explanation for why some objects activate body-selective neurons. Future work could involve using the same method for other cedteoor-ssllcctie areas, like face patches. Additionally, one

---

could test these visualizations by presenting highlighted fragments to the subject in closed-loop fashion.
Acknowledgments and Disclosure of Funding
The authors thank Rajani Raman and Prerana Kumar for valuable discussions about this project. Fur- ther, the authors thank C. Fransen, I. Puttemans. A. Hermans, W. Depuydt, C. Ulens, Verstraeten, T. Riyahi, J. Helin, and M. De Paep for technical and administrative support. AI. AB, GKN, AM, LMM MG and RV are supported by ERC-SyG 856495. MG is supported by HFSP RGP0036/0066, BMBF FKZ FE.0IIQQ704 704. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Alexander Lappe and Lucas Martini. Parts of the stimulus images are due to courtesy of Michael J. Tarr, Carnegie Mellon University, http://www.tarrabb.
Contributions. AL, AB, MG and RV conceptualized the study. AL developed and implemented the visualization method. AB, GGN and RV collected the neural data. AL. and AB analyzed the data. AM, AB, LM and AI. contributed to stimulus generation. AL wrote the initial draft of the manuscript, and all authors contributed to the final version. MG and RV supervised the project.
References
[1] Nancy Kanwisher, Josh McDermott, and Marvin M. Chun. The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception. The Journal of Newroscience, 17(11):4302-45 4311, June
1997. ISSN 0270-6474, doi: N0.1523/INUROOSCII.. 11-04302.1997
[2] Doris Y. Tsao, Winrich A. Freiwald, Roger B. H. Tootell, and Margaret S. Livingstone. A Cortical Region Consisting Entirely of Face-Selective Cells. Science, 311(5761):670-6)) February 2006. ISSN 0036-8075, 1095-9203, doi: 10.1 126//cience.. 111993. 19983.
[3] Paul E. Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A Cortical Area Selective for Visual Processing of the Human Body. Science, (3/5539):2470-4773 September 2001. doi: 10.1 122//ciienc..1633.2
[4] Serguei V. Astafiev, Christine M. Stanley, Gordon L.. Shulman, and Maurizio Corbetta, Extrastriate body area in human occipital cortex responds to the performance of motor actions. Nature Neuroscience, 7(5): 542 548. May 2004. ISSN 1097-6256. doi: 10. 1038/nn1241.
[5] Rufin Vogels. More Than the Face: Representations of Bodies in the Inferior Temporal Cortex. Annual Review of Vision Science, 8(1 8()):333445, 2022. doi: 10.1 144/ammnee- 113499.
[6] Kasper Vinken, Jacob S. Prince, Talia Konkle, and Margaret S. Livingstone. The neural code for "face cells" is not face-specific. Science Advances, 9(35) 1736, September 2023. ISSN 2375-2548, doi: 10. 126/sciadv.adg
[7] Alexandra Bardon, Will Xiao, Carlos R. Ponce, Margaret Livingstone, and Gabriel Kreiman. Face neurons encode nonsemantic features. Proceedings of the National Academy of Sciences, 119(16): e21 18705119, April 2022. ISSN 0027-8424, 1091-6490. doi: 10. 10.10733pnas.211 18770 18705119)
[8] Pinglei Bao, Liang She, Mason McGill, and Doris Y. Tsao. A map of spacetspaae cortex. Nature, (7)11)7811111011 July 2020. ISSN 1476-4687. doi: 10. 1038/44/ 186-020-2350-$3
space in primate inferotemporal
[9] Saloni Sharma, Kasper Vinken, and Margaret Livingstone. When the whole is only the parts: Non-holistic object parts predominate face-cell responses to illusory faces. bioRxiv, 2023.
[10] Yuelin Shi, Dasheng Bi, Janis K. Hesse, Frank F. Lanfranchi, Shi Chen, and Doris Y. Tsao. Rapid, concerted switching of the neural code inferotemporal cortex. bioRxiv, pages 2023-12, 2, 2023.
[11] Ivo D. Popivanov, Philippe G. Schyns, and Rufin Vogels. Stimulus features coded by single neurons of macaque body category selective patch. Proceedings of the National Academy of Sciences the United States of America, (13(1))E2450-2455 April 2016. ISSN 1091-6490. doi: 10.1073/pnas. 1520371113.
[12] Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks, November 2013.
[13] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models, September 2018
10

---

[14] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning Deep Features for Discriminative Localization, December 2015.
[15] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient- based Localization. International Journal of Computer Vision, 128(2)3336-359 February 2020. ISSN 0920-5691, 1573-1405. doi: 10. 126--11---122--7
[16] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualis- ing Image Classification Models and Saliency Maps, April 2014.
[17] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplic- ity: The All Convolutional Net, April 2015.
[18] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks, June 2017.
[19] Julius Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In Neural Information Processing Systems, October 2018.
[20] Sukrut Rao, Moritz Böhle, and Bernt Schiele. Towards Better Understanding Attribution Methods. In 2022 IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), pages 10213-10222, June 2022. doi: 10.1 N0.1009/CVR522888.22 2ww.:P/5526682222.002208 00998.
[21] Abby Stylianou, Richard Souvenir, and Robert Pless. Visualizing Deep Similarity Networks, January 2019.
[22] Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, (1((2):6619-8622 2014. doi: 10.1073/pnas.144 10073/ppss..411121111 14011
[23] Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque VI responses to natural images. PLOS Computational Biology, 15(4):1-27, April 2019, doi: 10.1371/journal, pebi. 1006897.
[24] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. http://aaxivoongabs2200...2.. 20000///,,.000000000.3,, February 2020.
[25] Kaiming He, X. Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[26] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Puttern Recognition, pages 248-255, 2009. doi: 1091100// 200.556888.
[27] Hadi Salman. Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust ImageNet models transfer better?, 2020.
[28] Konstantin Willeke, Kelli Restivo, Katrin Franke, Ame Nix, Santiago A. Cadena, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Alexander S. Ecker, Fabian H. Sinz, and Andreas S. Tolias. Deep learning-driven characterization of single cell tuning in primate visual area V4 unveils topological organization. bioRxiv the preprint server for biology, 2023. doi: 10.1 001/2023.05.12.540555
[29] Shahd Safarani, Ame Nix, Konstantin Willeke, Santiago A. Cadena, Kelli Restivo, George Denfield, Andreas S. Tolias, and Fabian H. Sinz. Towards robust vision by multi-task learning on monkey visual cortex, 2021.
[30] Jenelle Feather, Guillaume Leclere, Aleksander Mądry. and Josh H. McDermott. Model metamers illuminate divergences between biological and artificial neural networks. bioRxiv : the preprint server for biology, 2023. doi: 10. N0.1100/2022.05 19.49999999....
[31] Konstantin- Monntaantin-Klemeenn Klemens Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay K. Jagadish, Fric Wang. Edgar Y. Walker, Santiago A. Cadena, Taliah Muhammad, Erick Cobos, Andreas S. Tolias, Alexander S. Eoker, and Fabian H. Sinz. Generalization in data-driven models primary visual cortex. bioRxiv : the preprint server for biology. 2020. doi: 10.1100/2220.11 100.0020..0.0..3333.. 1226..3
[32] Anna Bognar, Albert Mukovskiy, Ghazal Ghamkhari Nejad, Nick Taubert, Michael Stettler, Rajani Raman, Martin Giese, and Rufin Vogels. Simultaneous recordings from posterior and anterior resooseeeo regions in the macaque Superior Temporal Sulcus. Journal of Vision, 23(9))5403, August 2023. ISSN 1534-7362, doi: 10. 20.1167/0v.23.9.540..

---

[33] Alina Kuznetsova, Hassan Rom. Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The Open Images Dataset V4: Unilied image classification, object detection, visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956 1981, July 2020. ISSN 0920-5691, 1573-1405. doi: 10.1007/s| 1263-122------ 020-0011-6.
[34] Timothy F. Brady, Talia Konkle, George A. Alvarez, and Aude Oliva. Visual long-term memory has massive storage capacity for object details. Proceedings of the National Academy of Sciences of the United States of America, 105(38): 4325-113229 September 2008. ISSN 1091-6490, doi: 10. 10.1073/pnas. 0803390105.
[35] Sophie Lebrecht, Moshe Bar, Lisa Feldman Barrett, and Michael J. Tarr. Affective Valence in Everyday Objects. Frontiers in Psychology, 3:107, April 2012. ISSN 1074-1078. doi: 10.3389/fxxy..2012..0
Perceiving
[36] Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017.
[37] Ivo D. Popivanov, Jan Jastorff, Wim Vanduffel, and Rufin Vogels. Tolerance of Macaque Middle STS Body Patch Ncurons Shape-preserving Stimulus Transformations. Journal of Cognitive Neuroscience, 27()):1001-1066, May 2015. ISSN 0898-929X, doi: 10.1162fjoc__a00771
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alhan Desmaison, Andreas Köpf, Edward Yang. 7ach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Pigh-Performance Deep Learning Library, December 2019.
[39] Abubakar Abid, Martin Zhang, Vivek K Bagaria, and James Zou. Exploring patterns enriched in dataset with contrastive principal component analysis. Nature communications, 91):21134, 2018.
[40] A. Bognár, R. Raman, N. Taubert, Y. Zafirova, B. Li, M. Giese, B. De Gelder, and R. Vogels, The contribu- tion of dynamics to macaque body and face patch responses. Neurolmage, 269:119907 2023. ISSN 1053- 8119. doi: http://doi.org//// http:///00.001/.oooooooaeeeee 2023.119907. URL http:: /www. sciencedirecttt com/ http:/ccenne//arre /ttpaa/c//aatttie/a/a////111111/
12

---

A Appendix supplemental material
A.1 Additional Results
A.1.1 Synthetic data
To test method wider of semantic categories, generate six synthetic, category-sclectiv neurons. so, we gather small set within-categon category images Iin.l.... V Fu,, and set Fam,1 of from the stimulus sets used for the electrophysiological experiments. Feeding these through the CNN and sampling spatial location using Gaussian readout with random location preference yields activation matrices Am E R and Avui RMxx , synthetic neuron with readout weights RR, that on average prefers the within-category images can then easily he found by solving
images
arg max (A AAn (A A M arg max w (ALA Auur-
This formulation useful since the Rayleigh quotient is solved by setting be the first eigenvalue of Ale Ain - A Avut 39 course, Resnet also contains category-sclectiv neurons that not need to be constructed artificially. However, we aim to make the experiment as similar to neural recordings as possible, where model neurons are usually given linear readout of latent activations.
Airplanes
Lamps
Chairs
Plants
Bodies
Pots
Figure 6: Results of applying the parallel backpropagation method to six synthetic, category- neurons. Each row corresponds to neuron, with the preferred category indicated above.
We observe that the visualizations clearly mark features that common among within-class images, showing why the ooc. stimuli drive the otherwise calegory-selective neurons.
A.1.2 Integrated Gradients
Our visualization procedure based on weighted sum tensor containing saliency maps latent features. As discussed in the main text, the method is agnostic towards how the individual saliency maps are generated. For the main experiments, we use the vanilla gradient method which results in reweighting the Jacobian of the latent features. Here, experiment with using different saliency backbone, namely Integrated Gradients [8]. For an image x, Integrated Gradients approximates the path integral of the gradients along the straightline path from image zeros Originally developed visualizing scalar outputs, we adapt formula from [18] to the multi-dimensional case to yield
IntegratedGrads(z) x
where (x) denotes Jacobian CNN features w.r.t. æ. Since 119 found that this formulation heavily influenced by the clement-wise multiplication with the input v. we omit this step in the computation. Results shown i demonstrate that the visualizations almost indistinguishable from those computed using vanilla backpropagation.
A.2 Additional details for neural data collection
Experiment details. We recorded neuronal responses during passive fixation task using 2x2 degree fixation window (EyeLink 1000 infrared eye tracker sampling 1000 Hz). Stimuli were shown camma-correcttdd
13

---

Integrated Gradient
Standard Gradient
Figure 7: Results for computing the Jacobian using the Integrated Gradient method (top row). Bottom row shows the results for the standard gradient computation from the main text for reference.
22.5-inch ViewPixx monitor at a distance of about 57 cm, with a resolution 1920x 1080 and a refresh rate of 120 Hz. A sct of 475 stimuli of a monkey avatar in various poses were presented 8 times in pscudorandom gray background, with 250 ms interstinulus interval. During the interstimulus
sequence for 200 interval, only fixation dot was present and monkeys could receive brief juice reward. Stimulus onset and offset were indicated by a photodiode, detecting luminance changes synced with the stimuli, the display corner (invisible for the monkeys). Control of stimulus presentation, event timing, and juice delivery was managed by in-house Digital Signal Processing- baasessing-baeed computer system, which also monitored the photodiode signal and tracked eye positions. Neuronal data collected from and surrounding patches in ventral superior temporal sulcus (STS) using 16-site linear electrodes (Plexon probe) with Open Ephys acquisition board and software (sampling rate: 30000 Hz, filtered between 500-5000 Hz). Multi-uni activitics were extracted using Plexon Offline Sorter, after applying high-pass Butterworth filter with a cutoff frequency of 250 Hz. From the stimulus event synchronized continuous data, 550 ms trials were extracted, comprising 200 prestimulus and 150 poststimulus periods. Responsive (at least for stimulus showing stronger than spikes/s net responses (baseline: -75 to +25 ms. response: +50 to +250 ms)), and body-selective MUAs (p<0.01 Kruskal- Wallis test), with a split-half reliability >0.5 (Spearman Brown corrected) were selected. The responses of these MUA sites were used to predict neural responses to our set of object and body images. and for cach neuron selected the highest and lowest predicted activator, well the object most similar the top-activating avatar stimuli, according to s(,, -). Finally, in a second experimental phase recorded responses to these object and body images as well as subset the original monkey avatar stimuli, to test recording stability (same experimental design as in the first phase). For all neurons considered in this work, we tested for body-sslecttiityy by comparing the median response to bodies to the median response objects using Mann-Whitme U-test, considering only channels for which the detected significant difference.
IMRI-detinee seleetvee
Animals and husbandry.
Two male years old rhesus monkeys (Macaca mulatta), weighing 9.2 and 11 study. The animals were housed in enclosures at the KU Leuven Medical
kg. respectively, contributed School and experienced a natural day-night cycle. Each monkey shared its enclosure with at least one other cage companion. On weekdays, dry food was provided ad libitum, and the monkeys obtained water, or other fluids, during experiments until they were satiated. During weekends, the animals received water along with mixture of fruits and vegetables. The animals had continuous scanning, we implanted a custom-Ne made plastic recording chamber, allowing dorsal approach to temporal body patches. In each animal, the location of the recording chamber was guided by the IMRI body localizer described in [40]. Surgery was performed using standard aseptic procedures and under full anesthesia.
toys and other forms of enrichment. After fMRT
14

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim to present a method for shared-feature visualization and then apply it to novel recordings from macaque IT. The former is presented in the methods section, the latter in the results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other setrings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of the paper in a dedicated subsection at the end of the manuscript.
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciflcciiii.nnnn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might he violated practice what the implications would be.
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
Justification:
The derivation of the visualization method is is based on simple, self- contained mathe- matical arguments. Necessary conditions for the upper bound of the saliency map clearly stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and reffeeeefere..ed.
15

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all major experimental details i This goes for both neurophysiological as well as in-silico experiments. Fine details are given in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how construct
model (e.g.. large language model), then there should either be
dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (e.g.. to registered users), hut it should be possible for other rescarchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification:
Code and data necessary to reproduce all results given in the paper are included in the submission.
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines for more details.
http:///iipsccccpplllpp
guides/Codee Sttleess//CoeSuuuiiiiiiooooooo
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://nipa.cc/ppun
Buidee//DoeeSubmiissoooPPPPo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The main paper contains all hyperparameters necessary to qualitatively reproduce the results. Finer details found in the attached code base.
Guidelines:
The answer NA that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix. supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
This question applicable model performance in terms correlations on held-out data. We provide the appropriate confidence intervals in Fig. Computation these intervals, as well as significance for single-channel correlations is part of the attached code base. The code base also includes the tests for body-selectivity of recorded neurons.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: We state that all experiments
single Nvidia RIX 2080Ti.
17

---

Guidelines:
answer means the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory
storage.
The paper should provide amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htth::/hcppp////a.rp....pp/aaate httpp/////c/ttcccccccccaoealllla
Answer: [Yes]
Justification: After review of the code of ethics, we believe that work does not violate any of the statements. Additionally, provide information about animal experiments the appendix.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make
preserve anonymity (e.g., if there is special considcration duc laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts the work performed?
Answer: [NA]
Justification: At this point in time, we cannot identify any harmful impacts our work.
Guidelines:
The answer means that there is no societal impact of work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it For example, is legitimate point out that an improvement the quality generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how system learns from feodback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We believe that misuse of our work unlikely this point in time, and therefore not provide any safeguards.
Guidelines:
18

---

The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: To the best of out knowledge, we have cited all creators of assets that were used in this work. For the creator of some of the stimuli, we have incorporated statement in the acknowledgements per his request.
Guidelincs:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should provided.
this information not available online, the authors are encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We answered NA hecause we do not publish new model architecture data set. However, we will make the code to the visualization method public upon publication of the manuscript.
Guidelines:
The answer means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, rememher to anonymize your assets (if applicahle). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: We not use human participants this work. We provide statements regarding the use of non-human hom-n primates in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
19

---

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: We provide statements about the role of non-human primates in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained TRB approval, you should clearly state this the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
20


Reviews:
[{"review_id": "iqWtBXqJof", "reviewer": "Reviewer_yQac", "summary": "The authors proposed a deep learning based method to visualize shared features in neurons that are selective to specific categories, such as faces or bodies, when they respond to out-of-category stimuli. The method identifies visual features driving the selectivity of neurons by modeling responses to images based on latent activations of a deep neural network. The paper highlights the application of this method to body-selective regions in the macaque IT cortex, demonstrating that neurons encode overlapping visual features for bodies and objects. This approach provides insights into why certain non-body objects activate body-selective neurons and offers a more fine-grained understanding of neural responses.", "questions": "Can The authors clarify the novel contributions of the proposed method in comparison to existing visualization techniques? How does the approach provide unique insights that are not achievable with current methods?\n\nHow generalizable is the method to other types of neurons or different brain regions beyond body-selective areas in the macaque IT cortex? Can the proposed approach be adapted to study other semantic categories or species?\n\nWhat are the broader implications of these findings for the field of visual neuroscience? We knew from previous studies that out-of-category stimuli can also activate neurons coding for in-category features.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-structured, with a clear abstract, introduction, methodology, and results sections. \n\nWhile the proposed method is based on well-established deep learning techniques, specifically leveraging latent activations of a deep neural network to model neuron responses, the application to novel recordings from body-selective regions in macaque IT cortex demonstrates its practical utility and empirical soundness.\n\nI think the primary contribution lies in providing a tool that allows for a more fine-grained investigation of neuron responses in visual neuroscience. By revealing why certain non-body objects activate body-selective neurons, the paper contributes valuable insights to the understanding of neural selectivity and visual processing.", "weaknesses": "The paper primarily applies existing deep learning techniques rather than introducing new machine learning algorithms or models. This might be seen as a limitation for those expecting significant advancements in machine learning methods, especially readers from NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for review and comments. We hope to be able to increase your confidence in recommending the manuscript for publication.\n\n> The paper primarily applies existing deep learning techniques rather than introducing new machine learning algorithms or models. This might be seen as a limitation for those expecting significant advancements in machine learning methods, especially readers from NeurIPS.\n\nIt is correct that we do not introduce new models or learning algorithms. Our contribution is more in line with efforts towards explainability, as we propose a novel method to study the behaviour of a trained model with fixed weights. We do believe that explainability methods can be of interest to a relatively broad community of people working on vision. Further, we argue that applications to neuroscience are of special importance to the community at NeurIPS, in order to foster a relationship between the two related fields.\n\n> Can The authors clarify the novel contributions of the proposed method in comparison to existing visualization techniques?\n\nOur method can be understood as an approach to turn any (single-image) attribution method into a (multi-image) similarity-attribution method by reweighting feature-wise attribution maps. In doing so, it is completely agnostic to the underlying visualization technique used to compute these feature-wise attributions. Previous work on similarity attribution methods (see related work) focuses on output layers to visualize global image similarity, while we are interested in local features that drive responses of neurons in the brain. Therefore, our approach is applicable to (functions of) hidden units.\n\n> How does the approach provide unique insights that are not achievable with current methods?\n\nWe are interested in why an out-of-category (ooc.) stimulus $x_{\\\\text{out}}$ activates an otherwise category-selective neuron.\nOne could use an existing visualization technique to study the neuron's preference of $x_{\\\\text{out}}$. This would provide an attribution map over the image, highlighting the visual features driving the model neuron's response. However, it does not yield insight into *why* the image would activate a category-selective neuron, specifically. We argue here, in line with previous work, that neural preference of these ooc. images is due to shared features with the preferred category. Therefore, a proper visual explanation should also include how the driving features in $x_{\\\\text{out}}$ occur among images of the preferred class, to understand why preference of these features has developed. The proposed method therefore explains neural responses to an ooc. image by providing a strongly driving category image, highlighting the preferred features in the category image, and then highlighting the corresponding features in the ooc. stimulus.\n\n> How generalizable is the method to other types of neurons or different brain regions beyond body-selective areas in the macaque IT cortex? Can the proposed approach be adapted to study other semantic categories or species?\n\nYes, the method is completely category-agnostic and therefore can be used for any semantically selective brain region, as well as selective units in artificial networks. Future work could apply the method to, e.g., face cells as another category in more depth; as a preliminary example, consider Fig. 4 panel (4,1) where we visualize an object driving a face preferring-neuron. For details regarding category agnosticism, we kindly refer to section 3 of the manuscript.\n\n> What are the broader implications of these findings for the field of visual neuroscience? We knew from previous studies that out-of-category stimuli can also activate neurons coding for in-category features.\n\nRegarding broad implications, we first provide additional evidence for the hypothesis of category-agnostic tuning properties in macaque IT cortex. There is still conflicting evidence regarding this question [1], necessitating further work.\n\nFurthermore, given the finding that out-of-category stimuli can activate category-selective neurons, the interesting question becomes why this is the case. Our work supports the notion that such stimuli exhibit features which are also common in images of instances from the preferred category. This is, to the best of our knowledge, the first attempt at visually characterizing these features. Our findings suggest, that the objects activating body-selective neurons have local parts that are visually similar to local body parts. We hypothesize that this finding transfers to other category-selective areas. For example, [2] found that simple shape descriptors like 'roundness' are not enough to explain face cells' responses to objects. In our experiments, we also found cells that fire in response to the head; their best-driving objects are not round, but our method finds parts which resemble a head (Fig. 4 (4,1), (4, 2), (5,3)). This yields insight into *why* these objects might activate the neurons, beyond descriptors like 'roundness'\n\n[1] Shi Y, Bi D, Hesse JK, Lanfranchi FF, Chen S, Tsao DY. Rapid, concerted switching of the neural code in inferotemporal cortex. bioRxiv [Preprint].\n\n[2] Kasper Vinken et al. ,The neural code for “face cells” is not face-specific.Sci. Adv.9", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0aycfOfI3b", "reviewer": "Reviewer_7NES", "summary": "The paper shows that a hypothesis for the brain (category-selective neurons are actually selective to generic lower-level features that are present in those categories, not necessarily features specific to those categories) can be reproduced and visualized in a ResNet trained to predict neural activity. If I understand correctly, it is capable of generating hypothetical pairs of images with similar features (as in A.1.1). One could then test if an actual neuron responds similarly to the pair as the model predicts. Or find a real neuron, record, then based on responses, use the model to get a stream of stimuli that should elicit similar responses (because of similar low-level features) and show it to the real neuron to see if it responds as we expect.", "questions": "see above", "limitations": "see above", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a clear and straightforward method that contributes to tools available for analyzing what drives individual units in regression-to-neural-data deep learning models. \n\nComprehensive and clear background/related work; very clear writing overall and description of methods on high and low levels. I got an intuition pretty quickly for what the method was even before the equations.", "weaknesses": "The results are interesting and intriguing to look at (Figure 4), however at the end of the day it comes across as a bit anecdotal as opposed to giving us insight into general principles of shape or object representation.  This is not a strong criticism though - I accept it is a good starting point, and still a worthwhile contribution.\n\nAlso, I note that all the objects are presented on blank backgrounds.  I wonder how things would change when the objects or bodies are presented in their natural context within a visual scene?\n\nother comments:\n111: Maybe this is standard terminology I'm not familiar with, but \"readout vector\" confused me at first because it makes it sound like it's logits or something? But it's just the weights from the latent representation to a single output unit. Line 131 more clearly calls it \"learned weight vector\"\n145: should be a_2=f(x_2)?\n\nIt would be interesting to see more mathematical/geometric analysis of the shared features (beyond stubby vs spiky) in the future, e.g. joints at a specific angle, combinations of contours in a certain way, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing the manuscript, and for your positive feedback.\n\n> The results are interesting and intriguing to look at (Figure 4), however at the end of the day it comes across as a bit anecdotal as opposed to giving us insight into general principles of shape or object representation. This is not a strong criticism though - I accept it is a good starting point, and still a worthwhile contribution.\n\nWe understand this criticism, as we steered the focus of the manuscript towards introducing the method in a clear and class-agnostic fashion to allow others to easily apply it to their own recordings of category-selective neurons. We acknowledge that further work should attempt to get a more systematic overview of the shared coding properties.\n\n> Also, I note that all the objects are presented on blank backgrounds. I wonder how things would change when the objects or bodies are presented in their natural context within a visual scene?\n\nWe decided for neutral backgrounds to allow for good control of low-level image features.\nFurther, we wanted the model to pick up on as few spurious image statistics as possible to enable it to generalize across categories. Since it seems like similarity between highly activating bodies and objects is primarily driven by local features / parts, we would now hypothesize that the tuning properties would be be preserved when presenting stimuli as parts of naturalistic scenes. We agree that it would be worthwhile to test this hypothesis.\n\n>  Maybe this is standard terminology I'm not familiar with, but \"readout vector\" confused me at first because it makes it sound like it's logits or something? But it's just the weights from the latent representation to a single output unit. Line 131 more clearly calls it \"learned weight vector\" 145: should be a_2=f(x_2)?\n\nThank you for pointing out the unclear language - we have adjusted these parts in the text.\n\n> It would be interesting to see more mathematical/geometric analysis of the shared features (beyond stubby vs spiky) in the future, e.g. joints at a specific angle, combinations of contours in a certain way, etc.\n\nWe agree that it would be of high interest to find quantitative feature descriptors to go along with qualitative visualizations. We are concurrently working on further ways of characterizing the features driving these neurons.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "16MPTFxEnF", "reviewer": "Reviewer_5Zdi", "summary": "A deep-learning-based approach is proposed in the paper \"Parallel Backpropagation for Shared-Feature Visualization\" to visualize shared visual features in high-level visual brain regions, which are typically thought to respond selectively to particular categories like bodies or faces. Despite this selectivity, these neurons occasionally respond to stimuli outside of their category, possibly due to shared visual characteristics. By backpropagating activations to the pixel level and enhancing shared features while attenuating non-shared ones, the authors present a strategy for highlighting these shared features using a deep neural network to model neural responses and identify relevant features.", "questions": "Subsequent research ought to concentrate on enhancing the underlying models and attribution methods as well as verifying the approach in various scenarios such as with different categories and in a closed-loop manner. The authors already acknowledge these directions.", "limitations": "The authors have a limitation section that addresses their work's assumptions.", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The text is very well structured, and the questions and methodology are written in a very clear and detailed manner. This work follows an innovative approach which presents a novel and exciting way to use DL in order to gain insights of single neuron selectivity across the visual hierarchy. The method's steps—which include parallel backpropagation, determining neuron-specific picture similarity, and training a linear readout on top of a CNN—are fully explained. Finally, this work includes an original experimental component that differentiates from a lot of other applications of DL on already existing datasets.\nA strategy for visualizing shared elements that drive brain reactions to out-of-category stimuli is presented in this research, which shows promise. The method has the potential to further our understanding of how the brain processes visual information, even though there is still room for improvement.", "weaknesses": "In this work, the method is heavily dependent on the deep neural network used, and how effectively it can fit the neural data. Fortunately, the cases where the tunning properties are not captured by the model can be identified as mentioned in the end of the results section. The ability to generalize to other categories such as face patches was not tested and remains to be tested. \nThe technique is predicated on the idea that within-category and out-of-category photos share a significant amount of attributes. It is not clear what we expect for the model to do and how useful outcomes are going to be in situations where there is little overlap.\nFinally, while the single neuron tuning specificity is the main topic of this work, it remains to be seen how a similar approach but with population representations compares.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments. We highly appreciate your confidence in our work. We agree that the mentioned limitations are largely adressed in the paper, but for the sake of completeness we post some remarks here.\n> The ability to generalize to other categories such as face patches was not tested and remains to be tested.\n\nWe agree that this is a highly interesting next step and are planning to apply this method to study tuning properties of both body and face patches in more depth in future work.\n> The technique is predicated on the idea that within-category and out-of-category photos share a significant amount of attributes. It is not clear what we expect for the model to do and how useful outcomes are going to be in situations where there is little overlap.\n\nIf a neuron is driven by a feature set $A$ for category-stimuli and a disjoint feature set $B$ for out-of-category stimuli, we agree that the proposed method would be unable to generate meaningful explanations, since the tuning for features $B$ simply cannot be explained (solely) by studying the tuning for features $A$. However, in that case the CNN activation vectors $a_{\\\\text{in}}$ and $a_{\\\\text{out}}$ should have distinct activation patterns, leading to a low image similarity $s(\\\\cdot,\\\\cdot)$ (equation (2)) which then implies an attribution map with low intensity (equation (7)). This allows the user to detect such cases. Furthermore, while we certainly don't claim that such neurons don't exist, the majority of neurons we recorded coded for similar features across categories (subsection 5.1).\n> Finally, while the single neuron tuning specificity is the main topic of this work, it remains to be seen how a similar approach but with population representations compares.\n\nThis would indeed be an interesting comparison. At the population level, we hypothesize that most if not all features of the within-category image activate the population due to variations in tuning properties of single neurons. For an out-of-category stimulus, the method should then highlight the subset of features present in that image as since only those features will be weighted according to equation (5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ghg86UCJqB", "reviewer": "Reviewer_aP8u", "summary": "This paper presents an innovative method for explaining why an IT neuron that is supposed to tune to a particular object class or category would respond to outside-of-category stimuli. The proposed method uses parallel backpropagation to highlight the features in the out-of-category stimuli that are shared by the stimuli in the tuned class. The method helps discover some shared features between highly activating bodies and objects in IT neurons but also fails in other cases.", "questions": "Perhaps the usefulness of the method is more general than the authors envisioned.  Can the method be applied to neurons in V4 and V1?  Can it be used to analyze artificial deep networks to understand \"neural codes\" in CNN?\n\nAlso, the assumption of the approach is that neurons are tuned to one single category, but could neurons be tuned to multiple categories or multiple features?", "limitations": "There are no concerns about the limitation issue.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method is original, innovative, sound, and very clearly presented. It can be useful for IT neurophysiological research to resolve some mysteries in the neural codes in IT.", "weaknesses": "The paper focuses on the presentation of the method. The presentation of the scientific results is not very systematic. They are shown as anecdotal illustrations of some success and failure cases. Hence, the scientific insights provided are a bit preliminary, and the significance of the results is perhaps limited. The contribution is solid but perhaps a bit thin.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments and the encouraging feedback. Regardings questions/criticisms:\n> The presentation of the scientific results is not very systematic [... and] a bit preliminary.\n\nWe agree that the results on feature selectivity in macaque body patches can be expanded on to get a more complete picture of their tuning properties for both bodies and non-body objects. As you stated, we put the focus of the paper more towards a rigorous introduction of the visualization method. While this approach has its drawbacks, we argue that it also has advantages.\nIdeally, we would like to see the proposed method applied to not only body patches, but other category-selective regions as well to shed light on the general concept of category-selective processing in the brain. To make that feasible, we aimed to introduce the procedure in a detailed fashion to facilitate easy use by other researchers. Further, focusing on the class-agnostic parts of the paper may make it interesting for the broader audience at NeurIPS. We would also like to point out that this is, to the best of our knowledge, the first work on visualizing the features underlying category selectivity in IT cortex.\nThat being said, we agree that a more systematic approach would be of interest and future work will focus on that.\n> Can the method be applied to neurons in V4 and V1?\n\nThis is an intriguing question, since semantic-category selectivity is mostly studied in IT. One interesting approach could be to replace the semantic selectivity by selectivity for a specific shape/texture descriptor. As a naive example, a neuron that responds to semicircles could be classified as a (seemingly) circle-selective neuron, as the semicircle feature is more common in the image distribution of circle images than in the global distribution. Applying the proposed method should then highlight that the neuron responds to semicircles because they are a subfeature of the more general shape descriptor. While this example is somewhat obvious, application of our method might be of use when considering the complex feature tunings in V4.\n> Can it be used to analyze artificial deep networks?\n\nWe would argue that the method can indeed be used for studying neural codes in CNNs. One interesting case is an application to a classification model's readout layer.  Here, category-selectivity is vitally important since any response to out-of-category images is interpreted as evidence for an incorrect hypothesis. Consider an image $x$ which is incorrectly classified as category $c$.  One could  then use a pool of images from category $c$, find the most similar image to $x$ according to $s(\\\\cdot,\\\\cdot)$ and visualize the shared features. This could yield insight into why $x$ is misclassified, and why the model uses those specific features as evidence for class $c$. One could also study category selectivity in hidden layers of vision models. We provide a proof-of-concept for this case in the appendix (A.1.1), where we compute visualizations for artificially created category-selective units.\n\n> Could neurons be tuned to multiple categories?\n\nThe question of a neuron being tuned to multiple categories is an interesting one, as definitions of a category can be quite fuzzy (consider, for example the case of face neurons and body neurons ). However, let’s assume that a neuron is tuned to two semantically rather different categories. In that case, our work, together with previous papers, would suggest that it is because there are shared features that are common in the image distributions of both categories. In that case, the proposed method could still be used to visualize these shared features. One could do this by sampling images from the two categories and feeding pairs of them through the visualization pipeline.\n> Could neurons be tuned to multiple features?\n\nWe identify two avenues in which a neuron's possible tuning for multiple different features relates to the proposed method. The first is the possibility that a neuron likes different features across categories, i.e. feature A for category 1 and feature B for category 2, where feature A and B are the results of different computational processes. \n\nWe employ two approaches to ensure that the method does not highlight these as shared features: First, we train the neurons' readout model only on within-category images and then test on outside-of-category images. Good performance on outside-of-category images implies that the features driving the responses are (at least partially) shared between the two categories. Second, the visualization method is able to detect cases in which features are not shared between two images. Considering the similarity metric in equation (2), the activation vectors $a_1$ and $a_2$ will be dominated by different dimensions if the features between the two images are not shared, yielding a small dot product and thus a small similarity $s$. Since the norm of the attribution map is bounded by the image similarity $s$  (equation (7)), the attribution map will have low intensity, reflecting that the two images drive the neuron due to different features.\n\nThe second relevant avenue is the situation in which a neuron is activated by multiple within-class features (e.g. arms and torsos for bodies). If the ooc. stimulus to be visualized contains only one of the features, this should be reflected by the method. The weighting for feature $i$  in the attribution map, as given in equation (5), is $a_{\\\\text{in}}^{(i)} a_{\\\\text{out}}^{(i)}(w^{(i)})^2$, yielding that feature $i$ will be ignored if it is absent in either of the images, even if the neuron is tuned to it. Conversely, if the neuron is tuned to multiple features and these features are apparent in both images, they will all be highlighted.\nTherefore, we argue that the proposed method is able to deal with cases in which a neuron is tuned to a variety of different features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their time and their valuable insights. We were glad to see that all reviewers found the presentation of the work clear and understandable, which is also reflected by the fact that all summaries clearly capture the main points of the paper. We also did not identify any factual errors in the reviews. Further, we appreciate that all reviewers suggest that the manuscript should be accepted for publication, albeit with varying confidence. We hope that we were now able to clarify remaining questions, and properly addressed the raised issues. Even though there was partial overlap between comments, we answered all of them individually to make the rebuttals for each reviewer self-contained. If some of our answers were not to the point and you would like to discuss further, we are happy to do so over the next couple of weeks.

Meta-review: This paper describes a novel approach for identifying visual features that are common between a preferred object class vs. other object classes with an application to characterizing the neural selectivity of visual neurons. 

The reviewers noted that while the deep learning techniques are relatively well established, their application to visual neuroscience remains relatively novel and compelling. On the negative side, reviewers commented that the reported results and analyses remain limited and appear to be relatively preliminary. 

All in all, there appears to be strong support from the reviewers to accept this paper, which introduces a novel, broadly applicable method for better understanding neural selectivity in visual areas. Hence, the AC recommends that the paper be accepted.
Decision: Accept (spotlight)