Neuron-based explanations of neural networks sacrifice completeness and interpretability
Authors: Nolan Simran Dey
Eric Taylor
Alexander Wong
Bryan P. Tripp
Graham W. Taylor
Venue: Accepted by TMLR ()

Abstract: High quality explanations of neural networks (NNs) should exhibit two key properties. Completeness ensures that they accurately reflect a network’s function and interpretability makes them understandable to humans. Many existing methods provide explanations of individual neurons within a network. In this work we provide evidence that for AlexNet pretrained on ImageNet, neuron-based explanation methods sacrifice both completeness and interpretability compared to activation principal components. Neurons are a poor basis for AlexNet embeddings because they don’t account for the distributed nature of these representations. By examining two quantitative measures of completeness and conducting a user study to measure interpretability, we show the most important principal components provide more complete and interpretable explanations than the most important neurons. Much of the activation variance may be explained by examining relatively few high-variance PCs, as opposed to studying every neuron. These principal components also strongly affect network function, and are significantly more interpretable than neurons. Our findings suggest that explanation methods for networks like AlexNet should avoid using neurons as a basis for embeddings and instead choose a basis, such as principal components, which accounts for the high dimensional and distributed nature of a network's internal representations. Interactive demo and code available at https://ndey96.github.io/neuron-explanations-sacrifice.

Published in Transactions on Machine Learning Research (03/2025)
Neuron-based explanations of neural networks sacrifice com- pleteness and interpretability
Nolan Dey
nolan @cerebros.net
Cerebras Systems, University of Waterloo, Vector Institute
Eric Taylor
eric.legloor@kvccttinnnsttttttte
Borealis AI, Vector Institute
Alexander Wong
University of Waterloo, Apple
Bryan Tripp
bptripp@rwaaerrooo.c
University of Waterloo
Graham W. Taylor
gutayylornuogullph..o
University of Guelph, Vector Institute
Reviewed on OpenReview: https: // openreviee. net/ forum? id= UWNa9Pu6q4
Abstract
High quality explanations of neural networks (NNs) should exhibit two key properties. Com- pleteness ensures that they accurately reflect a network's function and interpretability makes them understandable humans. Many existing methods provide explanations of individual neurons within network. In this work we provide evidence that for AlexNet pretrained ImageNet, neuron-based explanation methods sacrifice both completeness and interpretabil- ity compared to activation principal components. Neurons are poor basis for AlexNet embeddings because they don't account for the distributed nature of these representations. By examining two quantitative measures of completeness and conducting a user study to measure interpretability, we show the most important principal components provide more complete and interpretable explanations than the most important neurons. Much of the acti- vation variance may be explained by examining relatively few high-variance PCs, opposed to studying every neuron. These principal components also strongly affect network function, and are significantly more interpretable than neurons. Our findings suggest that explanation methods for networks like AlexNet should avoid using neurons as basis for embeddings and instcad choose a basis, such as principal components, which accounts for the high di- mensional and distributed nature of a network's internal representations. Interactive demo and code available lhttps://ndey:
Introduction
Large neural networks (NNs) are increasingly consequential. For example, widespread use of large language models brings a wide range of societal risks (Weidinger et al.) 2022 Shevlane et al.) (023). and NNs are increasingly used in high-stakes application areas such as healthcare (Rajpurkar et al., 2022), inchuding in many FDA-approved devices []] Despite their increasing importance, we still do not have satisfactory methods to explain how NNs arrive their outputs. Many sensitive applications would benefit from reliable
https://www. fda. htgp://eeeeeaataaaaoeoaa.eeetee.
learning-se ainl amabled-nedican

---

Published in Transactions on Machine Learning Research (03/2025)
explanations of neural networks. Developing improved methods of explaining NNs can also advance understanding of neural representations and lead to improved training techniques.
At the heart of the explainability challenge is the tradeoff between explanation completeness and explanation interpretability (Gilpin et al. 2018). , Completeness describes how accurately and thoroughly an explanation conveys a model's s function while interpretability describes how easy it for a human to understand an explanation. The most complete explanation would be to simply display the equation for a layer's forward pass. However, this explanation has poor interpretability. At the opposite extreme, many popular DNN explanation methods make choices that increase interpretability at the expense of completeness. For example, saliency map and feature visualization explanations (Simonyan et al. 2013 Olah ct al. 2017) are highly interpretable but offer unreliable explanations (Knnderaans et. al. 2017 Geirhos et al. 2023).
The problem of understanding a NN as a whole can be decomposed into understanding the non-linear transformation applied by each layer. Each NN layer applics a non-lincar transformation to input activation space to produce an output activation space. Understanding any such transformation means understanding how the output space is produced relative to input space. Unfortunately, humans cannot directly understand the meaning of point in the internal representations used by the network. Instead humans are capable of interpreting a point in a network's 's imput space (e.g. an image, sentence, or row of tabular data). This motivates the use of visualization methods that map hidden activations back to a network's input space. Some popular visualization methods are feature visualization (Olah et al. 2017), nearest neighbors search (Karpathy] 2014), and more complex generative modeling approaches (Rombach et al., 2020; Nguyen et al, 2016b). Using an activation visualization technique further narrows the problem of explaining the /th layer's transformation into understanding how the 7th layer's transformation maps back to the network's input space.
To solve this problem we need to select points to visualize and interpret that are representative of activation space. A practical way forward is to choose a basis and visualize points along each basis vector separately. However, the problem remains difficult because:
The curse of dimensionality - hidden activations commonly have thousands of dimensions.
Like biological ones, artificial neural networks have distributed representations: multiple neurons fire
together to represent concepts (Barrett et al. 2019), (Fong & Vedaldi 2018), (Gallego et al. 2017).
(Härkönen al. 2020), (Leavitt & Morcos 2020). Olah et al. 2018). (Rombach et al. 2020).
Neurons can be poly-semantic- a single neuron can represent multiple concepts (Olah et al. 2020b
Human attention is a limited resource- - the number visualizations that can be interpreted is limited.
Basis selection must take each of these difficulties into account to support relatively complete and inter- pretable explanations.
The most popular choice of basis is the standard basis which corresponds to focusing on understanding cach individual neuron in isolation. The neuron basis disregards the distributed nature of NN representations, detracting from explanation completeness. Furthermore, studying each neuron in layer requires an imprac- tical amount of human attention. Explanation methods that focus on individual neurons (e.g. Bau et al.
2017 2020; Mahendran & Vedaldi 2015 Mu & Andreas 2020 Nguyen et al. 2016a by Olah et al. 2017 2018
Bills et al. 2023 Schwettmann et al. 2023 Dai et al. 2022 typically only provide interpretable explanations
for ncurons that are highly selective of some concept. However, only a small fraction of individual neurons are highly concept-selective (Bau et al. 2017], (Olah et al., 2017]. Furthermore, highly concept-selective neurons can be ablated without removing model's ability to recognize that concept, meaning neurons that are not highly concept-sclective still play an important role in network function (Barrett et al. 2019). (Don-
nelly & Roegiest 2019), (Kanda et al. 2020), (Leavitt & Morcos 2021), [Morcos et al. 2018]. Finally due to the polysemantic nature of neurons, one would need to understand each concept that a neuron is involved in representing, requiring an even larger expenditure of human attention.
Instead, a powerful alternative is to study a basis obtained through a dimensionality reduction technique. This can be done by sampling hidden activations produced for each example in large-scale dataset and
2

---

Published in Transactions on Machine Learning Research (03/2025)
fitting dimensionality reduction model to obtain the basis (e.g. Alammar 2020 Karpathy 2014 Carter
et al. 2019). This general approach accounts for the distributed nature of NN representations, provides a greatly reduced set of basis vectors to analyze, and can take into account the polysemantic nature of neurons.
In this study we provide evidence illustrating the inferiority of the neuron basis compared to the principal component analysis (PCA) basis, a simple baseline. Our method can be applied to leed-forward networks as well as CNNs and we expect that it can be adapted to other architectures including Vision Transformers Dosovitskiy et al. (0021). While methods are widely applicable,
focus this paper on studying AlexNet
[Krizhevsky] 2014) pretrained on ImageNet (Deng et al.] 2009 Paszke et al., 2019) because it is studied in several related explainability works (Bau et al. 2017), (Fong & Vedaldi, 2018), (Mu & Andreas 2020), (Zhou (et al. 2018), (Rajpal et al. 2023) and its limited depth makes it feasible for us to study each layer in detail
in a user study.
To compare the explanations produced using the neuron and PCA bases, we propose two quantitative mea- sures of completeness and conduct a user study to measure explanation interprelability. Firstly, we show that small numbers of PCs account for much of the activation variance in each layer (Section 4.1.1 Also, through ablation experiments we show that PCs with large eigenvalues are generally more important for Alex (Krizevskyy 2014) performance than the most important neurons, and that PCs with small eigenvalues are generally less important for AlexNet's performance than the least important neurons (Section 4.1.2). Through a user study we also show that PCs with large eigenvalues tend to be interpretable and PCs with small eigenvalues are generally uninterpretable. Moreover, the most important PCs are more interpretable than the most important neurons (Section 4.2). In summary, our results show the most important PCs form more complete and interpretable basis than the most important neurons.
2 Related work
Several prior works have also examined the validity of neuron-based explanations. Morcos et al. (2018) perform cumulative neuron ablations that show highly class-selective units may be harmful to network performance. Zhhou et al. (2018) follow this up by conducting neuron ablations to show that ablating highly class-selective individual units hurts the accuracy of specific classes more than overall accuracy, supporting the view that it is mcaningful to study these neurons. Leavitt & Morcos (2021) continued this line of inquiry by showing that regularizing against learning class selective neurons can improve accuracy. Radford ct al. (017)) demonstrate the existence of "sentiment neuron" which is highly selective of text sentiment. However, Donnelly & Roegiest (2019) find that removal of the "sentiment neuron" only marginally affects sentiment classification accuracy and may even improve it.
Morcos et al. (2018); Leavitt & Morcos 2021]: ; Donnelly & Roogiest (0099 Barrett et al. (0199) caution
that methods for understanding neural nelworks based on analyzing highly selective single units, or finding optimal inputs for single units (e.g. Olah et al. 2017) may be misleading.
Another promising line of inquiry is Concept Activation Vectors (CAVs) Kim et al. 2018 which are activation vectors that align with pre-defined human-interpretabb concepts. CAVs provide interpretable explanations because they take into account the distributed nature of representations. However, CAVs can lack completeness because a user must predefine concepts to explain (similar to Network Dissection (Bau et al. 2017). making it unlikely to encompass the full set of concepts represented by a NN. Ghorbani et al. (2019h introduce automated concept-based explanations (ACF) to automate the process of defining concepts through an activation clustering algorithm. Both TCAV and AutoTCAV find directions that align with concepts but do not take into account overall explanation completeness. To remedy this, Yeh et al. (2020) propose ConceptSHAP to measure completeness and find concepts to maximize this metric. Notably they find the completenoss of concepts found via PCA is comparable or better than ACE but their method still outperforms PCA completeness. They show PCA maximizes the f surrogate of their completeness score but PCs lack human interpretability. In contrast, we find PCs can be highly interpretable if they are fit. to activations prior to non-linearities (see Section 3.1
Both biological and artificial neural networks have distributed representations; multiple neurons fire together to represent concepts Barrett et al. 2019: Pong & Vedaldi 2018 Gallego et al. 2017 Härkönen et al. 2020
3

---

Published in Transactions on Machine Learning Research (03/2025)
Olah et al. 2018 Rombach et al. 2020 A basis vector must have the ability to take into account the firing of multiple neurons at once. Barrett et al. (2019) highlight that population level analysis is useful for INNSs and ANNs and suggest dimensionality reduction could be a way forward.
Leavitt & Morcos 2020
The study individual neuron responses an approach inherited from neuroscience, where stimulus tuning of individual neurons has been studied since the 1960s (Hubel & Wiesel, 1962; (Dayan & Abbott 2005);
and the reverse correlation method has been used determine stimuui that optimally drive sensory neurons since the 1970s (De Boer & De Jongh, 1978 Dayan & Abbott, 2005). A focus individual neurons unavoidable at first because multi-neuron recordings were technically difficult. However, the importance of understanding neural activity at the population level has been recognized for decades (Georgopoulos et al. 1986 Zemel et al. 1998 Schneidman et al. 2003). Meanwhile, advancing methodology has led to a doubling in the number neurons that can be simultaneously recorded about every seven years (Stevenson & Kording, 2011 driving a shift. away from singlo-neuron analyses to population analyses (Saxena Cunningham
2019 Ebitz & Hayden 2021 Urai et al. 2022]. particularly in the last fifteen years. Linear dimensionality
reduction is valuable for studying the activily of large populations. For example, this approach has shown that population activity in the primate motor cortex during reaching is largely explained by low-dimensional rotational dynamics (Shenoy et al., 2013). and that related dynamics are important in perceptual decision making in the prefrontal cortex (Aoi et al. 2020). Activity in mouse primary visual cortex is correlated so that the th principal component variance decays as 1/n (Stringer et al. 2019), or more quickly for smaller components (Pospisil & Pillow 2024).
3 Methodology
As described in Section 1 the problem of explaining a NN can be decomposed into understanding the nonlinear transformation applied by each layer terms of the NN's input space. To facilitate this for a particular layer, we sample activations, fit basis for activation space, and visualize points along each basis vector. This is summarized in Figure []] Our methodology is computationally inexpensive and simple to apply to any feed-forward convolutional layers.
1. Sample layer's activations
3. Visualize points along basis vectors
4. Interpret visualizations
2. Identify basis vectors
"This basis vector seems to represent a color transition from green to purple"
Neuron acciuuliooss
PC2
PCI
Rd
Imsge from ILSVRC2012
Nouron activations
Figure Overview of our methodology. We sample Rd activations from a layer (1), then identify basis vectors of the layer's activation space (e.g. neurons or PCs) (2). Finally we visualize points along each basis vector (3) and interpret the visualizations (4).
3.1 Sampling layer's activations
We performed PCA large representative sample of activations obtained by forward propagating every image in the ImageNet training set through DNN to specified layer. We used an activation matrix A E Baad where n is the number of training examples, and d is either the number of convolutional channels or the number of fully-connected neurons, depending on the layer type. For fully-connected layers, each row of the matrix consisted of the full 2d response the corresponding image, where the number of neurons. For convolutional layers, since the same operation is applied to each spatial position, we sampled single spatial position from each channel. Sampling activations from the same spatial position for every
4

---

Published in Transactions on Machine Learning Research (03/2025)
image could bias our sampling towards imput features typically found at that position. To avoid this, we chose random spatial positions for each image, avoiding boundary positions. We adopted this method of sampling activations from (Carter et al.) 2019). We also found sampling activations before the non-linearity allowed for much better PCA fits and interpretable visualizations.
3.2 Identifying basis vectors
Activation space is high dimensional, often with thousands of dimensions. The neuron basis, defined by axis-aligned unit vectors corresponding to each neuron, the naïve choice of basis for activation space. Using dimensionality reduction we can find bases that capture more information with fewer basis vectors. To apply dimensionality reduction to large scale A, we require an invertible method that has compu- tationally inexpensive fit, transform, and inverse transform operations. For the purpose of this study, we believe studying the PCA basis is sufficient to illustrate the shortcomings of the neuron basis. Alternative dimensionality reduction methods such as ICA (Hyvärinen & Oja, 2000), NMF (Cichocki & Phan 2009), LLE (Roweis & Saul 2000). kPCA (Schölkopf et al., 1997). VAE (Kingma & Welling, 2014), t-SNE (van der Maaten & Hinton 2008), and UMAP (MeInnes et al. 2018 require the number of components to be known beforehand, are too computationally expensive apply to A, have many sensitive hyperparameters, and/or
do not possess an analytical inverse.
Using scikit-learn (Pedregosa et al. 2011 apply PCA to A to obtain transformed activations A' E R"xp where p is the number of principal components (PCs). PCA finds orthogonal basis vectors for Rd activation space, ordered by explained variance (Var(A)) for the ith basis vector). If activations have some covariance structure, then more activation variance can be explained with relatively few PCs. Significant activation covariance also implies neuron interactions are responsible for much of the activation variance.
3.3 Visualizing points along basis vectors
To gain insight into what each basis vector represents, we visualize m points along each basis vector indepen- dently of any other basis vectors. By projecting Rd activations onto p basis vectors, we obtain a transformed activation space R.. We uniformly sample points between the observed minimum and maximum values along each basis vector to obtain an (m x)) matrix of sampled points in transformed activation space. Then, we apply an inverse transform to obtain an (m x d) matrix of sampled points in untransformed activation C for a comparison of sampling methods.
space. See Appendix
Karpathy (2014) visualized point activation space atarget E Rd by displaying the receptive-field sized image patch whose activations have with the lowest f2 distance to the point atarget- In this work we extend Karpathy's method by displaying the k nearest receptive-fiele sized image patches that have the lowest lz distance to atarget when forward propagated. We perform this search over the N receptive field-sized image patches in A using the Faiss library (Douze et al. 2024 It. is possible that adjacent points along a basis share the same nearest neighbor.
Figures 2 and 4 show examples of AlexNet PC visualizations, where left to right moves from negative to positive values along each PC. For early network layers where the size of the receptive field is small, the visualizations are quite interpretable (Figure 2p. For deeper layers with larger receptive fields, it becomes more difficult to interpret what features the visualizations have in common because there is more area for distractors to be present. To help increase the signal to noise, mask away the regions of the visualizations that do not strongly affect the proximity of the representation to the point a in representation space that we are visualizing (Figure 3). See Appendix B for more details on the masking procedure.
We also experimented with optimized synthetic "feature visualizations" following Olah et al. 2017). We found synthetic visualizations bocome unintcrpretable after the first layer, corroborating the findings of Borowski et al. (2020). Borowski et al. (2020) conducted a user study and found natural image explanations to be more interpretable than optimized feature visualizations (Olah et al. 2017).
5

---

Published in Transactions on Machine Learning Research (03/2025)
conv1
PC
PC 2
PC 3
PC 4
PC 5
N2
N
4
N5
Figure 2: Visualizations of points along the 5 highest variance PCs and 5 highest variance neurons for AlexNet's conv1 layer.
3.4 Interpreting visualizations
Finally we must invest the precious resource of human attention to interpret the resulting visualizations. When visualizing activation space points we expect to see semantically meaningful changes we move along a useful basis vector. For example, Figures 2 and 3 show visualizations from the first. layers of Alex Moving from negative regions along the PC to positive regions (left to right), the visualizations change in some meaningful way. In these layers we see the angle of edges may change from horizontal to vertical, the color may change from orange to blue, patterns may change from vertical/horizonta. to diagonal, etc. This help give an idea of what features the DNN has learned to extract from the input to make classification decisions. To quantify the interpretability of each dimension, we report the results of a systematic study with human participants (scc Sec. 4..2b. Additionally, Appendix G contains visualizations of the 5 highest variance PCs and neurons for each AlexNet layer. The PCs for conv1 capture features such as brightness, Gabor filter phase, color contrasts, and color center surrounds. The PCs for conv2 capture features such as texture frequency, line orientation, and color. These results are consistent with the findings of study on early convolutional layers 'Olah et al., (2220a). The PCs for conv3 conv5 capture more high level features such textures and objects. Finally, the PCs of fc1 fc3 to separate classes. It is encouraging that

---

Published in Transactions on Machine Learning Research (03/2025)
conv2
PC
PC 2
PC 3
4
PC 5
22
N
N4
Figure 3: Visualizations of points along the highest variance PCs and 5 highest variance neurons for AlexNet's conv2 layer.
our method aligns with the general understanding of feature hierarchies in CNNs. In contrast, the neuron visualizations from conv3 fc3 appear semantically meaningless (Appendix G
Since it is difficult to exhaustively present these visualizations in paper, we have also created an interactive demo (https://ndey66.
including:
PC and neuron visualizations for every layer of pre-trained AlexNet Krizhevsky
2014
PC visualizations for every residual block of pre-trained ResNet-18 (He et al. 2016
PC visualizations for every residual block of pre-trained ResNet-50
(e et al. 2016
PC visualizations for 10 layers of pre-trained ViT-B/16 (Dosovitskiy al. 2021
4
Experiments
Throughout this section focus on comparing the PCA basis to the neuron basis. This comparison is important because the neuron basis used in many influential explainable artificial intelligence (XAT) works

---

Published in Transactions on Machine Learning Research (03/2025)
fc1
PC
PC 2
PC 3
PC
5C
N
4
N5
Figure 4: Visualizations of points along the 5 highest variance PCs and 5 highest variance neurons for AlexNet's fc1 layer.
that attempt to study layer representations (Bau et al. 2017). (au et al. 2020)
(Mahendran & Vedaldi
2015), (Mu & Andreas
2020
OOlah et al.] 2017). (Olah 2019) AlexNet (Krizhevsky Deng et al. 2009) because it is studied in several related works (Bau et al.
(gguyen et al. 2016a), (gguyen et al. 2016b).
ct al. 2018). (Bills ct al.
20233
In this work we study PoTor'''s (Paszke al.
2014)
pretrained on IetgeNet
2017),
(Fong & Vedaldi 2018), (Mu & Andreas 2020), (Zhou al.) 2018) and its depth makes it feasible
to study each layer in more detail. AlexNet has five convolutional layers (conv1 conv5) followed by three fully-connected layers (fc1- fc3). To compare the neuron and PC bases, we use two quantitative measures of explanation completeness and conduct user study to compare explanation interpretability.
4.1 Quantifying explanation completeness
Explanation completeness an abstract concept that could be measured in a variety of ways. We use complementary measures of subspace completeness below.
8

---

Published in Transactions on Machine Learning Research (03/2025)
4.1.1 Activation covariance structure.
One measure of completeness is the fraction of activation variance explained by a set of basis vectors.
this conlext, explained variance refers to the amount of activation variance along a basis vector. In Figure we plot the cumulative explained variance ratio of top-k basis vectors which we define as:
Cumulative sum of explained variance ratio of top-k basis vectors E i= Var(A Σ Var
(1)
where A RR. When measuring the explained variance of neurons: A' A (i.c. the activations). When measuring the explained variance of principal components, one could equivalently calculate
Σ
Cumulative sum explained variance ratio of top-k basis vectors
(2)
E==1> A
where X; is the ith eigenvalue of ATA Figure 5 shows comparison between the explained variance of the PCA basis and the neuron basis for AlexNet. activations. Much of the activation variance is concentrated in the most important PCs whereas explained variance is far less concentrated in the neuron basis. For example, to explain 80% of the activation variance for fc1, one could either study the first. 42 PCs, or the 2782 highest variance neurons. Similar trends are observed in every layer, showing PCA provides a more efficient basis for activation space. Notably, conv4 and conv5 roquire a similar number of PCs and neurons to explain 99% of the activation variance, implying these activations have a higher rank structure and PCA is a less suitable decomposition method compared to other layers. This stands in contrast to fc1 and fc2 which appear strikingly low rank.
The success of the PCA is due to significant covariance bet ween neurons in each layer; Without inter-neuron covariance, the PCA basis would be roughly as efficient as the standard neuron basis for explaining activation variance. This measure naturally favors PCA, but the size of the difference is not known in advance. The dramatic difference seen here suggests that is wasteful to omit dimensionality reduction when explaining neural networks.
conv1
DonV2
DonV3
conv4
1.00
Cumuaalive sum
explainad varfanca
0.75
PCA are ReLU
0.50
PCA post-Rbet U
0,25
80% explained var.
.....:
90% eaplained var.
0
20
60
50
100
150
200
100
200
300
400
100
200
Number of basis vectors
Number of basis is vectors
Number of basis vectors
Number of basis vectors
conv5
fc1
fc2
fc3
1.00
veriance
0.75
1008
Cumulelive
0.50
0.00
0
100
200
1000
2000
3000
4000
0
1300
2000
3000
4000
0
750 Numbar of basis vactors
250
500
1000
Number basis vectors
Number vectors
Number basis vactors
Figure 5: Cumulative sum of explained variance ratio for each Alex.et layer plouted against the number of basis vectors being used. Both PCs and neurons are ordered by descending variance. The number of basis vectors required to explain 80% and 99% variance is annotated.
4.1.2 Activation ablation.
Another quantitative mcasure of completeness is the amount of generalization ability that can be attributed
to a set of basis vectors (Amjad el al. 2021 Morcos et al. 2018 Zhou et al., 2018 Raghu el al., 2017)
2 As done scikit-learn (Pedregosa al. 2011
9

---

Published in Transactions on Machine Learning Research (03/2025)
To quantitatively test the importance of each basis, cumulatively ablate basis vectors and observe how much accuracy degrades. Specifically, we cumulatively ablate both neurons and PCs, in both descending and ascending order of activation variance, then measure the effect on AlexNet's validation accuracy.
conv1
conv2
conv3
Donv4
Original Aoc.
Val Acc
PCA 1 Var.
0.4
PCA Var
Nouron
0.2
Neuron Var
0.0
20
60
0
50
100
150
200
0
100
200
300
400
0
200
conv5
fc1
fc2
0.4
0.2
0.0
3000
4000
3000 4000
1000
Number abiations
Number ablations
Number ablations
Numbar ablations
Figure 6: For each Alex layer, we ablate basis vectors in activation space and measure the effect on ImageNet top-1 validation accuracy. Both PCs and neurons are ordered by their explained variance. Reverse order corresponds ascending explained variance.
Figure G shows that AlexNet accuracy degrades fastest when ablating the highest variance PCs, with the exception conv1. Ablating high variance neurons in conv1 is highly damaging because neurons in the first layer of CNNs learn fundamental vision features like edge dctection and color contrasting (Bergstra et al., 2011), OOlah et al. 2020a The lack of dropout. in convi means redundancy is not encouraged so ablating important neuron destroys vital feature for classification for the rest of the model depth. In contrast, since each PC is linear combination of neurons, ablating a PC is unlikely to entirely destroy an important neuron's activations. The fully-connected layers fc1,fc2,fc3 have highly redundant representations where a tiny fraction of PCs is responsible for performance, perhaps due to dropout and extreme width. This is evidence that the most important PCs offer a more complete explanation activation space than the most important neurons, and the least important PCs contribute the least towards explanation completeness. As Figure 5 showed, conv4 and conv5 activations have heavier tailed eigenspectra and PCA may be less suitable decomposition for these layers. We hypothesize this explains why ablating the ~50 highest variance PCs degrades accuracy faster than neurons, but after that point it is more damaging to ablate neurons. Zhou et al. (2018) found that ablating random basis vectors from AlexNet conv5 had less of an effect on accuracy than ablating individual neurons. We show that ablating the most important PCs has a greater effect on accuracy than individual neurons, and thus also has a greater effect than ablating random directions for conv5.
4.2 Quantifying explanation interpretability
Following best practices [Laavitt & Morcos 2020: Gilpin et al. 2018) we measure interpretability via a user study to validate that humans can indeed interpret coherent stimuli along the visualized basis vectors.
We presented users with two visualizations of each PC/neuron (see Appendix F for screenshots). One visualization was randomly shuffled while the other was in the correct order. Participants were instructed to select the visualization that displayed a coherent transition from left to right. If participants cannot accurately determine which one is random, then they cannot interpret the stimulus dimension; the continuity of the visualization is its defining feature.
We tested the following hypotheses: (1) neuron-based visualizations are less interpretable than PCA visu- alizations, and (2) visualizations from shallow layers should be more interpretable than ones from deeper layers.
10

---

Published in Transactions on Machine Learning Research (03/2025)
For both the PCA and neuron basis, we tested the 16 basis vectors with the largest variance from each of AlexNet's 8 layers. Stimuli were presented in random order. The position (top/bottom) of the original and scrambled stimuli was randomized. A visualization consisted of three rows of natural image snippets from the three ncarest neighbours in activation space to points along each basis. Synthetic visualizations were not included due to screen space constraints and limited interpretability. The scrambled versions contained identical snippets in pseudo-random order. 22 user study participants were recruited. See Appendix F for more details.
Figure 7 summarizes the results of the user study: PC visualizations were, average, more interpretable than Neuron visualizations for cach layer in AlexNet with the most pronounced differences seen in layers conv2, fc1, and fc2.
To test the first hypothesis, we ran a one-way ANOVA with basis method (PCA vs Neuron) as a within-subjeets factor. Results showed & significant advantage for perception of conti-
PCA
Use study eccuraay (sk-glden)
muity for the PCA basis visualizations: F(1,21) 34.50, p .001, confirming our first hypothesis.
0.85
0.80
To assess the relative interpretability of visualizations for bases drawn from different layers, we ran separate post-hoe factorial ANOVAs for conv and fc layers as they serve qualitatively different functions. Among fully connected layers, there was an advantage for PCA over Neuron basis interpretability, F(1,21)
0.75
0.70
0.86
convt com22 conv3 convé convi fol
fo2
fo3
= 32.91, p .001; effect of layer depth that is difficult to interpret, F(2,42) = 4.01, p < .5;; and an interaction wetween basis method and layer, F(2,42) = 8.95, p < .001, such that the difference appears pronounced in fci and fc2, but less so in fc3. Among convolutional layers, there was an advantage for PCA over Neuron basis interpretability, F(1,21) = 9.32, p < .001; an advantage for interpretability of visualizations for bases from shallower versus deeper layers, F(4,84) = 10.66, p .001; and no significant interaction.
User study accuracy for each AlexNet layer. Shaded regions indicate the standard error across 22 study participants.
Figure 7:
5 Limitations
We mainly study AlexNet-ImageeNet. We focused on AlexNet pretrained on ImageNet because it is one of the most studied networks in the explainability literature (Bau et al. 2017). (oogg & Vedaldi 2018). (Mu & Andreas 2020), (Zhou et al. 2018). However, we did not compare the explanation completeness and interprotability of neuron and PCA based explanations for other networks or datasots. This potentially limits the generality of our claims regarding neuron-based explanations of neural networks beyond Alex In Appendix A we include cumulative sum of explained variance plots for Imageeet--reerrined ResNet-18, ResNet-50, and ViT-B/16 (Figure concentrated in the high variance PCs compared to the high variance neurons, suggesting our findings may generalize to these networks. In our online demo, we include visualizations of the top 16 PCs for several layers in ResNet-18, ResNet-50, and ViT-B/16.
10 Similar to AlexNet, the activation variance is much more
There are cases where neuron-based explanations might be useful. In principle, a neuron basis could be more interpretable than top PC basis. Such an example could be constructed with an auxiliary loss that explicitly aligns neuron with clear concept. However, standard network training does not particularly encourage this, and the probability of this happening on its own just seems to be low. Neuron- based explanations should be most useful in the first and final layer of a network. Given their linear nature, it is well known that the convolutional filter weights in the first layer can be directly interpreted as Gabor filters, color detectors, edge detectors, etc. Our results in Figure sshow ablating the highest variance neurons in conv1 is more damaging than ablating the highest variance PCs. In the final layer, the weights act as a classification head so each neuron has an interpretable mcaning as it associated with particular class. However, our results in Figures 5 D. and 7 show that the PC bases were more complete and interpretable than the neuron bases for the fc3 layer of AlexNet. Finally, the activation ablation experiments suggest one
11

---

Published in Transactions on Machine Learning Research (03/2025)
could prune the lowest variance PCs neurons form of model compression. While pruning low variance PCs would better preserve accuracy in most layers, it is much easier to achieve hardware acceleration from pruning low variance neurons instead.
We define a low threshold for interpretability. The study in Section 4.2 tests for a low threshold of interpretability because participants do not necessarily need to understand the transition they see, even if they correctly recognize it as coherent. This means that poor performance on this task means that users truly cannot make sense of the visualization. We chose this test design because it was the highest threshold test we could conceive that had a well-defined ground truth. While our user study showed PC explanations were clearly more interpretable than neuron explanations, the insight it can give into deep network function is still rather limited.
We only study the forward pass, not the backward pass. Understanding the backward pass of DNNs can also provide insight into the function of each layer. Instead of sampling forward activations a E R one could sample gradients #c E Rd where L is the loss function. Then [a measures how much changing the Za ith neuron's output would affect the loss. This could be useful for explaining the types of examples where pretrained model still has room for improvement. Similar to forward activations, its possible that PCs provide a significantly more interpretable summary than individual neurons since gradients are also low rank and high dimensional (Yang et al.) 2024), (Zhao et al.] 2024p. The approach presented in our work could help to visualize and interpret dimensions of activation changes that would most affect the loss, but we leave this as future work.
@2
6 Discussion and Conclusion
This study compares the standard neuron basis for explaining deep network function with the principal components analysis (PCA) basis, which is the simplest alternative. In our analysis of AlexNet, the PCA basis was both more complete and more interpretable. In particular, our user study showed that large- cigenvalue PCs reprosented more coherent stimulus transitions than neurons. Furthermore, most of the activation variance in a layer may be explained by dramatically fewer PCs than neurons, and ablating the highest-variance PCs usually affects performance more than ablating the highest-variance neurons (AlexNet's convi layer is an exception). In contrast, smael-eigenvalue PCs are uninterpretable, and they have a small effect on validation accuracy when ablated.
To explain deep networks in terms of neuron or PC activations is to expound rather than clucidate. This aim is closely related to the kind of mechanistic understanding that neuroscience has been seeking for decades. Neuroscience seeks to understand brain function at multiple levels, including how individual cells combine their inputs, how large networks transform information, and how this Icads to behavior. Interestingly, it has been argued that neuroscience can benefit from high-level perspectives that deep learning takes for granted, such as the roles of architect ure, loss, and data in network function (Tripp) 2018 Richards et al.] 2019). At an intermediate level, much of neuroscience involves cataloguing how information is represented and transformed by particular groups of neurons in different parts of the brain, bridging the gap between cell function and behavior. As we discussed in Section 2 this work began with the study of individual neuron responses
due to technical recording limitations, but population- level analyses (Kriegeskorte et al., 2008 Saxena & Cunningham 2019 Ebitz & Hayden| 2021 Urai et al./ (2022), including methods based on dimension reduction [Briggman et al. 2005 (Shenoy et al., 2013 Aoi et al. (020)), are growing in importance, in parallel with
recording density. Our results with PCs motivate future work in XAI with non-linear dimension reduction
[Rombach et al. 2020 Carter et al.) 2019 Templeton et al. 2024 Engels et al., 2024), which is also used in
neuroscience (e.g. Cunningharn & Yu 2014 Niederhauser el al. 2022). More generally, this study supports the view that ANN explainability, like neuroscience, can benefit from a focus on population- analysis.
Broader Impact Statement
While we believe our work can provide practically useful explanations for DNNs, all explainability methods carry the risk that users will be overconfident in the explanations. To help mitigate this risk, we highlight two limitations. Firstly, there are non-linear phenomena in activation space but the PCA basis only provides
12

---

Published in Transactions on Machine Learning Research (03/2025)
linear view of the underlying non-linear manifold [Gallego et al., 2017). For example, tuning curves in
biological neural networks are captured by non-linear manifolds /Cammarata et al. 2020 GGallego et al.
2017 Georgopoolos et al.) 1986), (Hubel & Wiesel| 1962). (Kriegeskorte & Wei 2021 (ovvr et al,
2005 Non-lincar dimension reduction techniques can capture non-lincar manifolds, unlike PCA. This an
important direction for future work because most stimuuli are represented by multiple PCs.
Secondly, it is unclear what small eigenvalue PCs represent in activation space. Small eigenvalue PCs may be interpretable when considered in the context of their interaction with large eigenvalue PCs. They also may represent noise in activation space, Although they have uninterpretable visualizations and explain little variance, they still affect AlexNet's validation accuracy when ablated. Thus, small eigenvalue PCs play some role in network function. They may be needed for representing long-tail inputs that appear infrequently in the training set [Hokerr et al. 2020a), (Hoker et al. 2020b). They could also be artifacts of the non-linear manifolds in activation space (Gallego et al. 2017
Kriegeskorte Wei 2021
Acknowledgments
This research was supported by funding from BMO Bank of Montreal through the the Waterloo Artificial Intelligence Institute (SRA #081648). The authors thank Thomas Fortin for helping to run experiments with ResNet and VIT.
This research was supported, in part, by the Province of Onlario and the Government of Canada through the Canadian Institute for Advanced Research (CIFAR), and companies sponsoring the Vector Institute.
GWT is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada Research Chairs program, and the Canada CIFAR AT Chairs program.
This research was conducted with approval from the University of Guelph Research Ethics Board (REB #20-12-003).
References
Jay Alammar. Interfaces for explaining transformer language models, 2020. URL https://jalann: jamamarr
github. htt/e//ppaiiiinggrrrsffrrreesese
Rana Ali Amjad, Kairen Liu, and Bernhard C. Geiger. Understanding neural networks and individual
neuron importance via onfdeettion-ordered cummlative ablation. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-11, 2021. doi: 10.1109/TNNT. Stt.:/NNNNN...21.3333333 2021.08888..
Mikio C Aoi, Valerio Mante, and Jonathan W Pillow. Prefrontal cortex exhibits multidimennsinal dynamic encoding during secision-making. Nature newroscience, 24111):1400 2020.
David GT Barrett, Ari S Morcos, and Jakob Macke. Analyzing biological and artificial neural networks: challenges with opportunities for synergy? Current Opinion in Neurobiology, 55:55 64, 2019. ISSN 0959-4388. doi: httpp:://doi.org/100111/// wow...219.01..00 URL http://www.siieeeddreett com/ Machine Learning, Big Data, and Neuroscience.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017.
David Bau, Yan Zhu, Hendrik Strobelt, Agala Lapedriza, Bolei Zhou, and Antonio Torralba. Under- standing the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi:
URL https: / /tt::://wwww htwp://wwww....w/////////////... wwtpppaw....ppppp.ptpptppp..pppe arg/content/
James Bergstra, Aaron Courville, and Yoshua Bengio. The statistical inefficiency of sparse coding for images (or, one gabor to rule them all), 2011. URL httpps//aarrii.orrr///////// 6638
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Πya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https: 2023.
//openaipublic. blob. core windows wttaloos.nettneuuoo-eeee http://aeeooooeppppaaaarrraaapap
13

---

Published in Transactions on Machine Learning Research (03/2025)
Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Natural images are more informative for interpreting {cmn} activations than synthetic feature visualizations. In NewrlPS 2020 Workshop SVRHM, 2020. URL
https://openrevieer net/forrmmidd vh02VPjbba
Kevin Briggman, Henry DI Abarbanel, and WB Kristan Jr. Optical imaging of neuronal populations during decision-making: Science, 307(3711):896 901, 2005.
Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, 2020. doi: 222211//disiiill00002222
https:///isstill.pub/ pu/ei///////////riitttt// /itr/rttss/euveeeetttoosss curve--etecctor..
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 2019. doi: 10. 10.23915/die (0.23315/disttll.000 disiiill.0000011 hulps//ddsstilppub)) /ttp:////0///cciiiiooaaaaaass activaaioon-allss.
Andrzej Cichocki and Anh-Huy Phan. Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE Transactions, 92-A:708 721, 03 2009. doi: 10. Aeeaassuu..99.A7700... E92.A.708.
John P Cunningham and Byron M Yu. Dimensionality reduction for large-scale neural recordings. Nature neuroscience, 17(11):1500 1509, 2014.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers, 2022.
Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems. MIT press, 2005.
Egbert De Boer and H Rob De Jongh. On cochlear encoding: Potentialities and limitations of the reverse- correlation technique. The Journal of the Acousticol Society of America, (3(11:115-113) 1978.
Jía Deng, Wei Dong, Richard Socher, Li-Jia Lá, Kai Li, and Li Fei-Fei. TmageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Jonathan Donnelly and Adam Roegiest. On interpretability and feature representations: An analysis of the sentiment neuron. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia HaufT, and Djoerd Hiemstra (eds.), Advances in Information Retrieval, pp. 795- 802, Cham, 2019. Springer International Publishing. ISBN 978-3-030-15712-..
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http::///ggaareeee..eeee/o/oottp http://orrum..dd....d......
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmrnnem Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
R. Becket Ebitz and Benjamin Y Hayden. The population doctrine cognitive neuroscience. Neuron, 109 (19):3055 3068, 2021.
Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear, 2024.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. Tn 2018 IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 8730, 2018. doi: 10.1109/CVV CCVPR.2018.009)
Juan A. Gallego, Matthew G. Perich, Lee E. Miller, and Sara A. Solla. Neural manifolds for the control of movement. Newron, (4(5):978 984, 2017. ISSN 0896-6273. doi: https://do..orr//..00////// 2000 20177.05. 025. URL.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Robert Geirhos, Roland S. Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the ( of feature visualizations, 2023.
Apostolos P Georgopoulos, Andrew B Schwartz, and Ronald E Kettner. Neuronal population coding of movement direction. Science, 233(477)):1116-1410, 1986.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explana- tions, 2019.
Leilani H Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explain- ing explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Dala Science and Advanced Analylies (DSAA), pp. 80-89, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and pattern Recognition (CVPR), pp. 770-778, 2016. doi: CVV./.01...
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. deep neural networks forget?, 2020a. URL
What do compressod
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models, 2020b. URL htttps://arxiv.orgg htwpg//ab///000.. 03058
David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architec- ture in the cat's visual cortex. The Journal of Physiology, 160(1)1106-154, 1962. doi: 10.1113/jphysio.. 1962.sp006837. URL https://pbyso:. onl httpa//aaa.aaa.aa......o//oo///a
sp006837
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Networks, 13(4):411 430, 2000. ISSN 0893-6080. doi: http:://oo..or///1.111////// (0000000000000000000000000000030 6080(00))002665. URL http: www.wwww..oo..om sciencedirect.c ott.accccc..ccooooccccc.acttaaat
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In Proc. NeurIPS, 2020.
Yuta Kanda, Kota S Sasaki, Izumi Ohzawa, and Hiroshi Tamura. Deleting object selective units in fully- connected layer of deep convolutional networks improves classification performance. arXiv: Neurons and Cogmition, 2020.
Andrej Karpathy. t-sne visualization of enn codes. cnnembed/ 2014.
https://es..ttnffrd. wwwp.aaee...eepee////aattttt.///
Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.), , Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2668 2677. PMLR, 10 15 Jul 2018. URL http://proceeddnn mlr, www.ppresseveeee htt////00///m..... kiml8d. hamm8..html
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristol T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un)reliability of saliency methods, 2017.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312..11 2014.
Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Nature Reviews Newroscience, 22 Nov 2021. ISSN 1471-0048. doi:
URL https:
Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity analysis-conneeting the branches of systems neuroscience. Frontiers in systems neuroscience, pp. 4, 2008.
15

---

wtt.:ll///////////llaae/l Published in Transactions on Machine Learning Research (03/2025)
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. ArXiv, abs/ 0tss/1444.997, 2014.
Matthew L. Leavitt and Ari Morcos. Towards falsifiable interpretability research, 2020.
Matthew L Leavilt and Ari S. Morcos. Selectivity considered harmful: evaluating the causal impact of class selectivity in {dun}s. In International Conference on Learning Representations, 2021. URL https:
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In 2015 IEEE Conference Computer Vision and Pattern Recognation (CVPR), pp. 5188-5196, 2015.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2018.
Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learnäng Regresendations, 2018. URL
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In Advances in Neural Information Processing Systems, volume 33, pp. 17153-17163, 2020. URL http:://prcceeddngggooooo......p .dd.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the differ-
ent types of features learned by each neuron deep neural networks. Visualization for Deep Learning workshop, International Conference in Machine Learning, 2016a. arXiv preprint arXiv:1602.03616.
Anh Mai Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for ncurons neural networks via deep generator networks. CoRR, abs/1605.09304 2016b. URL htt:://///a http://aaiiv..rrrabb/666.00000 09304
Thibault Niederhauser, Adam Lester, Nina Miolane, Khanh Dao Duc, and Manu S Madhav. Testing geo- metric representation hypotheses from simulated place cell recordings. arXiv preprint (al:2211.09099 2022.
Harris Nover, Charles H. Anderson, and Gregory C. DeAngelis. A logarithmic, scale-invariant representation of speed in macaque middle temporal area accounts for speed discrimination performance. Journal of Newroscience, 25(43): 2005. ISSN 0270-6474. doi: 10.1523 JNEEROSCL.1611100.0.0. URL https://www.jneurrostt http:////srrrroooooot///////////
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Diställ, 2017. doi: 10.
33915/distill.0000. http:///iiiilllllo//////////eeee
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexan- der Mordvintsev. The building blocks of interpretability. Distill, 2018.
doi: 10.23915/distill0000
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
An overview of carly vision in inceptionvl. Distill, 2020a. doi: 10.23915 htt///iill...022..0000 https://diisiill.pbb/22222 http://////cciiissaaay---iiiieie
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An roducttioo to circuits. Distill, 2020b. doi: 10.23915/daa 133911//disill.oom distill.00024.000 https://distill.pubb/ httpa///220//rcus///ooooo
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurth,, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
16

---

Published in Transactions on Machine Learning Research (03/2025)
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Dean A Pospisil and Jonathan W Pillow. Revisiting the high-dimensional geometry of population responses in visual cortex. bioRrit, pp. 2024-02, 2024.
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment, 2017.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vec- tor canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.); Advances in Neural Information Processing Systems 30, pp. 6076 6085. Curran Associates, Inc., 2017. URL http::/paper:: opg.aaaa------eel-laaaaaaaaaaaee
ott.:eaaaaaaattooooooooooaaooooo canonical-oorrelatioo
-earnie--e-rnn----- dynamiiddnniiiss--- and-aandd--nnreeeeee interprettrrr pdf]
Ankit Rajpal, Khushwant Sehra, Rashika Bagri, and Pooja Sikka. Xai-fr: Explainable ai-based face recog- nition using deep neural networks. Wireless Personal Communications,, 129(1):663 680, Mar 2023. ISSN 1572-834X. doi:
URL httph::t////// //dois http:////ooooo...../o/o////// 011111..07///11 1000///11277000221127777
Pranav Rajpurkar, Emma Chen, Oishi Banerjee, and Eric Topol. Ai in health and medicine. medicine, 28(1):31-35, 2022.
Nature
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Chris- tensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761 1770, 2019.
Robin Rombach, Patrick Esser, and Björn Ommer. Making sense of cmns: Interpreting deep representations & their invariances with imns. In Proceedingss of the European Conference on Computer Vision, 2020.
Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embedding, Science, 290(3500):2323 2326, 2000. ISSN 0036-8075. doi: 10.1126/scciii 2tteemcc.290.55002222222.... URL https: //science. scw.ncemag....c......c.......... htt//ooteentt20/5500/2000
Shreya Saxena and John P Cunningham. Towards the neural population doctrine. Current opinion in
neurobiology, 55:103-111, 2019.
Elad Schneidman, William Bialek, and Michael J Berry. Synergy, redundancy, and independence in popula- tion codes. Journal of Neuroscience, (37)):115539 11553, 2003.
Bernhard Schölkopf, Alexander J. Smola, and Klaus-Robert Müller. Kernel principal component analysis. In Wulfram Gerstner, Alain Germond, Martin Hasler, and Jean-Daniel Nicoud (eds.), Arlificial Neu- ral Networks ICANN 197, 7th International Conference, Lausanne, Switzerland, October 8- 8-10, 1997, Proceedings, volume 1327 of Lecture Notes in Computer Science, pp. 583-588. Springer, 1997. doi: 10.1007/BFboOe
URL httpps://di..oo www//o./..........ooo 10011007/BB000200117
Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers, 2023.
Krishna V Shenoy, Maneesh Sahani, and Mark M Churchland. Cortical control of arm movements: a dynamical systems perspective. Annual review of neuroscience, 36:337-3503 2013.
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.1599 15324, 2023.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034 2013.
17

---

Published in Transactions on Machine Learning Research (03/2025)
lan H Stevenson and Konrad P Kording. How advances in neural recording affect data analysis. Nature neuroscience, 14(2):139-144) 142, 2011.
Carsen Stringer, Marius Pachitariu, Nicholas Steimmetz, Matteo Carandini, and Kenneth D Harris. High- dimensional geometry of population responses in visual cortex. Nature, 571 (7765))) 361-365, 2019.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum Mc- Dougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jormyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monoscmanticity: Extracting inter- pretable features from claude sonnet. Trunsformer Circuils Thread, 2024. URL https: http:///taafffoomrrrrr circuits.
htmll
Bryan Tripp. A deeper understanding of the brain. Neurolmage, 180(Pt A):114 116, 2018.
Anne Urai, Brent Doiron, Andrew M Leifer, and Anne K Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature neuroscience, 25(1):11 19, 2022.
Laurens der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579 2605, 2008.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Grifiin Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 214-229, 2022.
Greg Yang, James B. Simon, and Jeremy Bernstein. A spectral condition for feature learning, 2024. URL attps://arxiv.co httg/bb//23111 17813
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completenes--aware concept-based explanations in deep neural networks. In Advances in Neural Informa- tion Processing Systems, volume 33, pp. 20554-20565, 2020. URL https::/prooceciigs..em mewpoeidddnnnn.uuns...o.. neurips. cc/ .pff
Richard S Zemel, Peter Dayan, and Alexandre Pouget. Probabilistic interpretation population codes.
Neural computation, (0(2):403-430) 1998.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection, 2024. URL httpps://rrxiv.org abs/2403.03507
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation, 2018.
A Explained variance of ResNet-18, ResNet-50, and ViT-B/16
In addition to the cumulative sum of explained variance results for AlexNet (Figure we also include analogous plots for ImageNat-pretrained ResNet-18, ResNet-50, and VIT-B/16 in Figures g and 10 respectively. ResNet-50 activations appear more strikingly low-rank due to the bottleneck residual block structure which constrains the activation rank to 1/4 of the full rank. For all networks studied, much of the activation variance concentrated in the mast important PCs whereas explained variance far less concentrated in the neuron basis. Similar to AlexNet, these results suggest wasteful omit dimensionality reduction when explaining ResNet-18, ResNet-50, ViT-B/16.
18

---

Published in Transactions on Machine Learning Research (03/2025)
conv1
layer 1 Boonv2
layer1@ @comm
1aye/2 2on2
2 @con2
1.00
0.75
0.50
Noaron Rill
0.25
87% neckiiee vir
neckiner voe
20
40
00
20
40
60
20
40
60
50
100
50
100
Nimmerr bask vectooe
Numrer of base veclor
N.mbe bands redor
N.mber of baods vectors
Namberra bexc vedtors
1ayer3gg@conr/2
layer3 ggonnv?
layor40ggon2
layer4@1 1eerrgg1@.oom. gcoo22
Cunustaa BLEE ol
1.00
0.75
0.00
0.25
0.00
Numoer ofbasi
Number ofbasi
N.mber bumberofbadds vedtors
Number of basds
Number
Figure 8: Cumulative sum of explained variance ratio for the final layer in cach ResNet-18 residual block. Both PCs and neurons are ordered by descending variance. The number of basis vectors required to explain 80% and 99% variance is annotated.
Inyert carrd
182
Inyer? go@romd
by &
1.00
Ouruute urr
FCA peetal
20
40
106
200
130 co
200
106
208
200
200
40
lsyer2g
byerrgg 2oomv3
200
493
200
20
108
790
1930
200
12
200
100
1990
293
400
1009
Layerag 4ogcom
tyyer goomm
com
syerig goomm
Jayort com
1.00
110
250
750
1000
200 1109 2030
(20
100
2003
1000
1168
2003
420
1593
Harber
Figure 9: Cumulative sum of explained variance ratio for the final layer in each ResNet-50 residual block. Both PCs and neurons are ordered by descending variance. The number of basis vectors required to explain 80% and 99% variance is annotated.
B Masking natural image regions that do not contribute strongly to activation similarity
The visualizations for deeper layers with larger receptive fields are more difficult to interpret than early layers with small receptive fields because there is more area in the natural image for distractors to be present. To help alleviate this problem, we mask out the regions of the visualizations that do not strongly affect the proximity of the representation to the point a in representation space that we are visualizing.
Let hy, wi, and d, be the height, width, and number of channels in each image patch I. The sensitivity gradient G e X is the gradient of the & distance between the target point and a', the activation from forward propagating I, with respect to the pixels in the image patch I:
G
a
a' a'|l-
(3)
19

---

Published in Transactions on Machine Learning Research (03/2025)
conv_proj
layer Dell attention
layer_ -Bpppp
layer 30 silention
layer 3mmpp
1.00
0.75
0.00
hoaron RoL)
0.25
66% exposaee vor
sels eqpitaedd vor
200
400
m03
100
200
400
E03
100
200
400
103
100
0
200
400
600
800
200
400
600
100
Namberr boods swrzon
Number boods vedtor
N.moer bank redon
Namoer vector
Namoor hax Vectors
layer__ 70telff attention
layer_ 7@0pp
kyer 11 ti@ee attention
layer 11@mlp
head
1.00
Cunustia RuTE
0.75
0.00
0.25
0.00
Number badds
Number baods
Number baast
ofbas
ofbass
Figure 10: Cumulative sum of explained variance ratio for the ten layers in ViT-B/16. Both PCs and neurons are ordered by descending variance. The number of basis vectors required to explain 80% and 99% variance is annotated.
Next take the sum of the absolute gradient values over channels and apply 2D Gaussian filter g(,, y, o):
where a = 3, to obtain Gsmonth E R follows:
G Smmooth 6(00,,,0) (1 ( (G.).
To determine the Boolean image patch mask M R we threshold Gsmooth to be greater than one standard deviation of Gmmooth follows:
M Gsmooth std aamooth))
(5)
The resulting mask M is applied element-wise to the image patch I M to hide regions of the image that do not strongly affect the distance ||a a'l]-
c Comparison of methods for sampling points along basis vectors
We sample and visualize m points along basis vector with the goal of understanding how activation variance along the basis vector relates to input variance. We compared two strategies: sampling between the first and ninety-ninth percentile, or sampling between the minimum and maximum values along the basis vector. After studying several basis vector visualizations from each AlexNet layer, we concluded that sampling between the minimum and maximum values produced more interpretable visualizations.
In Figure [1 we show a comparison of the two strategies for the second PC of conv1, which represents the Gabor phase of vertical edge. Although the first and ninety-ninth percentile visualizations demonstrate the concept of Gabor phase, the extreme values along each basis vector produce visualizations that demonstrate the concept more distinctly.
D Activation maximization versus distance minimization
When we visualize basis vectors, use distance minimization rather than the more common activation maximization objective. Activation maximization involves producing a visualization that maximizes the activation along some basis vector (Mahendran & Vedaldi 2015), (Nguyen et al. 2016a). (Nguyen el al., 2016b), (Olah et al. 2017), OOla et al., 2018). One issue with activation maximization is that there are many degroes of freedom; to maximize neuron 1, visualization may also elicit large activations from neuron 2 and 3. /Geirhs et al. (2023) show how this freedom makes activation maximization visualizations unreliable. It is difficult. to precisely study distributed representations because neurons cannot. be studied independently
20

---

Published in Transactions on Machine Learning Research (03/2025)
Miamum mudi
to 68th percenble
101
Fargrry 1ag
16
-0
aking principal compareent
50.200 43.2266
300422 200611 32.0474 3.2085 6.41 18:72:0 21.324 37.743
47.37L
Histogram of activations from the ImageNet training set, projected onto the second PC (PC 2) of convi (log scale). The interval defined by the minimum and maximum is shown in green. The interval defined by the first and ninety-ninth percentile is shown in red. Middle and bottom: Visualizations of points along PC 2 of conv1, uniformly sampled between the first and ninety-ninth percentile (Middle), and uniformly sampled between the minimum and maximum (Bottom).
Figure 11: Top:
of each other. second issue that we wished to characterize how representations change along a given dimension, whereas activation maximization addresses only the extremes. Distance minimization involves producing visualization that minimizes the distance between the visualization's response in activation space, and some target point in activation space. This allows study how variance along single basis vector in activation space relates to variance in input space. The difference between these two objectives is ralustrated in Figure 12
V2
V2
V1
V1
Activation maximization minimization
Distance
Figure 12: Toy example illustrating the difference between activation maximization and distance mini- mization when attempting to visualize the basis vector VI. Blue dots represent the resulting optimized visualizations.
21

---

Published in Transactions on Machine Learning Research (03/2025)
E Computational resources
Our computations were performed on machines with an 8 core Intel Xeon CPU, NVIDIA T4 GPU, and 64 GB of RAM. Sampling activations from an AlexNet layer for the ImageNet training set (n -~ took approximately 2 hours. The runtime of fitting PCA to the sampled activations is dependent on the dimensions of A, and can take anywhere from seconds for conv1 (d = 64) to 22 minutes for fc1 (d 4096). Producing natural visualizations of 32 points along basis vector, with neighbors, takes approximately 4 seconds.
F
Additional User Study Details
We recruited 22 participants from Prolific in exchange for #6.6 CBP//r. All participants provided informed consent to use their anonymized data in accordance with the institutional review board.
The scrambled re-ordering for comparison images followed some consts. When piloting the task with fully random scrambling, we discovered it was trivially easy if the NN row contained duplicate snippets, because they would not appear next to each other in the scrambled version (detecting two or more identical snippets not adjacent was a giveaway that it was the scrambled stimulus, and could be used as a signal to complete the task). To remedy this, comparison images were psoudorandomly ordered such that NN snippets that appeared multiple times within row were required to be horizontally adjacent.
The full text instructions presented to each participant in the user study are shown in Fig. 13 Participants read these instructions at their own pace and proceeded with a button press. Fig. 14 shows a screenshot of the user study task screen. The distributions of correct responses for each AlexNet layer are shown in Figure
15
G Additional visualizations
To view visualizations PC and neuron visualizations for every layer of pre-trained AlexNet (Krizhevsky 2014), in addition to PC visualizations from every block of pre-trained ResNet-18 and ResNet-50 (He ot al. 2016) and layers of ViT-B/16 (Dosovitskiy et al. 2021), please refer to our interactive demo: https: wtt.gggtt....oeeeeeooeeeeeeppppa
It is difficult to oxhaustively show visualizations in paper have chosen show the visualizations of the 5 highest variance PCs and neurons for each AlexNet layer in Figures 2 After conv1, the neuron basis becomes quite uninterpretable, as confirmed through our user study.
3 16
19 and 20
Shttps//wwww.pooffff..om
22

---

Published in Transactions on Machine Learning Research (03/2025)
designed perform difficult tasis high level. Oftee, Al will perform understanding
cortain decision behaved cortain way. response, All engineers are developing tools explain how Al behawes.
that that features scene that the responds to-iil other words, the input features that it "Tooks for" and that guide its object idenifiiation This tool will show you bunch images that together the range of some feature, For esample, the images lines difforent orientation. transttioning
this expertment, can objects scene,
recently developed
below
images left
right:
Your task study is identify patterns. unbiased observers (you) can notice these parterms. On every trial, we will show you two versions the same visualtzaiio: that the output from our visualization
know whether
scrambled
original. short, your notice something transibioning from left right.
images
We've ve combined diffenent visuellation techniques that are supponed Illutrate the same feature. Use them togather to guide your decison. Togather, they look like this:
all threa, make out hor zontaa contours on the left that tranation to vertical right.
Here is enother example. This one easy! This visualization shows that some part Al senstiive to red lronge inputs while others look for blue.
Kammmber,, we are going to show you two versions the same visueloations the original and randomized version of the original You need choose the orighaal one, look visullaations to show some continutty from left right. Below, see what trial might look like They won't this
akalttionn diples sohesve lef
Bripee the seamrel 6. wrien alc aeeuootton donn srmve
Pleasse Labe your time throughout the study. you can't tell which one is rendom, melos your best guess.
Figure 13: Full text instructions presented to participants of the user study.
23

---

Published in Transactions on Machine Learning Research (03/2025)
Trial 25/288
Which of these visualizations displays coherent transition from left right:
Respond using the up arrow top visualization arrow ( select the bottom isualization, can't tell, make your best
Figure 14: Screenshot of the user study task.
15.0
Dendy
0.7
07
Tradion cannd
Tructon
conv5
62
0/6
16
38
Fradt of CO00OL 11000/00
Pradkon olovvd 2o.00 acs
Fradlon ofourec soporrice
Figure 15: Distribution of correct responses for PCA and Ncuron visualizations, plotted for each AlexNet layer.
24

---

Published in Transactions on Machine Learning Research (03/2025)
conv3
PC
PC 2
PC 3
PC 4
PC
N1
N2
N
N
5
Figure 16: Visualizations of points along the 5 highest variance PCs and 5 highest variance neurons for AlexNet's conv3 layer.
25

---

Published in Transactions on Machine Learning Research (03/2025)
conv4
PC
PC
3C 3
PC
PC 5
N
N
N4
55
Figure 17: Visualizations of points along the 5 highest variance PCs and highest variance neurons for Aet' conv4 layer.
26

---

Published in Transactions on Machine Learning Research (03/2025)
conv5
GE
P
Trac
2C
PC 3
PC
PC 5
N
N2
N3
N
N
Figure 18: Visualizations of points along the 5 highest variance PCs and 5 highest variance neurons for AlexNet's conv5 layer.
27

---

Published in Transactions on Machine Learning Research (03/2025)
fc2
PC 1
PC 2
PC
PC 4
N
N2
N3
N
N5
Figure 19: Visualizations of points along the 5 highest variance PCs and 5 highest variance neurons for Net's fc2 layer.
28

---

Published in Transactions on Machine Learning Research (03/2025)
PC
2C
PC
PC 4
PC
N1
N2
N3
N
N5
Figure 20: Visualizations of points along the 5 highest variance PCs and highest variance neurons for AlexNet's fc3 layer.
29


Reviews:
[{"review_id": "p8oizwKXzS", "reviewer": "Reviewer_EkJK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Understanding the inner workings of deep neural networks is an important problem.\n- The website demo can be a very useful tool for explainibility analysis of deep neural networks.\n\n## Weaknesses\nOverall while the authors are studying an important problem the paper is quite unclear about a lot of key details. Moreover, the empirical evidence is severly lacking.\n\n- The paper cites [1] for how they're visualizing the points along PCs. Some background on how this visualization technique works would be helpful.\n\n- I am also a bit confused about Fig. 4. It seems to be confirming a well known fact which is that the top PCs of a subspace account for the majority of the variance. However, it is not clear to me what the authors mean by \"explained variance\" or \"activation variance\". Could you provide a mathematical expression for what this quantity is?\n\n- The paper only studies AlexNet trained on ImageNet. Given that this paper is trying to make a general claim about the advantages of PC vs. neuron basis the authors should include experiments on other deep learning architectures.\n\n[1] Andrej Karpathy. t-sne visualization of cnn codes. https://cs.stanford.edu/people/karpathy/ cnnembed/, 2014.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper argues that neuron-based explanations on deep neural networks are insufficient. The paper claims that this approach fails to accurately represent a network's function and makes the interpretation difficult to understand for humans. The paper proposes to instead utilize the activation principal components for neural network explanations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- More experiments on other datasets / deep learning architectures.\n- A discussion on how these insights are different from the traditional understanding of PCA.\n- More details on how the visualization technique as well as Figure 4.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yGVJyVPs3H", "reviewer": "Reviewer_HkFH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The study is very well written, clear and to the point. The motivation is well explained, and the experiments well justified. The paper very focused and demonstrates, using the chosen measures, that activation based explainability is superior.\n\nWhile the overall conclusion is not entirely novel, having been hinted at in various places before, this study collects clear evidence in support of activation-based explanations.\n\nThe interactive demo is also very nicely done.\n\nWhile I'm convinced of the conclusion of the paper, perhaps a weakness of this paper is that it doesn't have a honest discussion of the limitations of their approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors demonstrate that the principal components of activation provide a more useful method for DNN analysis compared to neuron-based methods. They quantitatively measure completeness (using % of variance captured, and activation ablation) and interpretability (using user studies) to demonstrate this, and conclude that using activation based explanation methods is better.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would be useful to see a detailed and honest discussion of the limitations of the approach here, and perhaps provide some counter-examples (constructed cases where neuron-based explanations might be better).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u5Lzt4F6sD", "reviewer": "Reviewer_matW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n•\tExplainable deep learning is a critical area of research that holds significant potential for advancing the application of deep learning across various scientific domains. The topic is expected to be of considerable interest to the general audience of TMLR.\n\n•\tThe presentation of methodology is clear and accessible, and the illustration in Figure 1 is very helpful in understanding the big picture of methodology.\n\n•\tThe authors provide a clear and interactive demo that contains feature visualizations of neurons and PCs.\n\n•\tThe experiments include a user study to show the explainability of PC visualizations and the hypotheses are evaluated by conducting statistical analysis.\n\nWeaknesses:\n\nIt appears that the explanations provided for PCs and neurons in this paper are based solely on the representations generated during the forward pass. However, in my opinion, understanding the backward pass of DNNs is equally important for grasping the features learned by each layer during training. It would be beneficial to show or at least discuss how the proposed procedure could be adapted to analyze the backward pass. For instance, as noted in Section 4.1.1, PCA-based explanations require high inter-neuron covariance. Does this requirement consistently hold throughout the training process of DNNs?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the submission, the authors argue that neurons are a poor basis for the embeddings of deep neural networks (DNNs) and propose a new method for explaining the intermediate representations of DNNs based on dimensionality reduction, or more specifically, principal component analysis (PCA). For each layer in DNNs, we first perform PCA on samples drawn from the activations obtained by performing forward propagation. Then we find the orthogonal basis vectors and sample points along each basis vector. Finally, these points are visualized in 2D using t-SNE and interpreted in plain text. The authors test this methodology on AlexNet pre-trained on ImageNet and evaluate both the completeness and interpretability of the explanations using PCs and neurons. The results indicate that the most important PCs provide more complete explanations than the most important neurons, and the visualizations based on PCs are also more interpretable than those based on neurons.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "•\tThe comparison of visualizations from PCA and individual neurons in Appendix F provides a good demonstration of the interpretability and it would be better to move some part of the results into the experiments in the main body.\n\n•\tIn Figure 5, it appears that for some **intermediate** convolutional layers (e.g., $\\texttt{conv4}$, $\\texttt{conv5}$), the PCs exhibit a similar rate of performance degradation compared to neurons as ablation continues. This behavior seems to largely differ from that observed in fully connected layers. Could this phenomenon be related to the method used for sampling spatial positions in the convolutional layers? A more detailed explanation would be appreciated.", "broader_impact_concerns": "I do not foresee any ethical concerns in this work, and the authors have appropriately included a Broader Impact statement in the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have presented clear evidence in experiments to show that the most important principal components provide more complete and interpretable explanations than the most important neurons. Major concerns of reviewers have been addressed in the rebuttal.
Decision: Accept (poster)