Discovering Influential Neuron Path in Vision Transformers
Authors: Yifan Wang
Yifei Liu
Yingdong Shi
Changming Li
Anqi Pang
Sibei Yang
Jingyi Yu
Kan Ren
Venue: ICLR 2025 Poster (2025)
Keywords: Explainability
Vision Transformer
Neuron
Primary area: interpretability and explainable AI
TL;DR: We propose a method to identify and trace neuron path, the most influential neurons across layers in vision Transformers, revealing their internal mechanisms and enabling applications like model pruning.

Abstract: Vision Transformer models exhibit immense power yet remain opaque to human understanding, posing challenges and risks for practical applications. 
While prior research has attempted to demystify these models through input attribution and neuron role analysis,
there's been a notable gap in considering layer-level information and the holistic path of information flow across layers.
In this paper, we investigate the significance of influential neuron paths within vision Transformers, which is a path of neurons from the model input to output that impacts the model inference most significantly.
We first propose a joint influence measure to assess the contribution of a set of neurons to the model outcome.
And we further provide a 
layer-progressive neuron locating
approach that efficiently selects the most influential neuron at each layer trying to discover the crucial neuron path from input to output within the target model.
Our experiments demonstrate the superiority of our method finding the most influential neuron path along which the information flows, over the existing baseline solutions.
Additionally, the neuron paths have illustrated that vision Transformers exhibit some specific inner working mechanism for processing the visual information within the same image category. 
We further analyze the key effects of these neurons on the image classification task, showcasing that the found neuron paths have already preserved the model capability on downstream tasks, which may also shed some lights on real-world applications like model pruning.
The project website including implementation code is available at https://foundation-model-research.github.io/NeuronPath/.

Published as a conference paper at ICLR 2025
DISCOVERING INFLUENTIAL NEURON PATH IN VISION TRANSFORMERS
Yifan Wang', Yifei Liu', Yingdong Shi', Changming Li', Anqi Pang", Sibei Yang', Jingyi Yu', Kan Ren
'Shanghai Tech University, Tencent PCG {wangyf7, renkan) wwweeeeaaaaaahhheeeeeoo hhha edu.
ABSTRACT
Vision Transformer models exhibit immense power yet remain opaque to human understanding, posing challenges and risks for practical applications. While prior research has attempted demystify these models through input attribution and neuron role analysis, there's 's been notable gap in considering layer-level infor- mation and the holistic path of information flow across layers. In this paper, we investigate the significance of influential neuron paths within vision Transformers, which is path of neurons from the model input to output that impacts the model inference most significantly. We first propose a joint influence measure to assess the contribution of set of neurons the model outcome. And we further provide layer-pre progreesrgesseeeeee neuron locating approach that efficiently selects the most in- fluential neuron at each layer trying to discover the crucial neuron path from input output within the target model. Our experiments demonstrate the superiority of our method finding the most influential neuron path along which the information flows, over the existing baseline solutions. Additionally, the neuron paths have il- lustrated that vision Transformers exhibit some specific inner working mechanism for processing the visual information within the same image category. We further analyze the key effects of these neurons on the image classification task, show- casing that the found neuron paths have already preserved the model capability on downstream tasks, which may also shed some lights on real-world applica- tions like model pruning. The project website including implementation code is available at http:://foonnaaioommod------eeee htw.:archhgiiiubbboooeuoooaaaaoa
INTRODUCTION
Transformer (Vaswani et al. 2017) models in the vision domain, such as supervised Vision Trans- formers (Dosovitskiy et al. (ViT) or self-supervised pretrained models (He et al., 2022/Qquab ct al. 2023). have showcased remarkable performance in various real-world tasks like image classi- fication (Dosovitskiy et al. 2021) and image synthesis (Peebles & Xie 2023). However, the inner workings of these vision Transformer models remain elusive, despite their impressive achievements. Understanding the internal mechanisms of vision models is crucial for both research and practical applications. When confronted with the model decision outputs, one may raise some questions that, how is the vision Transformer model processing the input information by layer, and which part of the model is significant to derive the final outcome? Unraveling the synergy within these models is essential for comprehending machine learning systems. The current lack of complete understanding poses challenges and risks when deploying these models in real-world scenarios.
Literature has explored mechanism discovery and explainability on vision models. Previous works about the explainability of vision models mostly focused on the visualization of inner patterns of models (Zeiler & Fergus, 2014; Zhou et al. 2016). which straightforward in demonstrating the attention of each module in the model. However, visualization based methods are lack of theoretical support and highly dependent on human subjective understanding, which can be ambiguous. Other preliminary works focus on input attribution (Chattopadhay et al. 2018 Koh & Liang 2017 Li et al. 2016 Selvaraju et al. 2017 Sundararajan et al. 2017), trying to distinguish the influential
"Correspondence Kan Ren.

---

Published as a conference paper at ICLR 2025
Transformer Encoder
Feed-Forward Network across different layers
Closs probakkicc
Low-level information
High-level information
LX
Imputimage
Feed- Feed-forward Network
OOOO 0000
Class Quail Pincapple
Multi-Heed Hied Attention
Embecided Petches
Path
Input Imago
Unsolected Neuron
Selected Nouron
Nouron Path
Figure 1: The illustration of the main concept of our work, focusing on the feed-forward network (FFN) component within a standard ViT (Dosovitskiy et. 202T) encoder. In the left part, a typical VIT encoder is depicted, consisting of totally L Transformer layers. The right part illustrates the neuron path discovered by our method, which identifies path comprising of the neurons within the FFN module across the model layers. Each FFN in the encoder is denoted as FFN I E [1, [].
region of the input. However, imput explanation heavily relies on input images and cannot actually reveal the interior mechanism of the target model. Nowadays, discovering and explaining the im- portant neuron one trained model has drawn more and more attention. The related works either explain the patterns obtained at different network neurons via visualization (Foote et al. 2023 Ghi- asi et al. 2022 Karpathy et al. [0115] Vig 2019 Zeiler & Fergus 2014), or study the effects of individual neurons (You et al. 2025 Dai al. 2022 Dalvi et al. 2019 Durrani et al. 2020 Huang et al. 2023). Nevertheless, most of these methods overlook the correlation between the neurons in different layers and have not considered the complex joint influence of various neurons.
To recognize and understand the underlying mechanisms of multiple neurons across layers in state- of-the-art vision models, pathway in neuroscience (Gupta et al., 2020), our work deliberately concentrates on uncovering the influential neuron path within the vision Transformer models, which is sequence of neurons within each layer tracing from input to model output, to elucidate their significance and role in model inference. Details of neuron path are shown Figure [ To evaluate the importance of set of neurons, we initially define a comprehensive joint influence measure for the target model. Using this measure as guide, we demonstrate our approach using the image classification task (Deng al. 2009 Dosovitskiy et al.) 2 | He et al.) (022). illustrating how we identify the most influential neuron path that drive the model inference through a layer-progressive neuron locating algorithm.
explanation methods needed. Inspired by the concept of visual
We have conducted several quantitative and qualitative experiments on the found neuron path, il- lustrating the significant role plays and the advantage of our solution discovering and explaining the critical part of vision Transformer models. The initial discovery about the effectiveness of proposed method and the critical impact that neuron path exerts to the model inference. What follows is that common influential neuron paths exist in the target model for images within the same category, and categories with similar semantic information also exhibit relatively high similarity in their neuron paths. Furthermore, the discovered neuron paths offer potential insights for model pruning and specialization Vision Transformer models.
In summary, the contribution of our work is as follows. (i) We have proposed a novel method to reveal the crucial information transmission flow within vision Transformer model, based locating and analyzing internal neurons neuron path. (ii) We have performed series of experiments, not only demonstrating the eritical influence on the model inference brought by the discovered neuron paths, but also revealing the intrinsic potential mechanism of vision Transformer models on process- ing semantic information. (iii) We apply the identified neurons to guide model pruning, showing that vision Transformers exhibit redundancy and that the essential components are sparse.
2
RELATED WORK
2.1
VISION TRANSFORMER MODELS
In computer vision, various vision Transformer models (Caron et al. 2021 Dosovitskiy et al. 2021 Oquab et al, 2023 Radford al. 2021) have showcased superior performance real-world tasks
2

---

Published as a conference paper at ICLR 2025
such image classification (Dosovitskiy et al. 2021). Transformer-basee encoders typically consist of multiple structural identical layers, each comprising a self-attention module and a feed-forward network (FFN). Typically, vision Transformer operates by dividing input images into fixed-size patches, treating them akin to tokens in natural language processing (NLP) (Vaswani et al.) 2017), incorporating of positional encodings, classification token, and processing them through self- attention mechanisms with subsequent FFNs. While vision Transformers have achieved excellent performance the underlying mechanisms that contribute to their effectiveness remain opaque. Sev- eral recent works have attempted to unravel the inner workings of vision Transformer by attributing individual neurons to different concepts using language models (Guertler et al. [023]] |eemandez
et al. 2021) or multimodal model (Oikarinen & Weng. (023). visualizations (Chefer et al.) 2021 Ghiasi et al. 2022) or by observing inner behaviors or properties (Naseer et al. 2021 Paul & Chen 2022 Zimmerrman et al., 2024). However, the complexity of vision Transformer' architecture and the interactions between its blocks pose challenges in fully comprehending its capabilities. There- fore, our work seeks to explore the inner working of vision Transformer by investigating the role of
neuron paths, capturing essential patterns of information flow within the model.
2.2 EXPLAINABILITY METHODS IN VISION MODELS
Visualization based Methods. Visualization- based methods are pivotal in interpreting deep neural networks used in vision tasks. These techniques primarily include methods like activation maximiza- tion (Erhan et al. 2009), saliency maps (Selvaraju al.. 2017), and class activation maps (CAMs) (hou et al. 2016). Activation maximization involves optimizing input images to maximize the activation of specific neurons, thereby providing insights into what features activate particular neu- rons. Saliency maps compute gradients of the output with respect to the input image to highlight regions contributing most to the network's prediction. CAMs uses the weighted sum of feature maps to visualize class-specific regions in the input image. Despite their usefulness, these methods face limitations. They can be computationally intensive and may produce unrealistic images that do not correspond to natural inputs (Nguyen al. 2016). Further, they often suffer from noise and lack spatial localization, potentially leading to misleading interpretations (Smilkov et al. 2017). And these methods rely specific network architectures and may not generalize well across different model types or tasks (Selvaraju et al. 2017). Visualization based methods represent a distinct per- spective on explainability other than ours. They are more inclined intuitively display the concerns of the visual model and succinctly represent the learning results of the model, whereas our focus on the intrinsic mechanism of the model and an in-depth examination of the model's operational mechanism from input to output.
Neuron based Methods. The emergence of powerful deep learning models has stimulated interest in the study of model explainability. The concept of a neuron in machine learning, as discussed in
various works (Bau et al.] [017] Dhamdhere al.] [019 Oikarinen & Weng (023), encompasses hidden units, hidden layers, and individual latent variables. These elements are often considered fundamental computational components that process input information. Among the explainability approaches, neuron-based methods, a more intricate form of analysis (Fan et al. 2024). are typically divided into gradient- based and non- based methods. Non- Non-gradient based methods generally contain three approaches: neuron visualization (Foote et al., 2023 Karpathy et al., 20155] Li ct al. 2016 Zeiler & Fergus 2014), concept- baneept-based based methods (Bau et al. 2017] Hernandez et al., 2021 Kim et al.] 20188] Oikarinen & Wengl 2023 Panousis & Chatzis 2024) and baatittiebbsedd methods GGhorbani & Zou| 2020 Kwon & Zou| [022] Shha et 2021 Yuan et al. 2021). However, these methods either rely on a data set of close of input-concept pairs (Bau al. 2017). limiting their generalization ability to unseen concepts, or externally provided powerful models (Oikarinen) & Weng 2023 Panousis & Chatzis 2024). such as CLIP (Radford et al. T. rather than intrinsic model properties. Further, some of their optimization goals focus on model output metrics, failing to fully elucidate the circulation of model knowledge and concepts. Gradient-baan baasee methods com- pute the attribution from output to input or target neurons (Chattopadhay et al.) 2018 Dai et al.) 2022 Dhamdhere et al. 2019 Selvaraju et al. 2017) via gradient computation or integrated gradi- ent (Sundararajan et al. 2017). Though providing quantitative results for neuron explainability, they tend to treat all neurons equally and struggle with comparing neurons across different layers with the gradient flow numerically equitably, despite variations in depth. To address this issue, (Lu al.) 2020 (02T) try to discover a path of neurons rather than a cluster. However, all these methods fail to consider the influence of neurons across layers. It will not only result in biased outcomes due
3

---

Published as a conference paper at ICLR 2025
the domination of few abnormally prominent neurons, but it will also fail to consider the coherence of information transmission, which makes it challenging to truly explore the intrinsic mechanism of the model. The objective of this paper is to propose new method that addresses this issue.
In response to the limitations of existing methods, particularly those gradient- methods focus- ing individual neurons input features, introduce novel approach for interpreting vision Transformer models (Dosovitskiy et al. 20.1). Our method shifts the focus towards the collective contribution of selected neurons across different layers, offering a more comprehensive understand- ing of model behavior. Our method enables us to uncover influential pathways within the model, enhancing its explainability and providing valuable insights into its decision decision-making process.
NEURON PATH
As demonstrated above, vision Transformers (Dosovitskiy et al. 2021 exhibit complex structures and designs that present challenges for explainability. Research vision Transformers predomi- nantly focuses on visualization techniques related to attention maps (Caron et al. 2021; Oquab et al.) 2023), with limited exploration their neuronal anatomy. While recent studies, efforts have been made to associate individual neurons with various concepts using language or multimodal models (Guertler al., [023 Hernandez et al.
2021 Oikarinen & Weng 2023) and visualization tech- niques (Chefer et al. 2021 Ghiasi et al. 2022). the understanding of vision Transformer's 's internal neuronal organization remains relatively unexplored. Some related studies (Dai al., 2022 Geva et al. 2021 Mitchell et al., 2021) have suggested that the FFN component of Transformer models can function as memory mechanism due its structural similarity to the attention mechanism and high-level concepts, which motivated this work to focus these neurons our experiments. Based on this insight, is reasonable to hypothesize that knowledge or information related to visual patterns in vision Transformers may also be stored within the FFN. As stated before about the definitions a neuron (Bau et al., 2017; Dhamdhere et al.) 2019) Oikarinen & Weng. 2023), focus a specific type of neuron within the vision Transformer which is the activation of the output from the first linear layer in the FFN module of each Transformer block.
they encode
3.1 PRELIMINARY STUDY
We first conduct preliminary study on neuron im- portance analysis for vision Transformer models illustrate the motivation of our method. The exist- ing works studying neuron activities of vision model inference (Dhamdhere et al., 2019 (Sundararajan jan al. 2017) mainly focus on convolutional neural networks (CNNs), trying to analyze convolution fil- ter's sensitivity to the input. To assess the impor- tance of neurons themselves within pretrained mod- els, the concept of knowledge attribution (Dai et al. 2022) offers valuable insights finding the individ- ual neurons storing model knowledge such as "capi- tal of country" in Transformer-ban based language mod- els like BERT (Kenton & Toutanova| 2019). Inspired by this method, analyze the effects of individual neurons in vision Transformer model contribution to the model prediction, with most influential neurons marked "knowledge neurons". The details about the implementation of knowledge attribution can be found in Appendix B Note that, it focuses solely on individual knowledge neurons, overlooking the joint effects of the common neurons.
Neuron Distribution for VIT-8-16
Percentage
20%
0%
12
Layer
Distribution MAE-B-16
90%
80%
70%
30%
20%
10%
0%
1
2
3
10
31
12
Layer
Figure 2: The distribution of knowledge neu- rons two different pretrained vision Trans- former models. It can be noticed that com- paring ViT-B-16 (Dosovitskiy et al. (021I) and MAE-B-16 (He et al.) (022). their neu- ron attribution show completely opposite dis- tributions across layers.
By applying the method to two types of vision Trans- former models using different pretrain paradigms, supervised ViT ViT-B--16 ( (Dosovitskiy al. 202T)
and selr-supe Masked AutoEncoder IMAE-B-16 (He al. 2022). have derived unex-

---

Published as a conference paper at ICLR 2025
pected discovery, illustrated in Figure 2 Despite both models being pretrained and finetuned on the same dataset (ImageNet (Deng et al. 2009)) with almost identical model structures, ie., 12 Transformer layers with the same hidden size, and utilizing the same imput process, i.c., 16-size patching on the input, we observe that the distributions of individual knowledge newrons across the models are almost inverted. Moreover, numerous studies have demonstrated that, in vision models, information is progressively aggregated from low-level visual patterns to high-level visual concepts as the network depth increases (Bau al. 2017 Zeiler & Fergus 2014). This implies that, to accurately capture high-level concepts in an image, each layer should include at least one sig- nificant neuron to facilitate the flow of information. However, current attribution methods may fall short in fully capturing this requirement. Consequently, there is a pressing need for a new method that can provide deeper insights into the transmission of information by neurons within the model. It is true that the attribution of individual neurons using integrated gradients, as shown in Figure yields clusters of neurons that are most sensitive to image information in the visual model, but this does not capture the process of how visual information is transmitted and processed through neural pathways. Just as that in the human visual system, nerve cells close to the retina and visual cortex must be the most sensitive, but the visual neural pathways between the two equally important and worthy of study. Thus, a joint measurement of neuron path within the model is demanding.
3.2 JOINT ATTRIBUTION SCORE
To locate the most influential neurons, we first propose a joint influence measurement of set of dif- ferent neurons. Motivated by the concept of information flow (Lu et al. 2021), we introduce a novel integrated gradient- gradient-based method compute the joint attribution of all selected neurons, across dif- ferent layers in vision Transformer models. Consider a vision Transformer model F consisting of layers. Given an input-labet pair < x, y 2, where denoting the output result utilizing the neurons within the first N layers
Id and y is the related label, we begin by
(w
N oh -.g. w plyla, wi, vit, w n ,i WiN
. wix):
W
where wa represents the i-th intermediate neuron in the FFN module of the I-th layer ,wihh S I S N S and wf is the assigned value of wt. To calculate the contribution to the model output of the selected neurons jointly, we incrementally sum the values of {w!, i " w } from 0 to w. } using a control factor a E (0, 1). This process yields
their original values fw: w ..., "
differentiations with respect to each neuron in the neuron path. By aggregating these differentiations and integrating with respect to a. we obtain a score that represents the joint attribution of the neuron in the model. Formally, the definition of this joint attribution score is given as follow.
Definition (Joint Attribution Score) Given a model Rd + R containing laaers, whose
output with inpul is defined as Fa, with a set of neuron {w!, wiy..., ... w }, N < L, a Joint Attribution Score is defined as
N OF, (aww ow aw iN da.
JAS(w
W
Ouc.
1=1
For the convenience of computation, we use the Riemann approximation to estimate the continuous integral as follows,
( ..... w
m
N
OF
Fo
L m
m
12
ETN
(3)
m71
Ow..
j
where is the sampling step.
In Eq. (2), we extend and calculate integrated gradient, the cumulative gradient of the mea- sure of interest (classification output) F, to the path of the target component (the neuron set {w), w ..., wiy }) ranging from zero effect (a = 0) to full effect (a = 1), given the model input x. This assesses the full contribution of the target neuron set.
With Definition I stated above, we propose a novel neuron-based model analysis method Neuron Path, trying to find path consisting at least one important neuron each layer tracing from input to model output to elucidate their significance and role in model inference. The formal definition is as below.
5

---

Published as a conference paper at ICLR 2025
Definition 2 (Neuron Path) Civen model F: R + R containing L layers, with an input x, and a user-defined criterion St)), a neuron path Pa is defined as follow.
P== = fw", w² w²....,u"" "...,",} w²}
(4)
that maximizes the S(Pz), where w E 2 J.} stands for the selected neuron within layer l.
S()) utilizes a user-defined score evaluate the behavior of the network with respect to the input of interest When the score function is defined as the maximum activation, the path Par comprises the neurons with the highest activation each layer, as detailed in Appendix C.2 When Pattern Influence (Lu et al.) 202I) is applied, we obtain the path Pa, that maximizes the corresponding influence score, as detailed in Appendix C.3
For our method, we use the JAS in Eq.
3.3 LOCATING INFLUENTIAL NEURON PATH
To identify the neuron path within the target model FF that maximizes the JAS in Definition I we employ a layer-progressive neuron locating algorithm, to iteratively select neurons layer by layer, gradually constructing the path that maximizes JAS. Utilizing the definition of the JAS provided above, we formulate the algorithm as follows. For a vision Transformer model comprising L layers, we enumerate each layer and add neurons to the path to maximize JAS the current depth. The detailed procedure of our algorithm is outlined in Algorithm | found in Appendix
Time complexity analysis can be
Algorithm Layer-progressive Neuron Locating Algorithm
Input: Model F with L layers, input sample
Output: neuron path P
Initialization: P 2 1
while < L do
W is the set of neurons in layer I of F; Score 0,p None for w E W do if Score JAS(PP .w) then
Score = JAS(P P, w
P PU {p): | 1+1
Comparing with existing methods that focus on individual neurons or input features DDai et al. 2022 Dalvi et al. 2019 Durrani al. 2020; Huang et al. 2023); Sundararajan al.. 2017), ap- proach considers the collective contribution of selected neurons across different layers in the model. By computing and maximizing the joint attribution scores, we aim to identify the most influen- tial neuron path for the given input, which processes and conveys the most crucial information as flow through the model layers. Some related works (Lu et al. 2020 (02T) on Transformer-basee language models also investigate neuron pathways, but they overlook the collective influence of neu- rons across layers, resulting in suboptimal outcomes, as illustrated in the experiment. In contrast, our approach offers several advantages. By considering the collective contribution of neurons across layers and employing layer-prue progr-srogeessive neuron locating algorithm, we can effectively identify neu- ron paths that maximize the crucial information flow. Furthermore, our method provides holistic view of the model's ss information transmission during inference, offering valuable insights into its inner workings.
EXPERIMENTS
In this section, we present both quantitative and qualitative experiments to analyze the proposed method from multiple perspectives. In Section 4.2 we validate the neuron paths discovered by our approach by comparing them with baseline methods through quantitative analysis and intervention experiments. In Section 4.3 we provide in-depth statisties the discovered neuron paths and reveal their clustering patterns with respect to image classes. Finally, in Section 4.4, we perform network pruming other neurons while preserving the identified neuron paths, offering insightful observations on classification performance and highlighting potential applications of our method. 'Te code will be released upon the acceptance of this paper.

---

Published as a conference paper at ICLR 2025
4.1 IMPLEMENTATION DETAILS
Our experiments are conducted on ViT (Dosovitskiy et al., 2021) and MAE (He et al. 2022). two widely used vision Transformer models but with different pretraining methods. For our following experiments, we will mainly utilize 3 ViT settings and 1 MAE setting: ViT-B-16, ViT-B-32, ViT- L-32 and MAE-B-16 as the target models. Details of these models are in Appendix C.I
These models are all trained for the image classification task, they are all pretrained ImageNe12 and finetuned on ImageNetIk k (Deng et al. 2009), whose validation set shall be the probing dataset for the experiments below. As for the calculation of JAS, set the sampling step 20 in Eq. for the following experiments.
4.2 QUANTITATIVE COMPARISON
In this section, we evaluate different neuron explainability methods and perform neuron interven- tion experiments according to the results different methods. We start by presenting the different methods of comparison, the first one is commonly utilized in literature (Dhamdhere et al. [209 Dai et al. 2022) and the second one is presented in previous work (Lu et al. 2021). Details of the first two baseline methods implementation can be found in Appendix and C.3
Activation represents the method that locates the neuron with the largest activation at each layer.
Influence Pattern follows the setup of the previous work (Lu et (202), which utilizes the input to build path integral and find the neuron with largest integrated gradient layer by layer.
Neuron Path (ours) stands for the neurons sampled by the neuron localization method have proposed above Section 3.3
Experimental settings. We assess the significance of neurons identified by different methods to evaluate the effectiveness of the compared approaches. The first experiment measures the joint influence of the neurons given different inputs to the model, as defined by the Joint Attribution Score (JAS) Section 3.2
Additionally, we will conduct more in-depth comparison of the three methods through manipula- tion experiments by intervening in the identified neurons. These experiments involve two operations on the selected neurons: removal (zeroing out) and enhancement (doubling) the values of the selected neurons. For each image input, neurons the target model are identified using the three approaches, and manipulation experiments puts after these operations, we can observe in finer detail the impact of these neurons on the model's classification ability. Two metrics, Removal Accuracy Deviation and Enhancement Accuracy De- viation, are used to quantify the experimental results of the operations. These metrics measure the change in model accuracy after the corresponding operation. Further details about these metrics are provided in Appendix C.5] Moreover, we statistically analyze the deviations of the predicted probability regarding the ground-true -tuth image class after performing the two operations. Additional details regarding the probability deviation metrics can be found in Appendix C.4 The quantitative
conducted. By analyzing the changes in model out-
results are presented in Table T. and the statistical analyses are presented in Figure
Metries
Methods
Target Models
VIT-B-16
VIT-B-32
ViT-L-32
MAE-B-16
Activation
.0.0044
0.0288
-0.0065
0.0013
Joint Attribution Score
Influence Pattern
0.0412
0.0841
0.1227
0.0030
Neuron Path (ours)
0.4078
0.6610
1.0086
0.0095
Activation
0.07%
-0.15%
0.16%
-2.80%
Removal Accuracy Deviation L
Influence Pattern
-0.50%
-1.24%
-1.41%
-15.67%
Neuron Path (ours)
-2.40 %
-3.11 %
-5.28%
-26.50% %
Activation
-0.33%
-0.45%
-0.86%
-1.00%
Enhancement Accuracy Deviation
Influence Pattern
0.46%
0.83%
L.12%
4.15%
Neuron (ours)
3.06%
5.02%
7.28%
Table 1: Three different metrics were used measure the neurons obtained using three different methods. f (L) means that the higher (lower), the better. The best results are in bold font.
7

---

Published as a conference paper at ICLR 2025
Probabiltty Devibtion after Zoroigg OVE
VIT-8_16
VIT-L
MAE-B-16
40%
40%
40%
100%
Oura
Oart
20%
50%
0%
0%
015
-20%
20%
-00%
50%
-00%
-100%
Activation
Influence Fallern
Ours
Activation Influence
Activation Influenee
Activation Influence
Pattern
Pattem
Fattern
100%%
300%
100%
900%
Infunnce Natten
Irfuence feber
Infuecee Fatten
700%
sbbbbbbbtt Devistion
6.%%
60%
60%
500%
40%
40%
40%
0%%
0%
0%
Activation:
Influence
Durs
Actioation
Influence
Activation
Infixence
Durs
Actiowtion Influence Durs
Pattern
Pattern
Pattern
Pattern
Figure 3: The relative deviation in the model's predicted probability of the ground -runthhruthhhu label when the value of neurons selected by different methods is either removed (zeroed out) enhanced (dou- bled).
Finding 1: The Neuron Path method more effectively identifies the influential neurons within the model. In Table our proposed Neuron Path method demonstrates superior ability to identify neurons that have more significant impact on model outputs, as indicated by larger JAS values. This is expected, as our method directly maximizes the joint contribution of the selected neurons. We further quantify the impacts of the neurons discovered by the Neuron Path method and other compared methods by measuring the deviation in model accuracy of classification when manipulat- ing the value of neurons identified by different methods. As shown in the lower part of Table [ the removal and enhancement of neurons identified by our Neuron Path method result in significantly larger drops and improvements in model performance, illustrating the effectiveness.
The discovered neuron paths play vital role in model inference Above results im- ply that the intrinsic neurons of the model significantly affects prediction performance. In order to explore in depth how the model's 's output probability is affected by the internal neurons, we count the relative deviation in prediction probability caused by manipulating the values of the discovered neu- rons. As illustrated in Figure our method results in significantly larger positive contributions when enhancing neuron values, and more substantial negative effects when removing neurons comparing other two methods. It indicates that neurons selected by our methods can significantly affect the model prediction, which demonstrates the crucial role these neurons play in model inference.
Finding 2:
4.3 CLASS-LEVEL ANALYSIS
In this subsection, we aim to understand the internal workings of vision model predictions and determine if neuron paths reflect the intrinsic mechanisms of the model architecture that critically contribute to predictions. To this end, we analyze the common characteristics of neuron paths at the class level. Interestingly, this analysis reveals clustering properties within the same class and semantic relationships across different classes.
Intra-class analysis on neuron paths. We aggregated the neuron paths for images of the same category and compiled statistics on the frequency of each neuron selected by our proposed method. Figure shows the frequency distribution of each neuron (vertical axis) across different layers (hor- izontal axis) in the discovered neuron paths for four different classes. The width of the violin plot indicates the frequency with which a neuron is selected. The most frequently used neuron at each layer is connected with dotted line. Due to the restriction that category only contains limited number of images, the length of the violin plot is shortened if the the ends of neuron index are not sampled. See Appendix D.I for more details and results.
Finding 3: Some certain neurons contribute more at each layer to specific classes. From Fig- 4 it is evident that at most layers, certain neurons are selected with significantly higher frequency
8

---

Published as a conference paper at ICLR 2025
Class: albatross
Class: Weimaraner
Class: Border collie
Class: Eskimo dog
Neuran Index
Neuron Index
Neuran Index
Neuron Index
Layer
Layer
Layer
Layer
Figure 4: The frequency of each neuron at each layer occurred in the discovered neuron paths.
in neuron paths corresponding to images of the same class. These highly selective neurons exhibit clear clustering properties. This phenomenon reveals the intrinsic functioning of each layer in the vision model (ViT-B-16 (Dosovitskiy et al. 2021) in this experiment) in responding to inputs from specific classes.
Inter-class analysis on neuron paths. We extend our analysis to the inter-class level by examining the above stated neuron frequency distribution, which reflects neuron utilization in the neuron paths, to gain insights into the intrinsic working mechanisms of the vision model. We gather neuron uti- lization data - specifically, the frequency statistics of each neuron in the discovered paths - and construct neuron utilization matrix Me IR LxL where L is the number of layers and n is the number neurons per layer, for each class с E C. We then calculate the cosine similarity between Mc, and Meg for two different classes c1 C2 C. More details are in Appendix D.2
Finding 4: Neuron paths reveals semantic similarity. The matrix Mg reflects the neuron uti- lization of the neuron paths for image class highlighting the model's internal working properties for that class. As illustrated in Figure 5 semantically similar image classes exhibit similar neuron utilization patterns. This indicates that the discovered neuron paths tend to carry the semantic infor- mation corresponding to the category, suggesting that the vision model utilizes specific functional components in response inputs from particular classes. It further implies that other than directly using visualization tools (Erhan et al. 2009; Selvaraju et al., 2017 Zhou et al., (016) to subjec- tively illustrate the focus of model's each part, our method tends localize the concept neurons straightaway. More examples can be found in Appendix D.3
Sim:
Sim: 0.372
Reference
Sim:
0.144
Top similarity
Similarity
Nile
lizard
Kamodo
Kakatoe galerita
mileometer
Sim:
Similanity
Similarity
hornbill
drake
quall
Iron
Bemaar
Sim: 0.178
0.188
Bottom
Similarity
Norwich terrier
Irish
Yorkshire tenter
fireguard
Figure 5: Examples of category similarity analysis. Using ViT-B-16 as target model, we randomly select three categories and calculate the similarity with others using the neuron utilization matrices and sample the top 5% and bottom 5% similar items. Through visualization we can see that cate- gories with high (low) neuron path similarity tend to be also high (low) in semantic similarity.
4.4 MULTI-NEURON MODEL PRUNING
In this section, explore potential applications of neuron path method in model compression. Drawing inspiration from the Mixture-of Depths approach (Raposo et al. 2024). we hypothesize that neurons in Vision Transformer models may exhibit redundancy. We propose that by retaining only

---

Published as a conference paper at ICLR 2025
the most significant neuron paths, the model can sustain comparable performance, even when other neurons are randomly masked. To test this, we designed the following experiment. For a selected model, we retain the top E {1, 5, 10, 30, 50) most influential neurons per layer within the neuron paths discovered by our Neuron Path method. Following the statistical procedure outlined in Section 4.3] we identified neuron paths for each category using the 80% of the image data, and transfer the results and conduct the pruning experiment on the rest 20% image data, establishing a generalization setting. During the pruning phase, we randomly zero out p {10%, 30,, 50% 100% of the neu- rons, excluding the selected ones, to measure the effect on classification accuracy. The experimental results are presented in 6 with experiment details and comparisons in Appendix(E)
Finding 5: Neurons within Vision Transformer models are largely redundant, with only sparse subset significantly impacting model performance. Our results in Figure ( by retaining the critical neurons identified through our method, the model's performance remains robust, even when a significant percentage of other neurons are zeroed out. This implies that aside from a few key neurons, most neurons either redundant or may even negatively affect the model's performance. Moreover, the results show that the number of important neurons is limited; the top five neurons deliver the best performance during pruning, while increasing the number of retained neurons leads to a decline in performance. This experiment also demonstrates the strong general- ization ability of our approach. Also combining the results of important neuron value intervention in Section 4.2 we can conclude that our method can locate the most influential neurons conveying critical information flows across layers within the model.
indicate that
Pruning 10% other ncuTons.
Pruning 30% other neurons.
Pruning 50% other nourens.
Prunio 100% other nourons. basdiie
VIT418 men ecrureyy
... Sebnlse
baselie
70%
Bose Fn
65%
tep1
5np5
to
tap50
tap1
Eop5
hont0
hop30 30 top50
hopt
hop5
top15
Ina hoa50
tep1
1no5 topto
1o toat3
VITA32 mene accurry
54%
54%
Dackke
54%
bood
54%
baidin
52%
550%
50%
50%
50%
45%
46%
45%
46%
1u25
tap10
to050
tap50
top5
tap20
top50
tap1
tap5 lap10
tos30 to250
1o05 tapt0 top30 tos50 Ridir rearoes
Notan recatr
Kelam nearore
Figure 6: Model performance after different proportions of neuron pruning using V6 and ViT- B-32. The dotted line represent the original performance of the used model, the y-axis represent the mean accuracy of the edited model and the x-axis represent the number of neuron in each layer we preserved. Baseline means the performance of the original model.
5 CONCLUSION AND FUTURE WORKS
In this study, we aim to unveil the intrinsic working mechanisms of vision Transformer models by locating and analyzing the crucial neurons at each layer that have the most significant impact on model inference. We introduce novel approach called Neuron Path, guided by newly proposed joint neuron attribution measure, to progressively identify the neurons that play key roles in informa- tion processing and transmission through model layers. Our series of analytical experiments reveal not only the importance of the discovered neuron paths in model inference but also illustrate some valuable insights into the internal workings of vision Transformers. Our work contributes to a more nuanced understanding of how these models manage visual information, expecting to advance the field of model explainability and facilitate future research on safely deploying vision models.
However, our method has certain limitations, which point to directions for future work. First, ex- panding the analysis beyond neurons in the FFN components to encompass the entire Transformer block could provide deeper insights into Vision Transformers. Second, exploring more downstream tasks like extending our approach from discriminative tasks to the segmentation and generative paradigm of vision models offers promising avenue for future research.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This research was supported in part by National Natural Science Foundation of China (Grant No. 62406193). The authors are also grateful for the support from Shanghai Frontiers Science Cen- ter of cennan-eenteed Artificial Intelligence, MoE Key Lab of Intelligent Perception and Human- Machine Collaboration, and HPC Platform of ShanghaiTech University.
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision. pp. 9650-9660, 2021.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-bases based visual explanations for deep convolutional networks. In 2018 IEFE winter conference on applications of computer vision (WACV), pp. 839-847. IEFE, 2018.
Arnay Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang- Ting Cheng, and Eric P Xing. Vi- sion transformer slimming: Multi-dimension searching in continuous optimization space. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pallern Recognition, pp. 4931- 4941, 2022.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 782-791, 2021.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings the the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/1/22222acl-bbl
URL https: https:///claatthoooyyy org/
2022. a0122accl-ooo..5.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6309-6317, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-254255 255. leee, 2009.
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron. In Interna- tional Conference Learning Representations, 2019. URL /ttptpa://eeeeeeeeeeee... net.
SorumPid-Sy1Ko00ckk
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview.ro wwtp://nee.ooooodddtt.ee.
CNTTY
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neu- rons in pre-trained language models. In Bonnie Webber, Trevor Cohn. Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), ),, pp. 4865-4880, Online, November 2020. Association for Computational Linguis- URL ht tttss:: http///aaaannnnll...oooco wrgp/ooo.oor//20....
tics. doi: 10.18653/v
eemn p ma 395

---

Published as a conference paper at ICLR 2025
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing the hidden layers of deep networks. In Proceedings of the International Conference on Machine Learning (ICML) 2009.
Yimin Fan, Fahim Dalvi, Nadir Durrani, and Hassan Sajjad. Evaluating neuron interpretation meth- ods of nlp models. Advances in Neural Information Processing Systems, 36, 2024.
Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale. arXiv preprint urXiv:2305.19911 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer fordard layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing. pp. 5484-5495, Online and Punta Cana, Dominican Republic, November Associatin for Computational Linguisties. doi: 10.18653/v http//221..mmlplpmmmmemmmm main.446. URL wwtp:y....gg//20 2021 emnlp weallp-mmiiin ma main. 446
Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, An- drew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? visual explo- ration. arXiv preprint arXiv:2212.0672., 2022.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Ad- vances in neural information processing systems, 33:5922-5932, 2020.
Leon Guertler, M Ganesh Kumar, Anh Tuan Luu, and Cheston Tan. Tellme what you see: Using IIms to explain neurons in vision models. 2023.
Mohit Gupta, Ashley Ireland, and Bruno Bordoni. Neuroanatomy, visual pathway. 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2021.
Jing Huang, Atticus Geiger, Karel D' O'Osttelinckk, Zhengxuan Wu, and Christopher Potts. Rigor- ously assessing natural language explanations of neurons. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 317-331, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653//1/2023. blackboxnlp- 1 .44. URLhttps: http::://aaaaaataaaaaaaae wrg. htwp:////...cccccoooooooc....... http:///oo.aaaaaoooooo/oooo///aa
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint
2015.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAAC--LLT, pp. 4171- 4186, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Yongchan Kwon and James Y Zou. Weightedshap: analyzing and improving shapley based feature attributions. Advances in Neural Information Processing Systems, 35:34363-34376, 2022.
12

---

Published as a conference paper at ICLR 2025
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 681-691, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 10.11653/11/N11-10006 1082. URL htttps://aclanthhloloo
org/NI Sxg/N16-100 N18-1082
Kaiji Lu, Piotr Mardziel, Klas Leino, Matt Fredrikson, and Anupam Datta. Influence paths for characterizing subject-verb number agreement in LSTM language models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 4748-4757, Online, July 2020. As- URL https:
sociation for Computational Linguisties. doi: 10. 1/200 orq. ottpa//oooooooo...22200.... acl- 4330
7/aclanthologge
Kaiji Lu, Zifan Wang, Piotr Mardziel, and Anupam Datta. Influence patterns for explaining infor- mation flow in bert. Advances in Neural Information Processing Systems, 34:4461-4474, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2021.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuad Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296- 2021.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Steerable search for interpretable machine learning. Nature Communications, 7:12216, 2016.
Tuomas Oikarinen and Tsui-Wer Weng. CLIP-dissect: Automatic description of neuron representa- tions in deep vision networks. In The Eleventh International Conference Learning Represen- tations, 2023, URL htttps://openreviert net/ www.rmmmi--Piiiwwi......
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learn ing robust visual features without supervision. Transactions Machine Learning Research, 2023.
Konstantinos Panousis and Sotirios Chatzis. Discover: Making vision networks interpretable via competition and dissection. Advances Neural Information Processing Systems, 36, 2024.
Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pp. 2071-2081, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based lan- guage models. arXiv preprint arXiv:2404..02258 2024.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626. 2017.
Caihua Shan, Yifei Shen, Yao Zhang, Xiang Li, and Dongsheng Li. Reinforcement learning hanced explainer for graph neural networks. Advances in Neural Information Processing Systems, 14-22523-225333 2021.
13

---

Published as a conference paper at ICLR 2025
Daniel Smilkov, Sam Greydanus, Jonathon J. Shlens, et al. Smoothgrad: Removing noise by adding noise. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jesse Vig. A multiscale visualization attention the transformer model. Marta R. Costa- jussà and Enrique Alfonseca (eds.), Proceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics: System Demonstrations, pp. 37-42, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 111118/61/111930000 URL https: //aclanthology org/P19 P
Chenyu You, Haocheng Dai, Yifei Min, Jasject S Sekhon, Sarang Joshi, and James Duncan. The silent majority: Demystifying memorization effect the presence of spurious correlations. arXiv preprint arXiv:2501. 00001,, 2025.
Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. On explainability of graph neural networks via subgraph explorations. In International conference on machine learning, pp. 12241- 12252. PMLR, 2021.
Matthew D) Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 818-833. Springer, 2014.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2921-2929, 2016.
Roland Zimmermann, Thomas Klein, and Wieland Brendel. Scale alone does not improve mech- anistic interpretability in vision models. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
A LAYER-PROGRESSIVE NEURON LOCATING ALGORITHM ANALYSIS
Here the estimation of the computing complexity of our proposed searching algorithm. We firstly define some notations of the model:
L: Number of model layers
nt Number of neuron in each layer
m: Sampling step
d: Feature dimension
T: Transformer token number
In each layer, will firstly iterate each neuron, whose complexity is O(n), then for the calculation of JAS as Definition ! the complexity forward pass is O(T ((T And for backward propagation, the complexity is also O(T ).). Since have sampling steps and neurons, the whole complexity (m n- ). and for the addition operation, the complexity is ((m.n). Therefore, for one layer, the computing complexity is
O(N)+ O(m T dd) O(m (mmmn))=-)))))) x Om n. d
And for L layers, the whole complexity is
OLL m n d²).
In the implementation, we can parallelly compute the integrated gradient, the detail of hardware and run time is shown in Appendix C
15

---

Published as a conference paper at ICLR 2025
B PRELIMINARY EXPERIMENT METHOD
Given a pretrained model F with sample input x, can caleulate the attribution score for the i-th neuron in the 1-th layer wA, which is called Knowledge Attribution Dai et al. 2022), by the following formula
FF, (aw)) Ow!
A = w.
da
(5)
where w. is the original value of the i-th neuron in layer l. For the sake of easy calculation, we use the Riemann approximation instead
0F, w!)
Aur(w)))
m
(6)
Jw!
where m stands for the sampling step, and for this experiment, we set m = 20. For each sample input, compute the attribution of each neuron and select the 5 neurons with the largest attribution as the significant neurons for that sample.
C QUANTITATIVE EXPERIMENT DETAILS
All the experiments are NVIDIA A40 GPUs with batch size equals 10 and sampling step m equals to 20, using ImageNetlk validation set. The estimated time for an experiment is about 10-20 hours based the size of target model.
C.I TARGET MODEL SETTING DETAILS
ViT-B-16 stand for containing 12 Transformer blocks with patch size equals to 16; ViT-B-32 stand for containing 12 Transformer blocks with patch size equals to 32; ViT-L-32 stand for containing 24 Transformer blocks with patch size equals to 32 and MAE-B B-A---166 stand for containing 12 Transformer blocks with patch size equals to 16. For ViT-B-16, VIT-B-32, MAE-B-IK 1-E--116. 16, their hidden sizes all 768 and FFN inner sizes are all 3072, which stands for 3072 different neurons per layer; and for ViT- I--2,, its hidden size is 1024 and FFN inner size is 4096, which stands for 4096 different neurons per layer.
C.2 ACTIVATION
The activation method represents path sampling technique that locates the neuron with the largest original activation value each layer. The key idea trace the most activated neurons across layers, assuming that the most significant information flow is through these highly activated neurons. To find the Neuron Path P. (w-, w² ..., w²) in Definition 2 for an input sample æ, we first initialize the path Pa as empty set and then start layer-wise neuron selection. For each layer (
from FFN vision Transformer model, we compute the original values for all neurons the layer, which is denoted
A' (w.),
where W. is the original value of the i-th neuron in layer and n the number of neurons per layer. To identify the neuron in each layer, we pick the neuron with the largest activation, and add this to the path and finally we obtain
Pa = (n0 = arg max (A)). E (1,,,..,...
After iterating through all layers, the path contains the most activated neurons from each layer, representing the route with the highest neuron activation. The resulting path highlights the neurons with the highest activation at each layer, suggesting a direct route of significant information flow through the network.
16

---

Published as a conference paper at ICLR 2025
C.3 INFLUENCE PATTERN
The Influence Pattern method, based on (Lu et al. 2021]. utilizes the input to build path integrals and identifies the neuron with the largest integrated gradient each layer. This method focuses on the influence of input features neuron activations, tracing the path that maximizes the integrated gradient layer by layer. Their core algorithm greedy search with key metrie Pattern Influence. With target model F with L layers, we present
! II
Ow' (x' + alx - x')) da a(xx=xx) 1=1 Ow Ow a(x x'))
Px),
(9)
where x' is the all zero base input, (-) is the activation value of selected neuron of the I-th layer with respect to the input, Pa the path containing the neurons across all the layers. With the definition above we can now form the greedy search-based algorithm as follows.
Algorithm 2 Greedy Search-Based Influence Pattern Algorithm
Input: Model F with L layers, input sample x
Output: neuron path P Initialization: p 23,1 =
while < L do
W is the set of neurons in layer / of F; Score 0,p None for w E W do
if Score I(x, P {w}) then
Score I(x, P [w)):
p PU (p): 1= 1+1
After iterating all the layers in model F, we will have neuron path sampled based on Influence Pattern.
C.4 PROBABILITY DEVIATION
Suppose we have a model F Rd -> R, a neuron path Pa (1,2. .... L} stands for the selected neuron in layer l. Given a input pair < 24, y can have the output probability as
(w w² .u?.... w²) where w e
Pa. Fa, (w 2 wh) = p(un w² = w/ 2
w a
(10)
We conduct manipulation experiments by manipulating the intermediate neurons within the neuron path, resulting manipulated intermediate neurons of, The output probability with the manipulated intermediate neurons and the input sample x; then
P. = Fz, (w) w² ,..., w
(1I)
Then, the Probability Deviation with the input < Fss > pair APa./P,, defined as a ratio
AP
Pr.
Pa.
(12)
Pr.
P
The average Probability Deviation of the dataset is defined the average of the individual proba- bility deviations
N
AP
A P
(13)
averaage
N
P
1=1
17

---

Published as a conference paper at ICLR 2025
The median Probability Deviation of the dataset is defined as the median of the individual probability deviations
APzz Ps.
(It aeu
AP median median
(14)
C.5 ACCURACY DEVIATION
Suppose we have model F R- + R. neuron path Pa = {w", =² ..., w"} where w I E (1. L} stands for the neuron in layer 1. Given a dataset DD represented by < Tt. > pairs, where 2; denotes the input of the i-th sample and represents its target label. The accuracy, denoted by Acc, defined as
D Σ
Acc
I = vi))
(15)
[D]
where i defined as
01 argmax (Pa:)
(16)
In this context, Pa: represents the output probability of the model. After manipulating the interme- diate neurons within the neuron path as demonstrated in Appendix C.4. the output probability with the input sample v; become Pa.. and T changes to
i = argmax (Pa.)
(17)
The accuracy also changes to
DD
Acc
[D] Σ
T(ū: = vi)
(18)
Then, the Accuracy Deviation AAce defined as
AAcc Acc Acc
(19)
18

---

Published as a conference paper at ICLR 2025
D MORE EXAMPLES AND IMPLEMENTATION DETAILS ABOUT CLASS-LEVEL ANALYSIS
D.1 MORE FREQUENCY PLOT VISUALIZATIONS
In this section, we show more examples about the statistic visualization for four different models. In figures some violin plot may be shorter than others, this due to the fact that, constrained by the limited number of image samples within category, the length of the violin plot is shortened if the neurons the ends of index are never selected the neuron paths.
spiny
Neunn Index
Newon Undax
10
11
12
10
11
12
11
Class: sulphur butterfly
Class: hen
Class: keeshond
the
Neuron tn
Neuron Ino OX
10
11
12
Class: fountain
Class: wreck
Class: sarong
Nesmon irdox
Nauron Indxx
Class: indigo bunting
Class: Gila monster
Class: bluetick
Neurn Index
Neuron Index
Class: beagle
Class: marimba
Class: ballplayer
Neunn Intex
Neurin Index
Class: washer
Class: Indian cobra
Class: French buildog
Meuson irdox
Nouion Indox
Layye:
Layer
Class: slug
Class: barracouta
Class: mask
Mouron Indox
Neuron Inbox
Neuron Indox
10
12
10
11
12
12
Layer
Layer
Layer
Figure 7: More Path Statistic Visualization for ViT-B-16
19

---

Published as a conference paper at ICLR 2025
Class: bonnet
Class: limpkin
Class:
Neuron Neunn Index
Neuton Indoonnndoo Indoo
10
11
12
10
11
12
10
11
12
Layar
Layer
Layer
python
Neunon Intox
Nouson Indux
10
11
12
12
11 12
L2yar
agama
Neurn Indes
Nouron ind
10
10
11
12
11
Class: neck brace
Class: daisy
Cass: slug
Neuree Inoox
Nourin Ino
10
10
11
Cless: delmatian tian
Class: barometer
Class: pretzel
Neunn Irdex
Neuron Indax
Class: African grey
Class: Ameri can lobster
Class: Old English sheepdog
Meuro Index
Neuron Indax
Class: pretzel
Class: leatherback turtle
Class: triumphal arch
Neunn intex
Neurin Index
Layor
Figure 8: More Path Statistic Visualization for ViT-B-32
20

---

Published as a conference paper at ICLR 2025
Coss cridet
Class: souirel
Goss: Tendis
BETTANIAII 926252222224
shaepog
Class: green Lard
Class: Boston bull
Class ladybag
Läger
Laper
morkey
Clase digital watch
Clece liner
Classt darra
Liger
Mearon
Closs: mik beauty
Cass free frg
Clas: tntumpha srch
Leger
Leper
Leyer
Figure More Path Statistic Visualization for ViT-L-32
21

---

Published as a conference paper at ICLR 2025
Class:
Class: disk brake
Class:
Neuron Neunn Inder
Neuron Indoon
Nounn
10
11
12
6
10
11
12
10
11
12
Layar
Layer
Layer
loupe
Neumon Intex
Nouson Indux
10
11
12
10
12
10 11 12
L2yyar
stingray
Neunn Indes
Nouron Indax
10
11
10
11
12
11
Class: nedklace
Class: ballplayer
Class: Dandie Dinmont
Neune INCE
Nouran tne
10
10
11
12
Class: cellie
Class: dial telephone
Class: bessoen
Neuno ardex
Navron Indux
12
Class: partridge
Class: Japanese spaniel
Class: Sussex spariel
Neurn Indox
Neuron Indax
11
Class: bee eater
Class: hog
Class: tricycle
Neunn Entex
Neurin Index
Layor
Figure 10: More Path Statistic Visualization for MAE-B-16
22

---

Published as a conference paper at ICLR 2025
D.2 NEURON UTILIZATION MATRIX
For a category e E C. it will contain I different images, and in our setting, we use the Ima- geNetlk(Deng et al. 2009) validation set, I 50, [C| 1000. Each image will have its Neuron Path P. so that we can have a matrix f E R to count the number of times each neuron index occurs in each layer. where represents the number of layers and n stands for the number of neuron in each layer. By dividing the total number of neuron selections in each layer, we can define a neuron utilization matrix corresponding to category as
Mc E R I.xm
(20)
and for every pairs of neuron utilization matrix Map feg. we can calculate their cosine similarity through following approaches.
Mc.- Me,
(21)
M ll x M|M
since will have [C] [C] pairs, we can have similarity matrix S R/[xiel, where Si represents the cosine similarity between e; and Cj. that is 8i.j
23

---

Published as a conference paper at ICLR 2025
D.3
MORE SEMANTIC SIMILARITY VISUALIZATION
Sim: 0.559
Sim: 0.532
Reference
Sim: 0.212
Sim: 0.137
Bottom Similarity
Top Similarit
Beach
Pelican Sim: 0.448
Convertible
Bernard
Staffordshire
Bottom Similarity
Top Similarity
Water Ourel Sim: 0.456
Indigo Bunting Sim:0.387
Goldfinch
Tabby Sim: 0.156
Tiger cat Slm:: 0.129
Top Similarity
Similanity
Red-breasted Merganser Sim: 0.459
Microwave Owen
Sim: 0.584
Sim: 0.170
Bottom Similarity
Top Similarity
Staffordshire Sim: 0.150
Sim: 0.514
Sim: 0.499
Sim: 0.158
Bottom Similarity
Top Similarity
Chow Sim: Sim:0.372
Retriever
Shih-Tah
American robin Sim: 0.161
Letter Opener Sim: 0.127
Bottom Similarity
Top Similarity
Bald eagle Sim: 0.650
Albatross Sim: 0.591
Cocker Spaniel Sim: 0.190
Staffordshire Sim: 0.139
Bottom Similaitty
Top Similarity
Spaghetti Sim: 0.460
Granny
Bakery Sim: 0.194
Odometer Sim: 0,209
Sim:
Battom Similarity
Similarity
Maillot
Cloak
Hoopskirt
Horbontal Bar
Cocker Spaniel
Figure 11: More examples of category similarity for ViT-B-16
24

---

Published as a conference paper at ICLR 2025
Sim: 0.632
Sim: 0.617
Reference
Sim: 0.352
Sim: 0.272
Bottom Simillarity
Top Similarity
Snow Leopard
Egyptian
Bell Pepper Sim: 0.303
Conch Sim: 0.295
Sim: 0.739
Sim: 0.547
Bottom Similarity
Similarity
Chickadee
Oystercatcher Sim:0.624
Indigo bunting
Fountain Sim: 0.346
Chain Saw Sim: 0.307
Top Similarity
Similarity
1ITE
Border Terrier Sim: 0.515
Golden Retriever Sim: 0.512
Oxcard Sim: 0.299
Scoreboard Sim: 0.289
Greyhound
Boltom Similarity
Similarity
Lynx Sim: 0.521
Howler Monkay Sim: 0.511
Meerkat
Drop Sim: 0.286
Sim: 0.284
Bottom Similarity
Similarity
Cougar Sim: 0.562
Leopard Sim:0.549
Lion
Wing Sim: 0.307
Maze Sim: 0.255
Bottom Similarity
Similarity
Norfolk Terrier Sim: 0.636
Dandie Demont Sim: 0.628
Kerry Blue Terrier
Cliff Sim: 0.335
Fire Screen 0.319
Bottom Similarity
Top Similarity
Moving Van
Space Shuttle Sim: 0.296
Sim: 0.722
Sim:0.577
Sim: 0.323
Bottom Similarity
Top Similaritt
Yorkshire Terrier
Borzol
Norwich Terrier
Brambling
Space Shuttle
Figure 12: More examples of category similarity for ViT-B-32
25

---

Published as a conference paper at ICLR 2025
Sim: 0.365
Sim: 0.355
Reference
Sim: 0.175
Sim: 0.164
Bottom Similarity
Top Similarity
Redshank Sim: 0.314
Hen 0.306
Cock
Tape Player Sim:
Yellow Lady's Slipper
Bottom Similarity
Top Similarity
Flaminge Sim: 0.333
Red-backed Sandpiper
Ostrich
Cardoon Sim: 0.147
Dakky Sim: 0.137
Sim:0.280
Bottom Similaritt
Top Similarity
Indigo bunting Sim: 0.524
Goldfinch
jug Sim: 0.183
Sim: 0,583
0.156
Bottom
Top Similarity
Toy Terrier
Ridgeback
Scuba Diver Sim: 0.159
Sim: 0.371
Sim: 0.160
Bottom Similarity
Similanity
Basset Sim: 0.409
Pekinese Sim:0.389
Bluetick
King Snake
Broccoli
Sim:
Top Similarity
Similarity
white terrier Sim: 0.484
black-and-tan coonhound
Curly-coated Retriever
mashed potato Sim: 0.150
Bakery
Sim: 0.473
Sim: 0.125
Bottom Similarity
Top Similarity
Harvestman Sim: 0.447
Black Window Sim: D. 0.422
Scorpion
Sim: 0.198
Sim: 0.179
Bottom Similarity
Similarity
Horned Viper
Diamondback
Vina Snake
Bathtub
Ski Mask
Figure 13: More examples of category similarity for ViT-L-32
26

---

Published as a conference paper at ICLR 2025
Sim: 0.294
Sim: 0.257
Reference
Sim: 0.074
Sim: 0.074
Top Similarity
Similarity
bonnet
velvet
Apron
padd paddlewheel
Sim: 0.160
5im:0.130
Sim: 0.041
Sim: 0.037
Bottom Similarity
Top Similarity
SHe WHi Must
water bottle
bowl
coffee mug
banjo Sim: 0.054
sweatshirt Sim: 0.032
Bottom Similarity
Top Similarity
cauliflower Sim: 0.157
broccoli
head cabbage
refrigerator Sim: 0.033
Sim:
Bottom Similarity
Top Similanty
mushroom Sim: 0.108
alligator lizard 5im:0.106
mashed potato
banjo Sim:
fire screen 0.022
Tap Similarity
Similarity
paddle Sim: 0.114
scuba
schipparke Sim: 0.035
Sim: 0.120
Sim: 0,037
Bottom Similarity
Top Similariy
cliff dwelting Sim: 0.143
cliff Sim: 0.130
valley
cannon Sim: 0.049
acorn squash
Bottom Similarity
Top Similarity
soup bowl Sim: 0.095
water bottle 5im:0.093
whiskey jug
scoraboard Sim: 0.028
snowplow Sim: 0.022
Bottom Similarity
Top Similerity
jersey
Christmas stocking
Windsor tie
conch
brambling
Figure 14: More examples category similarity for MAE-B-16
27

---

Published as a conference paper at ICLR 2025
E IMPLEMENTATION DETAILS OF MODEL PRUNING
E. TOPK NEURONS PRESERVING ALGORITIM
The calculation of accuracy change is using the same implementation of Appendix C.5
Instead of keeping only the neurons with highest JAS along the neuron path, we will retain the neurons with the TopK highest scores. The detailed procedure of our TopK neurons preserving algorithm is outlined in Algorithm
Algorithm 3 Layer-progressive Neuron Locating Algorithm Maintaining TopK Neurons
Input: Model F with layers, input sample x, Topk= Output: neuron path P: Initialization: P.- 2, =
while S L do W is the set of neurons in layer I of F; Score = 2.P = None for w E W do
Score TopK K=t (JAS(Pt, w))
p - arg (
P. P, U (p)
E.2 PROBING AND TEST SET SPLITTING IN PRUNING EXPERIMENT
To evaluate the performance of the pruned models, we employ the ImageNetlk validation set (Deng eet al. (000)), which comprises 50 images per class. For each class, 80% (40 images per class) are randomly selected to identify the TopK neurons in each layer. Specifically, for each of these 40 images, we apply the TopK neuron preserving algorithm described in Algorithm 3 to compute the top neurons. The results are then aggregated across all 40 images, yielding set of 40 x neurons per class for each layer. By calculating the frequency of each neuron in this aggregated set, as detailed in Section 4.3 we determine the most frequently activated neuron paths. The neurons that appear most often are deemed the most critical for the respective class, and from this frequency distribution, the TopK neurons with the highest occurrence are selected as the representative top neurons for each layer. Subsequently, the pruning operation is executed using the remaining 20% of the dataset (10 images per class) inputs. For these images, neurons not included in the selected TopK set are zeroed out in each layer.
This validation approach ensures that the pruning operation is tailored to each class by retaining the most relevant neurons based on their frequency in the validation subset. Evaluating the pruned model on the remaining images allows us to assess the strategy's 's effectiveness in preserving classification accuracy while significantly reducing model complexity.
E.3 QUANTITATIVE COMPARISONS PRUNING METHOOSS
We tried using ViT-Slim (Chavan al. 2022
our compression baseline, here the results.
Method
Accuracy 1
ViT-Slim
9.96%
JAS-Prune 67.7%
Table 2: The model prediction accuracy after conducting pruning. 1 means that the higher, the better.
For fairness, we restrict ViT-Slim to focus solely on FFNs, mirroring approach. Notably, while ViT-Slim emphasizes module weights, our method targets individual neurons. For dataset config- uration, employ the validation set split consistent with that described above in Appendix E.2 Although ViT-Slim may be more efficient, its performance is constrained by training on limited dataset, limitation partly attributable its design.
28

---

Published as a conference paper at ICLR 2025
Furthermore, conduct ablation experiments on the pruning effect. We also perform additional comparisons by employing activation values as neurons, using the same configuration values in each FFN block are selected as neurons. In parallel. Here, Depth corresponds to the number of probing layers (i.e., the neuron path length), while Mask represents the percentage of neurons deleted, excluding the selected neurons.
path lengths investigate their impact
described in Section 4.2 wherein the TopK activation
Accuracy
Depth
Mask (%)
Activation topK=5
JAS topK=5
6
100
0.517991
0.521252
100
0.518137
0.527151
12
100
0.531976
0.540702
The model prediction accuracy after conducting pruning using different neurons and depth. f means that the higher, the better.
Table 3:
The results demonstrate that our method outperforms the activation baseline, and that the depth of the neuron path is a also significant.
29


Reviews:
[{"review_id": "k1JzXsfIvH", "reviewer": "Reviewer_TbDp", "summary": "This study focuses on uncovering the inner workings of vision Transformer models by examining critical neurons that influence model inference. They introduce the Neuron Path approach, which uses a new joint neuron attribution measure to identify key neurons in information flow across layers. The analytical experiments highlight the significance of these neuron paths for model inference and provide insights into the internal operations of vision Transformers, enhancing the field's understanding of visual information processing.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is relatively well-motivated as understanding vision transformer is a crucial issue. I also find the evaluations thorough. The strengths are as follows:\n\n1,The target issues of the paper are meaningful and worth exploring.\n2. The motivation is clear.\n3.The paper is easy to follow.", "weaknesses": "1. In section 4.4, neuron path method is used in model compression. It would be better if this method is compared to the the pruning method.\n2. This method aims at vision transformer. Can this method be utilized in NLP tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2u33pbmHKp", "reviewer": "Reviewer_jrVR", "summary": "This paper addresses identifying the neuron path within Vision Transformers (ViT) that comprises the most impactful neurons. Two methods are utilized to identify a neuron pathway. The proposed joint attribution score assesses the influence of the neuron pathway on the network's inference process. Another method referred to as layer-progressive neuron locating identifies the most significant neuron pathway for predicting the target image. The proposed method demonstrates its effectiveness in identifying neuron pathways. This paper presents a range of applications that leverage the proposed method.\n\nThank you for the authors' response. I think they answered my question adequately. Thus, I keep my score as a weak accept.", "questions": "How can we calculate the output by leveraging the neurons present in the initial N layers? Should we share the final classifier to compute the probability p of Eq 1?\n\nIt would be better to provide the meaning of Eq 2. Currently, there is no description for Eq 2. Also, the reviewer wonders about the rationale behind employing the sum of gradients as the proposed significant score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper presents innovative concepts for identifying influential neuron pathways. The proposed approach is capable of identifying the trajectory of neurons by analyzing the gradient magnitude. This method has the potential for application across diverse architectural frameworks.\n\nThis paper demonstrates the applicability of the proposed method across a range of scenarios. Initially, Figure 4 visualizes the pathways of neurons categorized by class. The visualization validates that various classes utilize unique neural pathways. Subsequently, Figure 5 demonstrates the class relationships derived from the discovered neuron pathways. This approach has the potential to facilitate the assessment of similarity between classes in forthcoming research.", "weaknesses": "This paper does not adequately present a thorough review of the existing literature regarding the proposed method. The reviewer, upon examining the current manuscript, encounters difficulty in clarifying the significance of the proposed method in relation to recent studies. The discussion should not merely enumerate related works; it should also engage in a comparative analysis of the proposed method to facilitate a clear understanding of its significance for the readers.\n\nThe manuscript requires enhancements in coherence for better clarity and flow. The abstract states, “showcasing that the found neuron paths have already preserved the model capability on downstream tasks.” However, there is a lack of research focused on downstream tasks.\n\nThe absence of crucial ablation studies for the proposed joint attribution score is a significant oversight in this paper. The proposed joint attribution score represents a sophisticated approach that necessitates multiple iterations of back-propagation. To establish its efficacy, it is essential to conduct a comparison with a more straightforward baseline method and demonstrate its superiority over that baseline. Nonetheless, Table 1 presents a simultaneous variation of search methods and scores, which complicates the identification of the specific component contributing to performance improvement. An appropriate ablation study is essential to address the increase in search complexity associated with the proposed method.\n\nThis paper does not include comparisons with the baseline method in Figures 5 and 6. The analysis studies are commendable; however, it is essential that they also incorporate quantitative metrics in relation to the baseline for a more comprehensive evaluation. The sole quantitative metric presented in this paper is presented in Table 1. The author needs to incorporate more extensive assessments to demonstrate the advantages of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WPeoEwS8Gd", "reviewer": "Reviewer_7H8A", "summary": "The paper introduces Neuron Path as a novel method for understanding Vision Transformers (ViT), focusing on the flow of information through the Feed-Forward Network (FFN) within the transformer encoder. Unlike methods that highlight input regions, Neuron Path examines influential neurons within the FFN across layers to reveal how information is processed. The authors propose the Joint Attribution Score (JAS), a gradient-based metric that measures the collective influence of neuron groups on the model’s output, along with a layer-by-layer selection method to identify impactful neurons, forming a path that traces critical information flow from input to output. Experiments show that Neuron Path effectively identifies neurons crucial for accurate predictions, with class-level analysis indicating that similar neurons activate for related classes, suggesting an internal organization of semantic information. Additionally, the authors find Neuron Path valuable for model compression, as retaining only essential neuron paths allows for nearly the same performance, revealing redundancy in Vision Transformers that could be leveraged for pruning and optimization.", "questions": "1. The proposed method focuses specifically on ViTs and is limited to FFNs. Could this approach be generalized to other model architectures or extended to vision tasks beyond image classification?\n\n2. The authors assert that JAS computation is computationally expensive. Are there possible optimizations or approximations that could be investigated to reduce the computational load and enhance the scalability of the Neuron Path method for larger vision transformer models? Furthermore, why not use zero-order derivatives as a substitute for partial derivatives in calculating JAS, as I believe they could achieve a comparable effect?\n\n3. The paper introduces JAS to measure joint neuron influence, yet a deeper theoretical understanding is needed. Could the authors clarify how maximizing JAS identifies crucial information pathways? Additionally, could they link JAS to information theory concepts like mutual information or information bottleneck to justify its use in measuring influential pathways?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces the innovative concept of \"Neuron Path,\" which identifies influential neuron paths specifically within the Feed-Forward Network (FFN) of vision transformers (ViTs). It also presents the Joint Attribution Score (JAS), a gradient-based metric that quantifies the collective influence of neuron groups across layers. Additionally, the Layer-Progressive Neuron Locating Algorithm is proposed to trace influential neuron paths efficiently.\n\n2. The research methodology is robust, with comparisons against established baselines such as \"Activation\" and \"Influence Pattern.\" The study includes comprehensive experiments, such as neuron intervention and model pruning, which provide strong empirical support for the effectiveness of the Neuron Path method.\n\n3. The paper is well-structured and clearly explains its concepts, with well-defined terms, illustrative figures, and organized sections that enhance readability and understanding.\n\n4. The study advances the interpretability of vision transformers by revealing insights into model behavior, potential for model compression, and implications for real-world applications. It fosters the development of more transparent and efficient vision models by uncovering redundancy that can be leveraged for pruning.", "weaknesses": "1. The current analysis is limited to neurons within the Feed-Forward Network (FFN) component of the Vision Transformer (ViT) encoder. Although FFNs are shown to play a critical role in encoding semantic concepts, excluding the self-attention mechanism from the Neuron Path analysis may lead to an incomplete understanding of information flow within the model, as it overlooks potential interactions between self-attention and FFN modules.\n\n2. The study is focused solely on image classification, a discriminative task, leaving the applicability of Neuron Path to other vision tasks unexplored.\n\n3. The paper does not explore theoretical connections between JAS and established interpretability frameworks, such as information theory or causal inference. Relating JAS to concepts like information bottlenecks or causal pathways could provide a stronger theoretical foundation, enhancing the rigor and interpretability of the findings.\n\n4. While the paper suggests a potential link between neuron paths and semantic information, it does not examine this relationship in depth. Conducting a detailed analysis of how neuron paths correlate with human-interpretable concepts could significantly enhance the model’s interpretability.\n\n5. The paper primarily focuses on identifying neuron paths and their impact on model performance but does not delve into the properties of these paths. Investigating the stability of neuron paths across varying inputs or their evolution during training could provide valuable insights into the model's robustness and behavior. However, due to the computational expense mentioned by the authors, such analyses may be challenging to conduct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aYDiSVSAsh", "reviewer": "Reviewer_BYaN", "summary": "The paper introduces a method to uncover influential neurons across layers in Vision Transformer (ViT) models. It identifies paths of significant neurons from input to output, aiming to enhance understanding of how these models process information internally. The method, called \"Neuron Path,\" employs a layer-progressive approach combined with a novel joint attribution score (JAS), designed to measure the collective influence of selected neurons on model predictions.\nThe paper proposes the Neuron Path method, which identifies key neuron paths that significantly contribute to the model’s decision-making process. This approach provides insights into information flow within the ViT model by tracing the most impactful neurons through each layer.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The Neuron Path method adopts a layer-progressive approach, which systematically identifies key neurons layer by layer. This method is a step toward understanding how information propagates through Vision Transformers (ViTs), aligning with the need for models that help illustrate the inner workings of transformers more clearly.\nThe joint attribution score provides a collective measure of neuron importance, enhancing typical neuron-based methods that often focus on single neurons. Although the score parallels existing joint attribution metrics, it offers a practical framework that reflects the relative importance of neurons throughout the model.", "weaknesses": "1. The paper introduces the \"Neuron Path\" method, which employs a layer-progressive approach to identify influential neurons across Vision Transformer (ViT) layers. However, this method significantly overlaps with established attribution techniques like [1， 2， 3]. The authors could clarify how their approach advances beyond these existing methods.\n\n[1] HINT: Hierarchical Neuron Concept Explainer, CVPR, 2022.\n[2] Neuron Shapley: Discovering the Responsible Neurons, NeurIPS, 2020.\n[3] WWW: A Unified Framework for Explaining What Where and Why of Neural Networks by Interpretation of Neuron Concepts, CVPR, 2024.\n\n\n2. While the paper underscores the structural complexity of ViTs as a barrier to interpretability, it narrowly investigates only one neuron type—the output of the first linear layer in each Transformer's FFN module. This limited focus raises questions about the motivation: effective explainability typically aims to enhance interpretability for either the model’s input-output relationship or its internal operations holistically. The choice to explore only specific layers restricts the scope, potentially overlooking valuable insights into other model layers or their contributions.\n\n3. Relying solely on joint attribution scores to assess neuron influence, though innovative, lacks breadth. Incorporating additional metrics—such as concept coherence or task-specific performance indicators—could provide a more comprehensive view of the Neuron Path approach’s effectiveness. Moreover, the experimental focus on ViTs for image classification lacks exploration into generalizability. Extending this analysis to other models and tasks, like object detection, could demonstrate the broader utility of the neuron paths identified here.\n\n4. The assumption that neurons in ViT FFNs are largely redundant warrants more rigorous testing. Studies show that FFN layers encode high-level visual features essential for robust model performance, and not all neurons in these layers may be extraneous. For example, Intriguing Properties of Vision Transformers (NeurIPS, 2021) and others illustrate that FFNs in ViTs play a significant role in preserving semantic information across layers, making them central to effective representation learning. Testing this redundancy hypothesis more thoroughly could yield valuable insights into FFN layer functionality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aqQnbz6kpY", "reviewer": "Reviewer_K2QS", "summary": "To discover the most influential neuron path, this paper propose a joint influence measure to assess the contribution of a set of neurons to the model outcome and a layer-progressive neuron locating approach. In the experiments section, this paper first conducted a quantitative comparison with two baseline methods: Activation & Influence Pattern; And then perform a  class-level analysis and pruning validation.", "questions": "The formula in `Sec.3.2`'s JAS may be a little confusing.\n1. The $w_{i_l}^{l}$ used in Formula(2)&(3) could be more simplified if the two $l$ are always same.\n2. A set of neuron $w_{i_l}^{l}$, original values $\\bar{w}\\_{i_l}^{l}$, assigned value $\\hat{w}_{i_l}^{l}$ could be more clearly defined before Formula(3).\n3. JAS in Formula(2) is difficult to understand, pseudo code like `Algorithm 1` can be helpful.\n\nI would be more than willing to reconsider my score based on the author's response to above `Weaknesses` & `Questions`.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and easily understood, especially for the `Introduction` and `Related Work` sections. \n\n2. What's more, the authors performed clear and comprehensive experiments (such as pruning for potential application) and demonstration.\n\n3. The method proposed in this paper is simple yet effective.", "weaknesses": "Apart from the given analysis in this paper, what interests me most is the different patterns between the supervised ViTs and self-supervised MAE, as demonstrated in `Fig.2` & `Tab.1`. At beginning, I personally attributed it to the lack of finetuning in self-supervised MAE, while the author pointed that this was also considered in `Line 218/328`. Although this almost **inverted distribution pattern** mentioned in `Sec 3.1's Preliminary Study` does not seem to have been resolved in `Tab.1`, the neuron path in MAE plays a much greater role than in supervised ViTs with lager *Accuracy Deviation*.\n\nPerhaps this issue is beyond the scope of this paper, but this phenomenon worth a targeted analysis or discussion (maybe in future work).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)