reviews: 
Comments: The paper introduces a method to uncover influential neurons across layers in Vision Transformer (ViT) models. It identifies paths of significant neurons from input to output, aiming to enhance understanding of how these models process information internally. The method, called "Neuron Path," employs a layer-progressive approach combined with a novel joint attribution score (JAS), designed to measure the collective influence of selected neurons on model predictions.
The paper proposes the Neuron Path method, which identifies key neuron paths that significantly contribute to the model’s decision-making process. This approach provides insights into information flow within the ViT model by tracing the most impactful neurons through each layer.
Strengths: The Neuron Path method adopts a layer-progressive approach, which systematically identifies key neurons layer by layer. This method is a step toward understanding how information propagates through Vision Transformers (ViTs), aligning with the need for models that help illustrate the inner workings of transformers more clearly.
The joint attribution score provides a collective measure of neuron importance, enhancing typical neuron-based methods that often focus on single neurons. Although the score parallels existing joint attribution metrics, it offers a practical framework that reflects the relative importance of neurons throughout the model.
Weaknesses: 1. The paper introduces the "Neuron Path" method, which employs a layer-progressive approach to identify influential neurons across Vision Transformer (ViT) layers. However, this method significantly overlaps with established attribution techniques like [1， 2， 3]. The authors could clarify how their approach advances beyond these existing methods.

[1] HINT: Hierarchical Neuron Concept Explainer, CVPR, 2022.
[2] Neuron Shapley: Discovering the Responsible Neurons, NeurIPS, 2020.
[3] WWW: A Unified Framework for Explaining What Where and Why of Neural Networks by Interpretation of Neuron Concepts, CVPR, 2024.


2. While the paper underscores the structural complexity of ViTs as a barrier to interpretability, it narrowly investigates only one neuron type—the output of the first linear layer in each Transformer's FFN module. This limited focus raises questions about the motivation: effective explainability typically aims to enhance interpretability for either the model’s input-output relationship or its internal operations holistically. The choice to explore only specific layers restricts the scope, potentially overlooking valuable insights into other model layers or their contributions.

3. Relying solely on joint attribution scores to assess neuron influence, though innovative, lacks breadth. Incorporating additional metrics—such as concept coherence or task-specific performance indicators—could provide a more comprehensive view of the Neuron Path approach’s effectiveness. Moreover, the experimental focus on ViTs for image classification lacks exploration into generalizability. Extending this analysis to other models and tasks, like object detection, could demonstrate the broader utility of the neuron paths identified here.

4. The assumption that neurons in ViT FFNs are largely redundant warrants more rigorous testing. Studies show that FFN layers encode high-level visual features essential for robust model performance, and not all neurons in these layers may be extraneous. For example, Intriguing Properties of Vision Transformers (NeurIPS, 2021) and others illustrate that FFNs in ViTs play a significant role in preserving semantic information across layers, making them central to effective representation learning. Testing this redundancy hypothesis more thoroughly could yield valuable insights into FFN layer functionality.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2