Discovering Influential Neuron Path in Vision Transformers
Authors: name: Yifan Wang, name: Yifei Liu, name: Yingdong Shi, name: Changming Li, name: Anqi Pang, name: Sibei Yang, name: Jingyi Yu, name: Kan Ren

Vision Transformer models exhibit immense power yet remain opaque to human
understanding, posing challenges and risks for practical applications. While
prior research has attempted to demystify these models through input
attribution and neuron role analysis, there's been a notable gap in considering
layer-level information and the holistic path of information flow across
layers. In this paper, we investigate the significance of influential neuron
paths within vision Transformers, which is a path of neurons from the model
input to output that impacts the model inference most significantly. We first
propose a joint influence measure to assess the contribution of a set of
neurons to the model outcome. And we further provide a layer-progressive neuron
locating approach that efficiently selects the most influential neuron at each
layer trying to discover the crucial neuron path from input to output within
the target model. Our experiments demonstrate the superiority of our method
finding the most influential neuron path along which the information flows,
over the existing baseline solutions. Additionally, the neuron paths have
illustrated that vision Transformers exhibit some specific inner working
mechanism for processing the visual information within the same image category.
We further analyze the key effects of these neurons on the image classification
task, showcasing that the found neuron paths have already preserved the model
capability on downstream tasks, which may also shed some lights on real-world
applications like model pruning. The project website including implementation
code is available at https://foundation-model-research.github.io/NeuronPath/.