NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions
Authors: Tue Minh Cao
Nhat Hoang-Xuan
Hieu Pham
Phi Le Nguyen
My T. Thai
Venue: ICLR 2025 Poster (2025)
Keywords: Explainable AI
Functional Interactions
Critical Neurons
Concept Circuits
Primary area: interpretability and explainable AI
TL;DR: In this paper, we propose a novel framework that shifts the focus from analyzing individual neurons to examining groups of critical neurons and their functional interactions that significantly influence model behavior.

Abstract: Understanding the inner workings of neural networks is essential for enhancing model performance and interpretability. Current research predominantly focuses on examining the connection between individual neurons and the model's final predictions, which suffers from challenges in interpreting the internal workings of the model, particularly when neurons encode multiple unrelated features. In this paper, we propose a novel framework that transitions the focus from analyzing individual neurons to investigating groups of neurons, shifting the emphasis from neuron-output relationships to the functional interactions between neurons. Our automated framework, NeurFlow, first identifies core neurons and clusters them into groups based on shared functional relationships, enabling a more coherent and interpretable view of the network’s internal processes. This approach facilitates the construction of a hierarchical circuit representing neuron interactions across layers, thus improving interpretability while reducing computational costs. Our extensive empirical studies validate the fidelity of our proposed NeurFlow. Additionally, we showcase its utility in practical applications such as image debugging and automatic concept labeling, thereby highlighting its potential to advance the field of neural network explainability.

Published as a conference paper at ICLR 2025
NEURFLOW:
INTERPRETING NEURAL NETWORKS THROUGH NEURON GROUPS AND FUNCTIONAL IN-
TERACTIONS
Tue M. Cao'
Nhat X. Hoang
Hieu H. Pham
Phi Le Nguyen"
My T. Thai
Institute for Al Innovation and Societal Impact (AI4LIFE), Hanoi University of Science and Technology. Hanoi, Vietaam (tue
hust edu.vn, lenp@soi ct. hust. edu.vn)
University of Florida, Gainesville, Florida, USA (hoangx, mythai @ythai]@uf]] @uuf edu
3 College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam VinUni-lllinois Smart Health Center, VinUniversity, Hanoi, Vietnam (hieu. pheeinnnhvinui... edu. vn)
ABSTRACT
Understanding the inner workings of neural networks essential for enhancing model performance and interpretability. Current research predominantly focuses on examining the connection between individual neurons and the model's final predictions, which suffers from challenges in interpreting the internal workings of the model, particularly when neurons encode multiple unrelated features. In this paper, we propose a novel framework that transitions the focus from analyzing individual neurons to investigating groups of neurons, shifting the emphasis from neuron-output relationships to the functional interactions between neurons. Our automated framework, NeurFlow, first identifies core neurons and clusters them into groups based on shared functional relationships, enabling a more coherent and interpretable view of the network's internal processes. This approach facilitates hicrarchical circuit representing neuron interactions across
the construction layers, thus improving interpretability while reducing computational costs. Our extensive empirical studies validate the fidelity of our proposed NeurFlow. Addi- tionally, showcase its utility in practical applications such image debugging and automatic concept labeling, thereby highlighting its potential to advance the field of neural network explainability.
INTRODUCTION
The explainable AI (XAI) field has seen significant advancement in understanding the mechanisms of deep neural networks (DNNs). This field emerges from the growing need decoding the internal representations, in hope of reverse engineering deep models into human interpretable program. Prior works have initiated on breaking down convolutional neural networks (CNNs) into interpretable neurons, understanding the models in the most fundamental units (Nguyen et al. 2016 Zeiler & Fergus| 2014: O' Mahony et al. [2023] Byykv et al., 2024). Extending further, one can examine the relation between neurons to gain insights on how the model works, within one layer (Vu et al.) 2022),
and between multiple layers (Cammarata et al. 2020). Ultimately, recent works try to generate
circuits (Cammarata et al. 2020 Bykov et al.) 2024 Wang et al. 2022c Conmy et al. 2023 that create exhaustive explanations of how features are processed and evolve throughout the model.
The majority of existing methods focuses individual neurons Oikarinen & Weng (2024b); La Rosa et al. (2023a) and their relationship to the model's final predictions Ghhorbani Zou (202bb); Wang al. 2022b); Ghorbani & Zou 2020a), while giving less attention exploring
"Corresponding Authors
Source code:
https://gthhh.ccm//ut////eeeeeee

---

Published as a conference paper at ICLR 2025
and quantifying the relationships and interactions between neurons across different layers. These approaches are not only constrained by scalability challenges arising from the extensive number of neurons, but they also hinder comprehensive understanding of the underlying mechanisms of DNNs. notable example is the polysemantic phenomenon Mu & Andreas (2020); O' et al. 2202); Olah al. (2020), where single neuron is activated by several unrelated concepts. Thhis phenomenon complicates the task of associating each neuron with a distinct feature and hampers the interpretation of how a model processes concepts based on the relationships among neurons. Draw- ing inspiration from human inference, which synthesizes information from variety of sources, we contend that, in addition to individual neuron encoding multiple concepts (as demonstrated in prior studies O' Mahony et al. (2023); Olah et al. (2020)), groups of neurons within each layer also col- lectively encode the same concept. Furthermore, the decision-making process in neural networks shaped not solely by the interactions between individual neurons, but rather by interactions among neuron groups
This study seeks to explore the roles and interactions of neuron groups in shaping and develop- ing concepts, enabling the execution of specific tasks. Due to the complex connections between large number of neurons, identifying those functions and there interactions is daunting task. To overcome this, we demonstrate that for particular task, only a subset of neurons neurons-referred to as core concept neurons-play crucial role influential and concept-defining elements in neural networks. These neurons, when deactivated, significantly alter the associated concepts.
concept neurons allows us view the intricate network in simplified way, reveal- ing the most important interactions between the groups of neurons. Therefore, we propose Neur- Flow framework that (I) identifies core concept neurons, (2) clusters these neurons into groups, and (3) investigates the functions and interactions of these groups. To enhance interpretability. we represent each neuron group by the set of visual features it encodes (i.e., named as neuron group's concept). Focusing on classification models, we construct, for each class of interest, a hierarchi- cal tree which nodes represent neuron groups (defined by the concepts they encode), and edge weights quantify the interactions between these groups.
Focusing
Our key contributions are summarized as follows:
i) We introduce an innovative framework that systematically builds a circuit to elucidate the mech- anisms by which core concept neuron groups operate and interact to achieve specific tasks. This entire process is automated, necessitating no human intervention predefined concept labels. To our knowledge, we are the first to employ neuron groups as the fundamental units for explaining the internal workings of deep neural networks.
ii) We perform empirical studies to validate the proposed framework, demonstrating the optimality and fidelity of core concept neurons, and the reliability of interaction weights between core concept neuron groups
iii) We provide experimental evidence showing that our framework can be applied to various tasks, including image debugging and automatic neuron concept labeling. Specifically, we confirm the biases found by Kim et al. (2024) ImageNet (Russakovsky al.) (2015), which have not been proven, by masking the core concept neurons related to the biased features.
RELATED WORK
In an effort understand the inner mechanism of DNNs, several branches of research have emerged:
Concept based. Kim et al. (2018) show that model can be rigorously understood by assigning meaning to the activations, referred to as concept activation vectors. Subsequent works (Ghorbani et al. 2019 ZZhan et al. (021) have explored more complex methods for extracting these mean- ings, however, the relationships between concepts remain understudied. Feel et al.) (2023); Kowal ct al. (2024) address this limitation by constructing graph of concepts with edges that quantify the relations. Their main intention is to see the evolution concepts throughout the network layers. Nevertheless, they unable to explain which parts of the model are responsible for these concepts.
Neuron based. Nguyen et al. (016)); Mu & Andreas (020): Mahony et al. 2023 : Bykov et al.
(024) invest effort in studying the meaning of neurons, in parallel, Vu et al. (022 Ghorbani & Zou (2020c); Khakzar et al. (2021b) propose different approaches in identifying important neurons to the model output. These researches shed light on the function of individual neurons and their
2

---

Published as a conference paper at ICLR 2025
The neuzon actndingg The get clsa
NeurFlow
710 Doin en nng
IDI Buld Tg
One reuron mullipe
grpup
each reuron prececing to the lagell reuron
Quardty intraction behwaes reJa
Claser reunce ince grous Quantily noron group irarnciion
Faandom prapa of barget daeal Imagee:
Figure 1: Workflow of NeurFlow, consisting of three main components: identifying core concept neurons in each layer, building the neuron circuit, and constructing the circuit of neuron groups.
impact on the prediction of the model. Recently, (Cammarata et al. 2020 Bykov et al., 2024 Achtibat et al.] 2023) connect the neurons to form circuits that explain the behavior of model throughout the layers, nevertheless, the circuits are constructed manually. Furthermore, a major limitation of all previous works is that they only analyze one neuron time. This approach is prone to the complex nature neuron, namely polysemantic neurons, where neurons may encode multiple distinet features, making model interpretation via neurons challenging (Cammarata et al. 2020
O' Mahony et al. 2023). Lastly, Wang et al. (2022)); Kalibhat et al. (2023) find the group of concept, however, the relations among the groups and the influence of
neurons that encode the a group on the model's outputs are left unexplored.
Graph based. Ren et al. 2023 Zhou al. (2024) try to approximate the mechanism of model by considering the causal relations between the inputs and outputs. Another notable method (Zhang al. 2018) generates graph that highlights what visual features activate feature map, for multiple layers. While this approach can be modified to form a circuit, the graph lacks meaningful edge weights. Consequently, it cannot quantify the contribution of each CNNs component to others and to the final prediction, unable to explain the inner mechanism (They use and-or- -rrd-or-gaahh (Zhang et al.) 2017) to form relations between components. However, unlike circuit, this new graph disregards the original structure of the model, where "concepts" of the first convolution could interact directly with "concepts" the last convolution.). Subsequent work (Zhang et al., 2019) fixes this issue by building decision tree to quantify the contribution of each feature map to the final predictions.
Our work aligns the most with explaining neurons and forming circuits. limitations of manual neuron labeling and circuit construction. We also propose way to look at neurons not individually but in groups to overcome the common problem of polysemantic neurons. Additionally, prioritize exploring the interactions between neuron groups across layers rather than focusing solely on the relationship between individual neurons and the model's output. Table in Appendix B provides a comparison of our method with the most relevant existing studies.
address the common
3 NEURFLOW FRAMEWORK
3.1 PROBLEM FORMULATION
Our goal is explain the internal mechanisms of deep neural networks (DNNs) by investigating how groups of neurons function and interact to encapsulate concepts, thereby performing a specific task. In partieular, we focus on the classification problem, exploring how groups of neurons process visual features to identify a class. Given the exponential number of possible neuron groups, we focus only core concept neurons. In addition, to facilitate human interpretation, we group these neurons through the common visual features they encode. In essence, we propose a comprehen- sive framework to address the following questions: (i) Which neurons play a crucial role in each layer? (ūi) How can these neurons be clustered, and what visual feutures does each neuron group encapsulate? (iii) How do groups of neurons in adjacent layers interact?
Our problem can be formulated as follows: Given pretrained classification network F and a dataset D. composed of exemplars from a specific class c, the goal to construct a hierarchical tree whose vertices represent groups of core concept neurons in each network layer, and the edges capture the relationships between these groups Figure | illutrates the workflow of our framework which comprises the following key components: (1) identifying
concept neurons (Section 3.3 (2)
determining inter-layer relationships among neurons (Section 3.4). (3) clustering the core concept neurons into groups, and analyzing the interactions between these neuron groups (Section 3.5)
3

---

Published as a conference paper at ICLR 2025
3.2 DEFINITIONS AND NOTATIONS
In this paper, the term neuron refers to either unit in a linear layer or feature map in a convolu- tional layer. As suggested by Cammarata et al. 2020); [ykov al. (2024); O O'MMhhoy al. (2023). each neuron is selectively activated by a distinct set of visual features, and by interpreting the neuron as representation of these features, we can gain insights into the internal representations of a DNN. We refer to these visual features as the concept of the neuron. In the following definitions, let represent an arbitrary neuron located in layer of the pretrained network F. In this study, we do not rely on any predefined concepts. Instead, we enhance the original dataset De by cutting it into smaller patches with varying sizes. These patches serve as visual features for probing the model. We refer to this augmented dataset as D, and denote v as an arbitrary element of D.
Definition (Neuron Concept). The neuron concept V. of neuron a is defined as the set of the top-k image patched that most strongly activate neuron a. Formally, the neuron concept of is expressed as Va = arg max Σvev (v), where (v) ID + R represents the activation of neuron a for D=
given input V DD and k is hyperparameter.
An empirical analysis of the impact of k (Appendix D.7 reveals that NeurFlow's performance is relatively insensitive to the selection of k.
Definition 2 (Neuron Concept with Knockout). Let M be the computational graph of the network F. be an arbitrary subset neurons of M, and M \ S be the sub-graph of M after removing S; let Φ5 be the activation of neuron computed from M SS. The neuron concept of a when knocking-out (denoted as V:S) is defined as V.S := arg max Evey (v).
VCD[VV=
We hypothesize that for each neuron a, only small subset of neurons from the preceding layer exert the most significant influence on a. In particular, knocking these neurons would lead to a substantial change in the concept associated with a. We refer to these neurons as core concept neurons and provide formal definition in the following.
Definition 3 (Core Concept Neurons). Given neuron layer I, core concept neurons of (de- noted as S,) is the sub-set of neurons at the previous layer I 1 satisfying the following conditions:
S. = arg min ScS
2 n nva
where S is set of all neurons at layer / - 1 and T is predefined threshold. Intuitively, the core concept neurons for target neuron a are those that play an important role in defining the concepts represented by In practice, the value of T may vary across the network layers, its impact will be elaborated upon in Sections
In the following, denote by d1.11(()) D + R the function that maps the input e to the feature maps at the (1 (1- 1)-th layer of the model, where m represents the number of channels, and h indicates the dimensions of each feature map. Furthermore, adopt the notation |-| to indicate the cardinality of set, while ||.|| is employed to represent the absolute value. We summarize all the notations in Table [ (Appendix A).
3.3 IDENTIFYING CORE CONCEPT NEURONS
Given a neuron a, we describe our algorithm for identifying its core concept neuron set S., This process consists of two main steps: determining a's concept Va according to Definition [ and identifying core concept neurons following Definition 3
Firstly, we generate a set image patches DD by augmenting the original dataset D., which consists of images that the model classifies as class Since neurons can detect visual features at different levels of granularity, we divide each image in D. into smaller patches using various crop sizes, where smaller patches capture simpler visual features and larger patches represent more complex ones. We subsequently evaluate all items in D to identify the top-k image patches that induce the highest activation in neuron a, thereby constructing V.. cach image patch is piece cropped from image set.
4

---

Published as a conference paper at ICLR 2025
With V. identified, one could determine the core concept neurons through brute-force search all possible candidates. However, this naive approach computationally infeasible. To this end, define metric named importance score that quantifies the attribution of a neuron 8, The importance score can be intuitively seen as integrated gradients (Sundararajan et al. (2017 ) of a to &t calculated across all elements of Va. calculated follows:
T(a, S4. Va)
Σ DEV.
E
xX
1
Σ
By xx) dx
N
ZEV 1 (e):
yEdl
" (v))
where 01/-1 is the element of rf corresponding to neuron 8i, %%-1, depicts the function map- ping from the activation vector of layer Utilizing the importance scores of all neurons in the preceding layer, the set of core concept neurons identified by selecting the top neurons that exhibit the highest absolute scores. To justify the use of integrated gradients, we empirically show a strong correlation between the absolute values of (a, S;. Va) and the change a's s concept after knocking out 8,, as demonstrated in Section Additionally, we compare our method with other attribution techniques in Appendix D.1
the activation neuron a, and N the step size.
3.4 CONSTRUCTING CORE CONCEPT NEURON CIRCUIT
For each class of interest c, the neuron circuit He is represented as a hierarchical hypertred with the root de being the neuron in the logit layer (ouput) associated with class e. The nodes each layer of the tree He are the core concept neurons of those in the layer above, and branches connecting parent node a and its child s, E S represents the contributions of s, to a's concept.
As mentioned in (Cammarata al. 2020; O Mahony al. 2023), neurons often exhibit polyse- mantic behavior, meaning that a single neuron may encode multiple distinct visual features. In other words, the visual features within concept V. of neuron may not share the meaning and can be categorized into distinct groups, which we term semantic groups. We hypothesize that each core concept neuron &; makes distinct contribution to each semantic group of neuron To model this relationship, we represent the interaction between s; and a through multiple connections, where the j-th connection reflects 8;'s 's influence on Va.j. the j-th semantic group of
At a conceptual level, the algorithm for constructing the hypertree He proceeds through the follow- ing steps: (1) employing our core concept neuron identification algorithm determine the children of each node in the tree (Section 3.3). (2) clustering the neuron concept of each parent node into semantic groups, and (3) assigning weights each branch connecting child node to the semantic groups of its parent. Figure @illustrates our algorithm. The complete algorithm for constructing the core concept neuron circuit is presented in Appendix We provide a detailed explanation of these steps below.
Determining semantic groups. Let the concept Va corresponding
composed of elements (u.....,u ).). For each visual feature vi. (i (i= 1, 1,..., k), we define its representative vector r r(v)) I.T as:
r(x))) = mean (z) (o)) ,.... mean (0 (d (c:))
(3)
where (1--1(( (v)) (j = 1,... m) represents the j-th feature map and mean (v:)) denotes
the average value across its all elements. Next, endre 2014) to divide the set (u....., [+1 into clusters, where the distance between two visual Teatures vg. vg is defined by the distance between their corresponding representative vectors r(v?), r(ug). ). The Silhouette score (Rousseeuw) 1987) is employed to ascertain the optimal number of clusters. The complete procedure is detailed in Algorithm
agglomerative clustering (Murtagh & Leg-
Calculating edge weight. The weight w(a, Si, Va.i) of the branch connecting child s; and its parent a's j-th semantic group Vo.i is defined as:
T(a, 84. Vo.j)
w(a, 84, Va.) E.es, [[T 8, Va.s)ll
(4)
where T(a, S1, Va.j) is the importance score of s; a calculated over Va.j-
A hypertree is tree in which each pair may be connected by multiple edges.
5

---

Published as a conference paper at ICLR 2025
Caltuiste representaion wador of semartic group
(1)) Decompose % iKb
(2) Caloulate Integroted semartic 1
Ckuster the semanti group
[ Cangrote chater somantic group
Figure The interaction between neuron 8; and its parent a.
Figure Illustration of our algorithm to determine groups neurons.
3.5 DETERMINING GROUPS OF NEURONS AND CONSTRUCTING CONCEPT CIRCUIT
This section describes our algorithms to (1) cluster the set of core concept neurons & {s) .... 35} into distinct groups, (2) identifying the concept associated with each group, and (3) quantifying the interaction between the groups
Clustering neurons into groups. As mentioned in the previous section, single neuron can encode multiple distinct visual features, while several neurons may also capture the same visual feature Cammarata al. (2020). We hypothesize that, due the polysemantic nature of neurons (Cam-) marata et al. 2020 O O'Mahonn et al., 2023), model may struggle to accurately determine whether a concept is present an input image by relying on a single neuron. As a result, the model pro- cesses visual features not by considering individual neurons in isolation but rather by operating at the level of neuron groups. Intuitively, a group of neurons consists those that capture similar visual features. This can be interpreted two neurons belonging to the same group if they share similar semantic concept groups.
Building this intuition, we develop neuron clustering algorithm based on the semantic groups of each neuron's concept (Figure 3. Specifically, let V., represent the concept of neuron 8; (i.e., the primary visual features it encodes), which
be decomposed into several semantic groups
(V (Va....-.., Vs..n:) (see Section (344) where n, is the number of semantic groups encoded by 8,- For each semantic group Ve.j. j. we calculate its representative activation vector Ts,.) by averaging the feature maps of all its visual features, i.e., 3 == [V.,, Ev,Ev., meenn(621-1(0
We then apply the agglomerative clustering algorithm to group the semantic groups Va., (i L,..., 1, ...., m.),,,)),, na), where the distance between any two groups Vs., and Vey.u is determined by the distance of their respective representative activation vectors, Ter,u and fsy,w- Finally, assign neurons $1;... 8g to the same groups based on their semantic concept groups. Specifically, neurons s; and 8j clustered together if there exists semantic group Va,,u (of 8;) and semantic group Vag.an (of Kj) belonging to the same group.
Finding neuron group concept automatically. We define the concept associated with group of neurons as the union of all visual features from the corresponding semantic groups. Specifically, let (V.. 3a, represent the semantic groups categorized into cluster, with their corresponding neurons {sg.1. &G. grouped together in the same set, denoted as G. The concept of this group, denoted as Vo. is then defined as the union of the sets {Vc....... Va,a i.e., Ve == U Va,i- We leverage a Multimodal LLM to automatically assign labels to the concept, thereby eliminating the need for a predefined labeled concept dictionary. Further details on the design of the prompts are provided in the Appendix
Constructing concept circuit. For a given class c, the concept circuit C. is hierarchical tree where each node represents neuron group concept (NGC), and each edge illustrates the contribution of the child neuron group to its parent. For a node G. we denote by VG = (Vc.1...., Ve..i,...,V.vvvv.. Vc.Ival) the set of semantic groups associated with G, and Se
{sc.1,... SG, sal} represent the neurons corre- sponding to the semantic groups in Va, i.e., sG.j is the core concept neuron possesses the semantic group Ve,; (j 1, ..., (V))) ). Let G; and G, be a child-parent pair in the tree, then, the relationship between G; and G (quantified by WGG, G2)) is represented by two aspects: the number of edges connecting elements of G, and G,, and the weights of those connecting edges. The more the edges and the higher the edge weights, the stronger the relationship between G2 and Gy. Accordingly, define the weight of branch connecting child G, to its parent G, as sum of the attribution of each neuron in Sc. with each neuron in Sc, W (G,, G5)= : Eecg.0Bee w($G, SG.,g VG,,p).

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTAL EVALUATION
We perform an extensive empirical study to investigating three aspects, including: optimality of core concept neurons, fidelity of core concept neurons, and fidelity of neuron interaction weights. Our ex- periments are performed on ResNet50 (He et al.,2016, and GoogLeNet SxegeeN et al. (2015) using the ILSVRC2012 validation set (Russakovsky al.] 2015). The models are pretrained in Pytorch (Paszke et al. 2019), and layer names follow Pytorch's 's conventions (e.g.. layer4.2 for ResNet50). Unless otherwise specified, the input parameters are т 16, N 50, and k 50, where the top 50 images with the highest activation on the target neuron are considered as its concept. We will release the source code once the paper is published.
Optimality of core concept neurons. Ac- cording to Definition 3 the core con- cept neuron set S. for neuron a is the set that minimizes the objective function De n VV/l without exceeding the cardi- nality т. To evaluate our heuristic solu- tion, we define loss function L(S, a) ==
ResNet50
GoogLeNet
0.6
Loss difference
0. 0.2
U.4
32
Layars
Lagers
Baseline
C-m (Tocaa
30 (Tocal)
2- 50 (Tocal)
V nVaa //Val. balancing these two ob- jectives. In this experiment, our objec- tive is to demonstrate that the core concept neurons, S., identified by our algorithm are npar-op Ideally, a brute-force search over all possible combinations of neurons would be conducted to demonstrate that these combinations yield a higher loss function value compared to S.. However, such an approach is computationally infeasible due to its pro- hibitive cost. Consequently, we perform experiments using a large set of randomly selected com- binations. Specifically, we use three different values of т, specifically 10, 30, 50. For each setting, we randomly select 50 target neurons (denoted by a,) from 10 distinct classes (five neurons for each class). For each target neuron 0g, we determine its core concept neuron set Sa, using our algorithm and generate 100 random neuron combinations, with the same cardinality as S.., from the preced- ing layer of ag. In total, the experiments are performed over 15, 000 cases per layer for each model. We compare the loss differences between Sa, and the random neuron combinations. These average differences along with 99% the confidence intervals are shown in Figure 4 Additionally, we report cases where the random combinations resulted in a smaller loss than our core concept neurons. As observed, the average differences are positive in all cases, indicating that replacing the core concept neurons identified by our algorithm with random ones generally leads to a significant increase in the loss for both models. Furthermore, only a few cases show a random combination achieving a smaller loss than our core concept neurons, and in those instances, the gap is negligible.
T- 1D (llequtioo)
30 (Mlegatioe)
50 (Meggstive)
Figure 4: The difference in losses between core concept neurons and random neuron combinations. The blue- toned bars represent the average losses, while the pink-toned bars indicate instances where random neuron combinations result in smaller losses compared to core concept neurons.
Fidelity of core concept neurons. We evaluate the impact of the identified core concept neurons on the model's 's performance by comparing two variants: (1) Retaining version-all neurons masked except for the core concept neurons, and (2) Masking version version-only the core concept neurons masked. Intuitively, a higher performance in the Retaining version and a lower performance in the Masking version would indicate that the core concept neurons play a significant role in the model's performance. We compare the performance of these two versions against models obtained by performing retraining and masking on equal numbers of random neurons. We select 50 random classes and apply the retaining and masking operations two levels: on single layer or across multiple layers. In the multi-layer scenario, masking or retaining is applied from the linear classifier down to a specified layer. Figure 5 presents the results for 7 4,8, and 16. The y-axis indicates changes in model accuracy, where a value of implies that masking neurons does not affect pre- dictions. is evident that masking core concept neurons consistently results in a more pronounced decline in performance compared to masking random neuron combinations. Moreover, the rate of decline in accuracy, moving from higher to lower layers, is considerably steeper for the core con- cept neurons than for random neurons. The most significant discrepancy occurs at layer 5a of the GoogLeNet model, where masking core concept neurons at this layer reduces model accuracy to nearly O, while masking random neurons has minimal effect on performance. Retaining version, preserving only the core concept neurons allows the model to maintain its performance substantially better than when random neurons retained. This experiment also demonstrates that the value of
7

---

Published as a conference paper at ICLR 2025
T-600
Randam
Randdom
T-I6-CC mutt-ayee Byer masking
Random
ResNet50 single byer massing
single eyer retaiiing
mub-layer reaiing
1,00
Accurnnn
2-075
0.50
0.25
0.00
4.2
4.0
3.4
3.2
3.0
4.0
3.4
3.2
3.0
4.2
4.0
3.4
3.2
3,0
4.2
4.0
3.4
3.2
3.0
GoogleMet dingl layer masking
GoogLehett single layer retaining
GoogLetist mult mutt-layrr masking
GoogLeMet mutti byyer retaining
Accurayy
0.50
0.25
0.00
Layers
Figure 5: Effects of neuron groups on model'ss performance. Retaining only the core concept (denoted as CC) neurons preserves high accuracy, whereas masking them leads to a significant drop in performance. In contrast, random neuron combinations show the opposite trend.
T represents trade-off between the simplicity of the circuit and the comprehensiveness of capturing the core concept neurons. A smaller r results in greater instability in the model's performance dur- ing the retaining experiment, leading to a more pronounced performance drop. For more discussion on the impacts of T, please refer to Appendix D.6
We conduct an experiment to show that adding non-core neurons the concept core neurons set identified by NeurFlow has minimal impact the model's performance. Specifically, we perform the Fidelity experiment with r = 16, incorporating 50% more non-core neurons (i.e., those that are not concept core neurons), and evaluated their impact on model accuracy. These neurons were selected greedily, prioritizing those with the highest scores as ranked by NeuronMCT Khakzar et al. (2021a). The results in Figure 17 (Appendix (.6) indicate that for ResNet-50, adding non-core neurons had little to no effect on improving model performance, confirming that when 7 is sufficiently large. our algorithm ensures completeness.
Fidelity of neuron interaction weights. The edge weight representing the interaction between core concept neurons (or groups of core concept neurons) is defined using Integrated Gradients (IG) (Definition B) Without the ground truth, we evaluate the fidelity of edge weights based on the following rationale: if the weights assigned by our definition are meaningful, they should accurately rank the importance of neurons in the preceding layer in detecting the concept represented by a target neuron in the subsequent layer. We demonstrate that our IG-based scores exhibit strong correlation with the loss, not only for single neuron setup but also for groups of neurons. Specifically, we randomly select 10 target neurons from 10 distinct classes (denoted as a;, where = 1...., 10). For each target neuron ai we generate random combinations of neurons from the preceding layer. We then measure the correlation between the losses caused by these random neuron combinations and the sum of the absolute values of their IG-based importance scores with respect to a, The experiments are conducted using 500 neuron combinations, with cardinality (r) varying from 1 to 50. Figure 66 presents the average correlation across all combinations. The results indicate that for r < 50, IG-based scores maintain a high correlation across all layers. Notably, for = 1, the correlation consistently exceeds 0.6 in both models, and up almost perfect correlations for several layers in ResNet50. While the correlation diminishes as т increases, our focus is on a small subset of core concept: thus, for sparse sub-graph of core concept neurons, these results are considered satisfactory. We further compare our defined IG-based score with other attribution methods, including the one used in Vu et al. 2022). in the Appendix D.I
Quantitative comparison of NeurFlow with existing approaches. While our approach fo- cuses on identifying core concept neurons rela- tive to a specific target neuron, we demonstrate that the neurons identified by our method also significantly influence the model's final output. To validate this, we analyzed the overlap be- tween our core concept neurons and the critical neurons identified by Vu et al. (2022), and Neu- ronMCT (Khakzar et al.] 2021a). The F, scores for these overlaps are presented in Table 3 (Appendix D.4.. The results indicate that NeurFlow iden-
Res/Nel 50
GoogleNeet
-02
-0.2
Complation
-3.0
40 4.2
3.4
32
3.0
2.0
11
30
46
30
[=1
T=5
10
20
50
Figure 6: Correlation between loss and our defined IG- based importance scores.
8

---

Published as a conference paper at ICLR 2025
Class: bo eogle Claoiie:: Toolied0
Group
FC dide
Strkk
Loyerd BEEE
Morc of prea teels
G
Group 2
Certlt
Layend
terch Lf grao
shery
lest grop
Hoo0 on grop
[940))
[D47]
(0:40
[0.0)]
Group4
Group
Group
GE MITH
Why buikd ougle?
Figure Demonstration for automatically labelling and explaining the relation of NGCs on class "great white shark" using GPT4-o [OpenAl] (2024). The cap- tions and the names of the NGCs are highlighted in blue, while the relations are black.
Figure 7: Using NeurFlow to reveal the reason behind model's prediction. The top concepts can be traced throughout the circuit.
tifies core concept neurons largely similar to those found by NeuronMCT, even though it does not explicitly find critical neurons to the model's output. Additionally, we compare our approach for identifying core concept neurons for specific target neuron with the method proposed in Cam- marata et al. (2020). Details of this experiment can be found in Section [..] (Appendix [D.5). The results, summarized in Table g (Appendix), show that our method is more effective in identifying core concept neurons.
5 APPLICATIONS
We outlines some applications of NeurFlow. We hypothesize that, as one neuron can have multiple meanings, DNN looks at group of neurons rather than individually to determine the exact features of the input. Hence, we propose a metric that assesses a model's con- fidence in determining whether the input contains specific visual feature. For group G with core concept neurons &c = exp( Bal Ex (og(llo.()) / D) where v E D and max(é,, D) is the highest value of activation of neuron E So dataset D. This returns high score when all neurons in G have high activation (indicating high confidence), while resulting in almost zero if any neuron in the group has low activation (indicating low confidence). We can use this metric to determine how similar the features in the input image are to the predetermined neuron groups concept. The specific setup can be found in the Appendix [F Figures 7 and demonstrate the usage of the metric and the concept circuit. We use the term NGC to denote the concept of a neuron group.
the metric denoted as (u, D) =
5.1 IMAGE DEBUGGING
We aim to use the concept circuit to identify concepts contributing to false prediction, which we call image debugging. If concept contributes to a class when it should not, we say that the prediction (or cquivalently, the model) is biased by that concept. Kim et al. (224)) propose a framework for detecting biases in a vision model by generating captions for the predicted images and tracking the common keywords found in the captions. With this method, they concluded that the pretrained ResNe150 is biased by "flower pedals" in the class "bee". However, correlational features do not imply causation and can lead to misjudgments. We verify and enhance the causality of their claim by examining the concept circuit of class "bee". and conducting experiments on the probabilities of the final predictions with and without neurons that related to "flowers". Additionally, we discover that the model also suffers from "green background" bias (resemble "leaves"), which is not mentioned
in Kim al. (2024).
Figure shows the process of debugging false positive images. Three different concepts are pre- sented in layer4.2 of ResNet50, representing "pink pedals", "green background", and "bee" respec- tively (we choose this layer as it has a small set of NGCs, however, our following experiment is consistent for multiple layers and with different classes). We discover that most of the false positive images have high metric score for "pedal" and "green background". To further verify the impact of these biased features, we mask all neurons in the groups of the respective concepts and find that the probability of the predictions are distorted drastically (and predictions is longer "bee"), as opposed to masking random neurons, which yield negligible changes.
This implies the dependence the biased concept. But how do we know that the groups reflect the respective visual features? ? If these groups indeed represent the visual features, then masking them should hinder the classification probability for images that include those features. We highlight the
9

---

Published as a conference paper at ICLR 2025
Faloe positive images
Does combination reflect the concept?
Prediotodl Trunlabel: 303 (Seatia)
Treettel P00PPM
Traditted out duiely lead
Mooking Fig conceet
loght diop
Concapet
Augreented
9.
Valtesion
25% 311
Original
Concept2
Augreented
2114
311
Masking oomeegtt 1,2
Wastssinn
304
009
52N
41%
774
Randem masking
35%
Figure 9: (left) The metric scores false positive images for each concept layer4.2 of ResNe150, (right) Showing the images that have the greatest drop in the activation of the logit neuron when masking each group concept. Verifying that the neuron groups indeed reflect the concepts.
top images that have the largest decrease in the value of the logit neuron (corresponding to class "bee") on both validation set of the target class and augmented dataset (see Section 3.3). As shown in Figure 9 this process indeed yields the images that contain the respective features.
To demonstrate how NeurFlow's 's findings differ from those of existing methods, we conduct qual- itative experiment comparing the core concept neurons identified by NeurFlow with those identified by NeuCEPT Vu et al. (2022). Detailed information about this experiment is provided in Appendix D33 Our observations indicate that the top logit drop images identified by NcurFlow align bet- ter with the representative examples of the corresponding concepts. Moreover, masking the core concept neuron groups identified by NeurFlow resulted in more significant changes to prediction probabilities while utilizing fewer neurons compared to the groups identified by NeuCEPT.
5.2 AUTOMATIC IDENTIFICATION OF LAYER-BY-LAVER BY-LAYER RELATIONS
While automatically discovering concepts from inner representation has been a prominent field of research (Fel et al., 2023), automatically explaining the resulting concepts is often ignored, relying on manual annotations, Bykov et al. (2024) utilize label description in ImageNet dataset to generate caption for neurons, however, these annotations limited and can not be used to label low level concepts. Drawing inspiration from Hoang- Xuan et al. (222)); Kalibhat et al. (2023), we go one step further and not only use MI.LM to label the (group of) neurons but also explain the relations between them in consecutive layers. Thus, show the prospect of completing the whole picture of abstracting and explaining the inner representation a systematic manner.
Specifically, for two consecutive layers, we ask MLLM to describe the common visual features in a NGC, then matching with those of the top NGC (with the highest weights) at the preceding layer. This can be done iteratively throughout the concept circuit, generating comprehensive explanation without human effort. We use a popular technique (Wei et al.] 2022) to guide GPT4-o (OpenAT, 2024) step by step in captioning and in visual feature matching. Figure & shows an example of applying this technique to concept circuit of class "great white shark". We observe that MILMM can correctly identify the common visual features within exemplary images of NGCs. Furthermore, MLLM is able to match the features from lower level NGCs to those at higher level, detailing formation of new features, showing the potential of explaining in automation, capturing the gradual process of constructing the output of the model. The prompt used in this experiment is available in Appendix G
CONCLUSION
We introduced NeurFlow, framework that systematically elucidates the function and interactions of neuron groups within neural networks. By focusing on the most important neurons, we revealed relationships between neuron groups, which are often obscured by the inherent complexity of neural network structures. Furthermore, we fully automated the processes of identifying, interpreting, and annotating neuron group circuits using large language models. Our method aims to provide a more efficient and comprehensive approach to the automated interpretation of neural activity and applicability of NeurFlow across a variety of domains, including image debugging and automatic concept labeling.
10

---

Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schu- bert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. http:://disilllpbb/200eerrrrruu Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was funded by Vingroup Joint Stock Company (Vingroup JSC), Vingroup, and supported by Vingroup Innovation Foundation (VINIF) under project code SANIF:222I.DA01111
This work is partially supported by the US National Science Foundation under SCH-2123809 project.
REFERENCES
Reduan Achtibat, Maximilian Dreyer, lona Fisenbraun, Sebastian Bosse, Thomas Wiegand, Woj- ciech Samek, and Sehastian Lapuschkin. From attribution maps to human-underssanddble expla- nations through concept relevance propagation. Nature Machine Intelligence, 1(0):100--0019 2023.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus- Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina Höhne. Labeling neural representations with inverse recognition. Advances Neural Information Processing Systems, 36, 2024.
Distill, 2020.
Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:' model- x'knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):551-577, 2018.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances Neural Information Processing Systems,
2023.
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
2711-2721, 2023.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. In Ad- vances in Neural Information Processing Systems, volume 33. pp. 5922-5932. Curran Associates, Inc., 2020a.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Ad- vances in neural information processing systems, 33:5922-599 5932,, 2020b.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Ad- vances in neural information processing systems, 33:5922-5932, 2020c.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in neural information processing systems, 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Nhat Hoang-Xusa Suan, Minh Vu, and My Thai. LIm-assisted concept discovery: Automatically iden- tifying and explaining neuron functions. arXiv preprint arXiv:2406.0885 08572, 2024.
Neha Kalibhat, Shweta Bhardwaj, C Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces in image representations. In International Conference on Machine Learning, pp. 15623-15635 PMLR, 2023.

---

Published as a conference paper at ICLR 2025
A Khakzar, S Baselizadeh, S Khanduja, C Rupprecht, ST Kim, and N Navab. Neural response interpretation through the lens of critical pathways. in 2021 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13523-13533 2021a.
Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Seong Tae Kim, and Nassir Navab. Neural response interpretation through the lens of critical pathways. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13528- 13538, 2021b.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp. 2668-2677. PMLR, 2018.
Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discov- ering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11082-11092 1092, 2024.
Matthew Kowal, Richard P Wildes, and Konstantinos G Derpanis. Visual concept connectome (vec): Open world concept discovery and their interlayer connections in deep models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10895-10905, 2024.
Biagio La Rosa, Leilani Gilpin, and Roberto Capobianco. Towards a fuller understanding of neu- rons with clustered compositional explanations. In Advances in Neural Information Processing Systems, volume 36, pp. 70333-70354. Curran Associates, Inc., 2023a.
Biagio La Rosa, Leilani Gilpin, and Roberto Capobianco. Towards a fuller understanding of neurons with clustered compositional explanations. Advances in Neural Information Processing Systems, 36:70333-7033 -0334,,-7334, 2023b.
Scott Lundberg. A unified approach to interpreting model predictions.
arXiv preprint
1rXiv:1705.07874. 2017.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Informa- lion Processing Systems, 33:17153-17163 2020.
Fionn Murtagh and Pierre Legendre. Ward's hierarchical agglomerative clustering method: which algorithms implement ward's criterion? Journal of classification, 31:274-295, 2014.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv:1602.03616, 2016.
Tuomas Oikarinen and Tsui- Weng. Linear explanations for individual neurons. arXiv preprint 20550:2405.06855 2024a.
Tuomas Oikarinen and Tsui- Wei Weng. Linear explanations for individual neurons. arXiv preprint arXiv:2405.06885, 2024b.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024 001, 2020.
Laura O' Mahony, Vincent Andrearczyk, Henning Müller, and Mara Graziani. Disentangling neuron representations with concept vectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3770-3775, 2023.
OpenAl. ottpa/////////////////aa//aaa//a
pttps://openai.comm
gpt-4o-to -4
2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jie Ren, Mingjie Li, Qirui Chen, Huiqi Deng, and Quanshi Zhang. Defining and quantifying the emergence of sparse concepts in dnns. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20280-20289 2023.
12

---

Published as a conference paper at ICLR 2025
Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of clus- ter analysis. Journal of Computational and Applied Mathematics, 20:53-65, 1987. ISSN 0377-0427. doi: http://bbbao////.000000777777710 sci eeeeencciieee rect... com.co/////oooo..
URL https: :ttp::://ww...
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision
(IJCV), 15(3))211-255 2015. doi: 10. 10010077661223--11--81116611
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks:
Visualising image classification models and saliency maps, 2014. URL https://arxiv.
org/abs/ 131.. 6034
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:/706.0382) 2017.
Asher Spector and Lucas Janson. Powerful knockoffs via minimizing reconstructability. Annals of Statistics, 2021+. To Appear.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.68066 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Minh N Vu, Truc D Nguyen, and My T Thai. Neucept: Locally discover neural net- works' mechanism via critical neurons identification with precision guarantee. arXiv preprint arXiv:2209.00 08448, 2022.
Andong Wang, Wei-Ning Lee, and Xiaojuan Qi. Hint: Hierarchical neuron concept explainer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10254-10264, 2022a.
Andong Wang, Wei-Ning Lee, and Xiaojuan Qi. Hint: Hierarchical neuron concept explainer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
10254-10264, 2022b.
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint 2rXiv:2211.00550 2022c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thougth prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24882 2022.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 818-833. Springer, 2014.
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Clon Chun Zhu. Growing interpretable part graphs on convnets via multi-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu. and Song- Chun Zhu. Interpreting enn knowledge via an explanatory graph. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018
13

---

Published as a conference paper at ICLR 2025
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting emns via decision trees. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6261- 6270, 2019.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. In- vertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11682-11690, 2021.
Huilin Zhou, Hao Zhang, Huiqi Deng, Dongrui Liu, Wen Shen, Shih-Han Chan, and Quanshi Zhang. Explaining generalization power of dnn using interactive concepts. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17105-17113, 2024.
14

---

Published as a conference paper at ICLR 2025
A
NOTATIONS
We summarize the notations used in this work in Table
Table 1: Notations summarization.
Notation
Mcaning
neuron a model layer
S
A set of neurons at layer
Va
Concept of a: the top-k highest image patches that activate a
Va
Concept of a when knocking out S
Vad
The j-th semantic group of a
The number of core concept neurons of The set of core concept neurons of a
Su
The function that maps from the dataset to the activation at layer of the model
T(a, Si, Va)
The importance score of 8; E S. w.r.t a on Va
w(a, Si. Va.5)
The normalized importance score 8; w.r.t a on Va.j
r(v)
The activation vector of an input
Vaui
The representative activation vector of the j-th semantic group
G
neuron group
Sc
The set of neurons of G
VG
The concept of
GGG,, G4)
The edge weight between G, and G2
B
RELATED WORKS SUMMARIZATION
Table 2 compares our proposed method and existing approaches.
Table Comparison of NeurFlow and existing approaches.
Method
Objectives
Level of granularity
Interaction quantification
Vu al.
Finding critical neurons the model's output
Ghorbani & Zou
2020c
Khakzar
202
O O'Mahony el al.
2023
Mu & Andreas
2020
Neuron
N/A
Rosa ct
20233
Oikarinen & Weng
2024a
Individual neuron explanation
& Andreas 2020
Bykov ct al. 2024
Kalibhat 2025
Group of neurons
Wang et al,
2022a
Kowal et
2024
Determining concept connectivity
Concept
Concept interaction
Determining groups of neurons"
NeurFlow (Ours)
Group of neurons
Neuron group interaction
funetion and interaction
C LIMITATIONS AND DISCUSSION
While we view NeurFlow as a significant step toward understanding the function and interaction of neuron groups, it is not without limitations. Our approach defines the concept of neurons the top-k most activated visual features, a common practice in the field (O MMMony et al. 2023 Mu & Andreas 2020 Nguyen et al., 2016). However, other researchers have broadened this definition to include concepts spanning a wider range of activation patterns (La Rosa et al.) 20236 Oikarinen & Weng 2024a). This limitation highlights a promising direction for future research: developing more
15

---

Published as a conference paper at ICLR 2025
flexible frameworks that incorporate both top-k activation and more distributed neural activation patterns.
Furthermore, research primarily focus CNNs, which follows the main focus of range of previous works the field (Cammarata et al. 2020 O M'Mhooyy et al. 2023 Nguyen et al. 2016 Mu & Andreas 2020). However, we can apply our framework onto different DNN architectures by following several steps: define the granularity level of neurons (i.e. individual units, feature maps, attention heads etc.); 2) iteratively identify the target neuron concept and the core concept neurons; 3) cluster the core concept neurons into groups and construct the concept graph. While exploring the differences in the inner workings of various architectures is valuable, we leave this promising direction for future works.
D ABLATION STUDIES
D.1 COMPARISON OF AITRIBUTION METHODS
ablation study different choices of attribution method apart form integrated gradient (IG) approach, verifying that IG-based score is the most suitable for the quantifi- cation of edge weights. We assess four additional common pixel attribution methods, including LRP (Bach et al. 2015), Guided Backpropagation (Springenberg et al., 2014). SmoothGrad (Smilkov al. 2017), Saliency (Simonyan al. 2014), Gradient Shap (Lundberg 2017). Notably, Smooth- Grad and Gradient Shap are a follow-up versions of IG. Furthermore, we also evaluate attribution method used in Vu et al. (2022). which also find important neurons and attributing scores to them, referred to as Knockoff (Candes et al. 2018). We run on the same setup as in Section [ T ranging from 50. For easier comparison, we report the mean correlations of all values of т. Figure show the mean correlations across the last 10 layers of ResNet50 (He et al.] (016)) and GoogLeNet (Szegedy al., 2015). The Integrated Gradient consistently yields higher correlations compared to other attribution method, surpassing its follow-up version SmoothGrad, while being comparable with Gradient Shap. Furthermore, Knockoff shows a poor performance in ranking the importance of neurons compared to other attribution methods.
In this section,
Additionally, we also assess the running time of each method. Specifically, we recorded the run time of each method on 50 images on CPU (we implement Knockoff on KnockPy library (Spector & Janson 2021+) which does not run on GPU, hence, we evaluate all others on CPU for a fair comparison) across all layers of GoogLeNet. The results in Figure [0] show that IG maintain small running time compared to the follow-up method (i.e. SmoothGrad and Gradient Shap), while yielding the best correlations among the attribution methods, Hence, we choose IG-based assign the edge weights in NeurFlow.
Run time comparison
ResNet50
GoogLeNet
average Correaation
0.0
-0.2
Gradients
-0.4
UtP LPP
-0.6
COP 0.2
GBP- Guided Bacesropapaoon
Sel-Seelncy
-0.0
0.5
50 Smouthrk
KO Incctalf
G6 Gradient Shop
-1.0
3,2
Layers
integrated Gradients
LRF
Knockoff
Gradient Shap
Time
Guided Backpropagation
Saliency
SmoothGrad
Figure 10: The comparison of average infer- ence time across all layers in GoogLeNet on CPU.
Figure 11: The comparison of different attribution methods for edge weight quantification.
D.2 NEURON GROUP RELATION WEIGHTS AGGREGATION
In this experiment, we compare our choice of summing the edge weights with averaging the edge weights in forming W (GG,, G,) in Section 3.5 Our aim is to verify that: groups of neurons with higher sum of scores will have higher impact on a target neuron, regardless of the number of neurons in the group.
16

---

Published as a conference paper at ICLR 2025
We randomly sample 500 groups of neurons of varying sizes, ranging from {1, 5, 10, 20,50, For a target neuron the upper layer, we analyzed the correlation between the loss function (defined in and two metrics: the average edge weights within each group and our original scoring method, which sums the edge weights of neurons in the group. Higher absolute correlation values indicate more effective scoring method. The results in figure [2] are the average of 10 neurons of different labels in both GoogLeNet and ResNet50.
50}.
ResNet50
GoogLeNet
-0.2
-0.2
Correlation
-0.4
-0.4
-0.8
-0.8
-1.0
-1.0
4.2 4.0
3.2
3.0
4c
Layers
Layers
Original
Average
Figure 12: The correlations across 10 layers our proposed aggregation (denoted Original) average aggregation (denoted as Average) on GoogLeNet and ResNet50.
D.3 QUALITATIVE COMPARISON OF IMAGE DEBUGGING WITH NEUCEPT
We conduct qualitative experiment to compare the set of critical neurons identified by Vu et al. (2022) (the core concept neuron w.r.t the output logit of the model) and our set in the image de- bugging experiment. Specifically, following the setups in the experiment in section 5.1 identify the top r = 16 core concept neurons at layer 4.2 of ResNet50 for both methods, which are used determine the top-2 groups of core concept neurons for given misclassified image. Groups of neurons were identified following the methodology described in section 3.5 where the groups with the highest metric scores (defined in equation 5) are selected. Furthermore, quantify the contribu- tions of the selected groups to the model output, we mask all of neurons in each groups and measure the changes of probability of the final predictions. The higher the changes, the more "critical" the groups of neurons. We select three classes, without cherry-picking, namely: Bald Eagle, Great White Shark, and Bee (corresponding to the classes in figure and 9. The results are presented
in figure 13 [4. and 15
logit
Neurflow top concepts
Originalimage
NeuCEPT concepts
Top lagit drop
NourFoww
NeurFlow Maskadd muurnos
NeuC
NeuCEPT
Masieed
-Masked
Masleed 12 neurons
Figure 13:
The comparison of the top-2 groups of neurons with the highest metric score of our method and Vu at. 2022) on class Bald eagle. The top logit drop images of NeurFlow are more resemble the original concept
(.e. NeurFlow concept 1 vs NeuCEPT concept 1). And the prediction probability changes when masking core concept neurons more significant while masking fewer neurons.
17

---

Published as a conference paper at ICLR 2025
drop
concepts
Originakimege
concepts
Top logit drop
Hasking concept1 Neurflow Maskedd nourons
Masking and
Origina
Masking concept NeuCEPT -Mooked neurons
Masking land2
Neurflow
NeuCEPT Maalied 16 neurons
-Maskod 144 nouroes
Figure 14: The comparison of the top-2 groups of neurons with the highest metric score of our method and resemble the
Vu al.
2022 class Great white shark. The top logit drop images of NeurFlow original concept (i.e. NeurFlow concept NeuCEPT concept 2). And the prediction probability changes when masking our core concept neurons are more significant while masking fewer neurons.
NeurFlow
NeuCEPToop
Top
Moskinggonouptt
Husking concupt and NeurFlow neurons
Original
Maskingconongt
Maskingsonsopt and2
NourFlow meurons
NeuCEPT neurons
Figure 15: The comparison of the top-2 groups of neurons with the highest metric score of our method and Vu al. (2022) class Bee. The top logit drop images of both methods similar to the exemplary image of the concept. And, hoth methods ahle to alter the prediction of the model.
Qualitatively, observed that method identified the top-2 concepts closely resembling the original images. Additionally, our top logit drop images (i.e., "images showing the largest de- crease in the target logit value" as described in 5.1 better matched the representative examples of the identified concepts. Furthermore, masking the core concept neuron groups identified by our method resulted in more significant changes the prediction probabilities, using fewer neurons, compared to the groups identified by NeuCEPT (Vu et al. [2022). For instance, with the labels Bald Eagle and Great White Shark, masking NeuCEPT's core concept neurons had no effect on prediction proba- bilities, whereas masking the neurons identified by our method substantially altered the predictions. These findings suggest that our approach identifies more impactful neurons and concepts directly related to the model's predictions compared to NeuCEPT.
D.4 QUANTITATIVE COMPARISON OF CORE CONCEPT NEURONS OF THE MODEL OUTPUT
We run an experiment to further verify: although our method focuses on the set of core concept neurons w.r.t specific target neuron, our identified neurons also have strong influence to the perfor- mance of the model.
Specifically, we evaluate the overlaps between our core concept neurons and the critical neurons (which are specifically designed to find important neurons for the model output) determined by Khakzar al. (2021a) and Vu et al. (2022), then average the results across all layers of ResNet50 and GoogLeNet of 10 random classes. The numbers of core concept neurons are set to be the same for all three methods. We measure the F, scores of the overlaps, which are shown in table The
18

---

Published as a conference paper at ICLR 2025
Table 3: Overlapping ratio of critical neurons between NeuronMCT (Khakzar et al. 2021a), NeuCEPT Vu 2022). and core concept neurons NeurFlow
Overlap
NeuronYCTT-NeurFby NeuronMCT-NeuCEPT
NeurFlow-NeuCEPT
ResNet50
0.72
0.48
0.49
Googl.eNet 0.79
0.55
0.56
Table 4; Average subtraction of the losses. Negative means our loss better and vice versa
Model
Average Subtraction of the Losses
ResNet50
-0.082
GoogLeNet
-0.013
results imply that NeurFlow contains mostly similar core concept neurons to NeuronMCT while not directly identifying core concept neurons of the output.
D.5 QUANTITATIVE COMPARISON OF CORE CONCEPT NEURONS OF A TARGET NEURON
We assess our method of identifying core concept neurons given a specific target neuron with the method used in Cammarata et al. (2020). In Cammarata et al. 2020), neurons ranked based on the top neurons with the highest L2 weights connected to the target neuron. Note that this method not applicable in other experiments since calculating weight magnitude is limited to consecutive layers.
For this comparison, we identify the top (separated by one convolution layer, per the setup in Cammarata et al. (2020) using both methods. We then knock out these core concept neurons to observe how the target neuron's concept affected.
core concept neurons in two consecutive layers
The extent of this change is quantified by the loss function defined in [ where a lower loss indicates better performance. We randomly selected 100 neurons across different convolution layers from both models and caleulated the average difference in losses between the two methods. A negative result indicates our method produces a better loss, while a positive result indicates otherwise.
The results summarized in table These findings demonstrate that our method is more effective at identifying core concept neurons. Additionally, gradient-basee approaches are more versatile, as they can be applied to non-consecutive layers (e.g., ResNet Block 4.2 - ResNet Block 4.1 in experiments), whereas the Lz-weight-baune based approach is limited to consecutive layers.
D.6 DEPENDENCE ON THE CHOICES OF
The trade-off of the parameter In this experiment, aim to study the choices of parameter on the set of core concept neurons of a model. Specifically, in the experiment "Fidelity of core concept neurons"". the choice can be seen as trade-off between simplicity (the number of core concept neurons) and performance (the accuracy of the prediction when retaining only the core concept neurons). However, for r 4, the results are vary across our tested models. We conduct additional experiment to highlight that for sufficiently large the results are less dependent on the parameter.
We evaluate 10 different labels with the same setups in the experiment "Fidelity of core con- cept neurons" for 7 = 20, 24. The results in figure 16 show that with these higher values, the performance drops of the model become negligible. Furthermore, the differences between retaining for т = 20 and r 24 at all layers are minimal, suggesting that the dependence on 7 decreases as increase the value.
Completeness of core concept neurons on the output: Additionally, we run an experiment to assess the completeness of NeurFlow in identifying the important neurons for the model's output. By greedily adding 50% more neurons in each layer, of which the neurons are ranked by the importance scores defined in Khakzar al. (2021a). The higher the scores, the stronger the influence the
19

---

Published as a conference paper at ICLR 2025
ResNet50 single layer masking.
ResNet50 single layer retaining
Re mult-layer masking
ResNet50 mulb-ayer retairing
Accuracy
0.50
0.25
4.2
4.0
3.4
3.2
3.0
4.2
4.0
3.4
4.0
3.4
3.2
4.0
3.4
32
5.0
GoogLeNet single layer masking
GoogLetiet single layer retaining
GoogLeNiet multi mutbllaye masking
GoogLeNiet mutti mutt-layrr retaining
Accuracy
0.75
0.50
0.25
0.00
Layers
Figure 16: Effects of neuron groups model's performance for 16, 20, 24. The effect of increasing negligible most the layers both models.
Extended
Restlet 50 single layer masking
ResNet50 single layer retaining
Restier 50 mult-layer masking
ResNet 50 mult-layer retaining
1.00
Accuracy
0.50
0.25
4.2
4.0
3.4
3.2
3.0
4.2
4.0
3.4
32
3.0
4.0
3.4
3.2
3.0
4.0
3.4
3.
3.0
Googl WNet single layer masking
GoogLefest single layer retairing
GooogaNNet multi-ayer masking
GooglaNet multi-layer retaining
Accuracy
0.25
0.00
Layers
Figure 17: The comparison of the influences on models' performances of core concept neurons and the extended set core concept neurons
prediction of the model. We then re-run the "Fidelity of core concept neurons" for 16 (denoted as "CC") and its extended version (50% more neurons denoted as "Extended"). The results in figure 17 show that, for ResNet 50, adding non-core-concept neurons had almost effect improving model performance. For GoogleNet, only in the most critical case (where the retaining operation is applied up to layer 5b), adding 50% more con-cet nodes led to an improvement in model performance by 25% only at layer 5b in the retaining setup. These results show that when 7 is sufficiently large, our algorithm ensures completeness.
D.7 DEPENDENCE ON THE CHOICES OF k
To evaluate the dependence the results on the choice kt we conducted additional experiments with various values of & and measured the number of core concept neurons overlapping with the bascline setup of & = 50. Greater overlap indicates less dependence on the choice of
16 (i.e., the maximum number of core concept neurons per target neuron is 16) and & E {30, 40, 50, 60, 70, 90, 110, 130, 150, 170, 190), evaluated across 50 random neurons. The results show that for all tested values of kt the overlap ratio is always at least 14/16 (> 86%), demonstrating that the results our proposed algorithm are independent of the choice of k.
Table 5 summarizes the results with
20

---

Published as a conference paper at ICLR 2025
Table 5: The overlap of of concept neurons of different compared the baseline 50
40
50
60
90
110
130
150
170
190
GoogLeNet ResNet50
15.0
15.4
16.0
15.5
15.3
15.3
15.0
15.0
14.9
14.9
14.9
14.9
15.6
16.0
15.6
15.3
14.7
14.5
14.3
14.1
14.0
14.0
Alexnet
Googlenet
Resnet50
Percentage (%)
c5
62
112
19
f7
15
10
50
4e
4d
4C
46
3b
3a
1c
4.2
4.1
4.0
3.5
3.4
33
3.2
3.1
3.0
Layers
Figure 18: Illustration of the percentages of crop sizes in the concepts of concept
50 ran-
dom classes are assessed for three models and three different crop sizes. The layer names are abbreviated (e.g. ".g.:feature.122 to "f12").
D.8 MULTIPLE CROP SIZES AUGMENTATION
For target class our input dataset is created by randomly cropping the images that the DNN classified as class c, similar to Fel et al. (2023). However, since each neuron can detect feature at different granularity, different levels. Intuitively, small crop sizes indicate low level while large crop sizes indicate high level features. In our experiments, 100%, 50%, and 25% of the original dimensions. The cropping is performed using a sliding window with 50% overlap, resulting in roughly 2500 patches in total.
crop the images into multiple crop sizes in order to capture features at
crop the original images into patches of three different sizes-
Figure shows the percentages each crop size in the concepts of core concept neurons through- the networks. As demonstrated, lower layer's neurons often activated on small crop size images and vice versa. This aligns with the common believe that high level features are detected at the later stages of DNNs. This approach can be improved further by including more complex augmen- tation methods. However, in this work, our main focus is functional of groups of neurons and their interactions.
E DETAILED ALGORITHMS
E. IDENTIFYING CORE CONCEPT NEURONS AND CONSTRUCTING NEURON CIRCUIT
Algorithms 2 and provide detailed pseudocode for identifying core concept neurons, determin- ing the semantic groups, and constructing the neuron circuit respectively.
Algorithm Identifying core concept neurons
Input: Target neuron a, dataset D, constraint Output: Set of core concept neurons S. V f- arg max VC=
Evey (v)
T calculate T(a, 8i, Va), Vs; $
S, f select the top-r neurons with the highest ||]]]
return S
21

---

Published as a conference paper at ICLR 2025
Algorithm 2 Determining semantic groups
Input:
Neuron concept V.
Parameter: Max number of clusters Netuster
Output: Semantic groups Va.j Vj
r(mg) <- Calculate the representative vectors Vug E Va sccorsssseesseeeeeee -
best_n 4 Initialize
for Number of clusters n in ( .......,,eee, N......,Neuutteer ] do
V f- Agglomerative clustering with clusters on (r(v)), Vog V.) sil_score t- calculate the Silhouettes score given the results of clustering if best sil score < sil score then
best. sil score +- sil score best_nn end if' end for Vajj f Agglomerative clustering with best.n clusters on (r(v)), Veg E Va) return Vajj Vj E (b....,beett... best.n)
Algorithm 3 Forming neuron circuit
Input: Logit neuron ae, dataset D, constraint Output: Neuron circuit He
He < 3 St {a He He U.; for Layer in {L - 1...,, 1] do
S + 0
for Target neuron a in S do S: 4 St U Identify core concept neurons (Alg of
Va.j <- Determine semantic groups (Alg
Vj
w(s Va.) t T(a, st Va.)/ E.es, end for
| a, Vo.))l
H. + H. US;
end for
return He
F IMAGE DEBUGGING SETUP
For arbitrary input D. we want to which parts of are detected by the group of neurons G. Thus, we crop the image into multiple crops, similar to what we do Section 333 The crops, denoted U; are passed into the model to get the activations, which M (w, Sc. D), Ve,- Then we can set a threshold for each group, so that, the crops with the scores above the threshold can be visualized.
then measure the metric
However, since the metric can be greatly affected by only one neuron in the group (i.e one neuron with low activation leads low metric score), the metrie is prone outliers. Thus, only assess the metric on the subset & C Sg. In practice, S/ contains the top-5 neurons that are closest to the group's center, where each neuron represented as Te.. for neuron s; E Se with the semantic group's 's index j. The center of the cluster is the average of all representative vectors, and the distance between pair of neurons evaluated using la distance.
G MLLM PROMPT FOR AUTOMATIC CONCEPT LABELLING
In this section, we provide prompts for reproducibility. We employ two types of prompt, which are responsible either captioning the common concepts in the exemplary images of a neuron concept, or describing how NGC formed from NGCs at the preceding layers. Our prompts include three parts. Firstly, we provide a role for MLLM model, marked as role description. Secondly, the main prompt is presented where shows the general instruction for the task that MILMM should do. The
22

---

Published as a conference paper at ICLR 2025
role description and main prompt is the same for all setups. The last part is the answer form where we give specific instruction on how to generate appropriate captions and the template of the answer. The structure of the whole prompts are: Role description Main prompt Answer
G.
ROLE DESCRIPTION AND MAIN PROMPT
Role descriptions: "Act Image Captioning Language Model."
Main prompt:
"# Core Responsibilittes:
Analyze a set of similar images to identify common features.
Generate descriptive captions that highlight these common features.
You must adapt to detect both simple and complex features. # Important notes:
You don't have to generate captions for every image, focus on the common features.
Outliers exist in the images, you could ignore them if they are not relevant to the common theme.
You should describe the images with objective visual features, not subjective (like powerful or beautiful or scary etc., because these are only your opinion).
You should only describe visual features, not the context or the story behind the images.
You should keep succinet caption, keep it one or two sentences long, that only describe a few most common features.
# Role Summary:
Your role is provide accurate and coherent captions for set of similar images by identifying and describing common features. These features can range from simple elements like edges and colors to complex patterns such as specific object particular setting."
G.2 ANSWER FORM
Answer form for single concept captioning:
"# Answer form:
Common features: a list of features
Caption: your caption in one two sentences"
Answer form for describing NGC's formation:
# Key note of the input:
There are many different groups of images, make sure you get the number of groups right.
Each group of images has a common feature.
-The higher level feature is the first group.
Other groups are lower level features that combine to form the higher level feature of the first group.
Key note of the output:
You should only focus the common features of the images but also describe how the features from the lower level groups combine to form the higher-level feature of the first group.
You should focus on the common features that shared among both the high and low level.
"# Step by step:
Find the lists of common features in Group 2, .....
23

---

Published as a conference paper at ICLR 2025
For each feature from those lists: match it with the features in Group 1.
Some of the features in the lists might have no matches: they might be combined with others to form new features, match the features in Group I with some simple combination of the features in Group 2, .... N (e.g. blue and green + blue-green, multiple curve orientations + circle, two angle, etc.).
edges with different orientations
you don't find any visual features that match, please don't describe features that not presented, instead, you say "There is matches".
From the matched features, derive the common features in Group
Generate caption for Group 1.
Answer form:
Group 1 Common Features: list of common features
Group 2 Common Features: list common features
Group Common Features: list of common features
Feature Evolution:
Group 2: has feature A match feature in Group 1 (for Group to if there is no matches, please say "There is no matches"))
Group N: has feature B match feature in Group
Caption: one two sentences capturing the common features and their evolution"
24


Reviews:
[{"review_id": "7x9N5SZl1m", "reviewer": "Reviewer_gfeH", "summary": "Summary:\nThe presented manuscript  proposes  the framework NeurFlow which offer interpretability to neural networks by focusing on groups of critical neurons. The framework tries to detect critical neurons and clusters them based on shared functionality, constructing a hierarchical circuit to model their interactions. The importance of the selected neurons is validated by comparing their impact on the model’s performance to random groups of neurons. Finally the manuscript also offers potential applications, e.g to debug model biases.", "questions": "Questions:\n1)\tHow dependent are the results on the choice of the hyperparameter k in the neuron concept $\\mathrm{V}_a$ ? \n2)\tIn the “neuron clustering algorithm based on the semantic groups”, is the concept of the neuron $s_i$, namely $\\mathrm{V}_{s_i}$ a subset of the parent neuron’s concept $\\mathrm{V}_a$? Intuitively, if the neurons are indeed critical for neuron a this should be the case correct? From Algorithm 3 in supplemental material it seems that the neuron concept $\\mathrm{V}__{s_i}$ is identified independently of the parent’s concept. \n3)\tIn the experiment to detect the optimality of the critical neurons the authors mention: “For each setting we randomly select 50 target neurons (denoted by a_i) from 10 distinct classes”. Does this mean the authors select in total 500 (50 x 10 neurons) or are 5 neuron selected per class ?\n4)\tIn the experiments leading to Figure 4: Could the authors please indicate the confidence intervals as the results are averages over different selections of random groups of neurons ? It would be interesting to see the spread of the loss to note if some random groups of neurons reach near optimality. \n5)\tThe weight $W(G_i,G_j)$ in Equation 5 seems to be unnormalized with respect to the number of neurons in $\\mathbb{S_i}$ and $\\mathbb{S_j}$. Doesn’t this make it difficult to interpret these weights over different groups of neurons?\n6)\tHow are the random neurons selected in the masking/retaining experiment? Are the number of random neurons per layer selected in an equivalent amount as the number of critical neurons at that layer? Or is only the total number of random neurons matched to the cardinality of the set $\\mathbb{S}_a$ ?\n7)\tThe behaviour of the critical nodes in the multi-layer retaining experiment does not  seem to support the interpretation of critical nodes, as the performance significantly decreases (or fluctuates) with increasing layers.\n\nAdditional comments\nLine 056: “there” -> “their”\nLine 181: “our” -> “out”\nLine 250: “its all” should be replaced with “all its”\nLine 311: There seems to be a word missing from this sentence to make it coherent.\nFigure 4: The current choice of color & marker shape is not very intuitive to grasp the message of the figure. Using a consistent colour for critical/random and varying the marker between different values of $\\tau$ might make interpreting the figure more easy.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n•\tThe identification of groups of critical neurons offers a way to tackle a node’s polysemantic behaviour.\n\n•\tThe application of MLLMs to annotate group circuits is novel and original. This offers a nice way to explain a model’s decision.\n\n•\tThe papers is very complete and self-contained. The proposed methodology is tested on different level as the authors don’t only look at the dependence of the model’s performance on the critical neurons, but also assess the optimality of their selection with the additional “Optimality of critical Neurons” experiment. Additionally the authors explore applications of their proposed approach.", "weaknesses": "Weaknesses:\n•\tThe formal definition of a  group of critical neurons of is not obvious to me. Heuristically critical neurons are defined as neurons who, if knocked out, would result in a substantial change in performance. The formal definition however defines the critical neurons of neuron a as the group of neurons who, when knocked out, cause the neuron concept of neuron a ($\\mathrm{V}^{\\mathbb{S}}_a$) to be the most different from the original neuron concept ($\\mathrm{V}_a$), as the cardinality of the intersection will then be small. \n\n•\tThe description of the experiments are difficult to understand and it misses some important aspects to properly assess the results ( see questions below). The conclusion taken from these experiments are therefore not necessarily supported by the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "776RMgsbFF", "reviewer": "Reviewer_fHh7", "summary": "This paper argues that it is most useful to analyze neural networks through the lens of groups of neurons rather than individual neurons. It describes an algorithm for constructing such groups, which can be used to construct hypertrees of which neuron groups are involved in each class prediction. The algorithm works by creating a dataset of visual features (which I believe are different crops of different dataset classes), then for each neuron constructing the \"neuron concept\", which is the top K visual features that the neuron activates. The algorithm then looks for the critical input neurons to each neuron, which is defined as ones which would change the neuron concept if removed. (This is actually done via an importance score, since calculating this fully in set form would be too computationally costly). Then, one can construct a hypertree of neurons involved in predicting a class, based on finding the critical neurons of the final logit and then on downwards.", "questions": "- I was unclear on how the semantic grouping vector was calculated, and which feature maps were explicitly within each visual feature\n- Are visual features individual images? Images of a class? Some subset of images within a class? I was unclear on this, and it made it harder to understand the rest of the paper", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper engages with the problem of neuron analysis in a way that seems to pragmatically balance between the purist but unscalable approach of individual neurons, and the combinatoric explosion of looking at every possible circuit \n- The paper did rigorous analysis of both the correctness and the usefulness for interpretability of the neurons they find, as well as calling out explicit useful applications", "weaknesses": "The paper defines a lot of terms (critical neuron group, neuron concept, neuron circuit/hypertree, visual feature) that are in colloquial language somewhat similar, and I think it could have benefitted from a glossary figure that explicitly defines all of these with reference to one another, so that a reader can flip to it if needed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NVtRuuflin", "reviewer": "Reviewer_qMvj", "summary": "The paper proposes NeuroFlow, a framework designed to identify the set of connected neurons (i.e., circuit) that most strongly influence the predictions of a target class. The framework begins by clustering neurons based on their semantics and then reconstructs the circuit through interactions between clusters across the network structure. The framework's main steps are as follows:\n\n1. For a given neuron at layer L, the critical neurons for its output at layer L-1 are identified. These critical neurons are the ones whose removal causes the highest change in the top-k patches associated with the target neuron. This set is approximated by an algorithm that sums attributions (calculated using Integrated Gradients) for the identified top-k patches.\n2. The Top-k patches activated by the neuron are clustered into semantic groups using agglomerative clustering, where each cluster represents a semantic concept captured by the neuron.\n3. The algorithm calculates the contribution of each critical neuron in the preceding layer to each semantic group identified in the previous step.\n4. Semantic groups from the entire layer are clustered based on their similarity, and neurons that share at least one clustered semantic group are clustered together.\n5. A multimodal LLM labels the clustered semantic groups\n6. The concept circuit is generated using the connections and the attributions of the critical neurons within each semantic group.\n\nThe proposed framework is evaluated by assessing (1) the approximation accuracy of the heuristic used to compute critical neurons, (2) the impact of critical neurons with respect to a random baseline, and (3) the correlation between attribution scores and accuracy loss when neurons are removed. The paper also demonstrates an application of NeuroFlow to the problem of bias detection.", "questions": "Please check the weaknesses section. Several parts include direct or indirect questions for the authors and I would appreciate their answer and clarification.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- The paper is well structured, clear, and quite easy to follow (with some caveats noted in the weaknesses section) \n- The polysemantic nature of neurons is often overlooked in the current literature, particularly in works focused on circuits. The paper's attention to this topic, even if only partially addressed (see weaknesses), is highly relevant and of interest to the ICLR community.\n- Some of the characteristics of the frameworks are both novel and promising. In particular, step (3), the possibility for a neuron to be “active” in multiple clusters, and the connections between semantic groups are a clever way to deal with polysemantic neurons. \n- The integration of previously explored directions into a unified framework offers a fresh perspective on the topic.", "weaknesses": "The main weakness of the paper regards the evaluation and supporting experiments. There is a mismatch between the strong claims and the evidence provided. The current evaluation adequately validates the framework’s design choices. However, it does not place the work in the broader context of the ongoing research on this topic. Given substantial improvements in this direction, I would be willing to increase the score. \n\nIn more detail, several steps of the framework have been explored in prior literature, albeit sometimes with different objectives. Analyzing the connections between the outcomes of these steps and findings from other approaches would greatly strengthen the paper. Below are some specific examples of this point.\n- Step 1 identifies critical neurons. The way in which they are identified (in relation to the top-k patches) seems novel to me. However, the concept of critical neurons for a particular prediction or a class has already been studied in the literature. For example, authors of [1]  identify class-specific critical neurons. They found that a small subset of neurons, when removed, leads to significant shifts in prediction. Similarly,  (Vu et al., 2022) explore related concepts.  How do the findings in these papers relate to those in this work? Is there an overlap between their critical neurons and those identified here? If not, what could explain the differences?\n- The paper claims to “demonstrate that for a particular task, only a subset of neurons—referred to as critical neurons—have a substantial influence on the model’s performance”.  However, the experiments only show that, on average (but not always), the identified critical neurons have a greater impact than randomly selected neurons. This mismatch should be addressed either by providing additional experiments that prove that no other neurons outside the critical impact the class or by lowering the claim.\n- A large body of literature explores the extraction of circuits (like the ones cited in the paper). Although the circuits’ structures may differ, it would be valuable to compare their components. Do they share the same neurons? What important features are missing in prior approaches that the proposed framework captures? This would help clarify the unique contributions of this work.\n- The point raised by the authors about “groups of neurons within each layer also collectively encode the same concept” is quite well known. On this topic, a highly related work is [2], which identifies collaborative neurons responsible for the recognition of a concept. Similarly to the previous points, it would be great to see a comparison in terms of finding between this paper and the cited related work.\n\n\nOther weaknesses that didn’t impact my current score:\n- Polysemantic behavior is not limited to the highest activations. Several previous works indicate that neurons can recognize concepts at the lowest activations or across a range of activations [3,4,5]. However, the current framework relies upon and analyzes only the highest activations to compute the “visual features”. Therefore it is not guaranteed that the critical neurons extracted are the only ones contributing to the recognition of those concepts. Note that some alternative approaches [1,  (Vu et al., 2022)] for computing critical neurons are immune to this problem. Given the emphasis on polysemantic behavior, I encourage the authors to acknowledge this aspect as a limitation, a future direction, or to provide a more generalized formulation of the framework applicable to a broader range of settings.\n- Some terminology in Section 3.2 could be confusing for practitioners unfamiliar with the domain. Specifically, terms like \"neuron concept,\" \"top-kkk patches visual features,\" and \"top-kkk patches\" may cause confusion. For example, definition 1 defines the set of top-k patches as the “neuron concept”. That name implies that these patches represent a single concept, whereas, in later sections, they denote a set of visual features (or concepts) recognized by the neuron. Based on the equation and the paper's narrative,\"top-k patches\" would be a better term here.  The words “visual features” and patches are used interchangeably throughout the text, even if they are not exactly the same. Traditionally, “visual features” refer to the semantic: the same visual feature can be captured by several patches, so there could be five patches that identify one visual feature. For clarity, I suggest using \"patches\" where possible and reserving \"visual features\" for references to their semantic meanings. For instance, lines 201-205 would be clearer if the first and last occurrences of \"visual features\" were replaced with \"patches.\"\n\n[1] Amirata Ghorbani and James Zou. 2020. Neuron shapley: discovering the responsible neurons. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS '20). Curran Associates Inc., Red Hook, NY, USA, Article 497, 5922–5932.\n\n[2] Wang, A., Lee, W., & Qi, X. (2022). HINT: Hierarchical Neuron Concept Explainer. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10244-10254.\n\n[3] Biagio La Rosa, Leilani H. Gilpin, and Roberto Capobianco. 2024. Towards a fuller understanding of neurons with clustered compositional explanations. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 3082, 70333–70354.\n\n[4] https://transformer-circuits.pub/2023/monosemantic-features/index.html\n\n[5] Tuomas Oikarinen and Tsui-Wei Weng. Linear Explanations for Individual Neurons. ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ajZhmMDkSn", "reviewer": "Reviewer_KddZ", "summary": "The paper proposes a novel framework, NeurFlow, to enhance the interpretability of deep neural networks (DNNs) by focusing on critical neuron groups and their interactions, while existing works focus more on interpreting individual neurons. The framework proposes constructing hierarchical circuits encapsulating critical neuron interactions across layers, providing a more holistic view of model decision-making.\n\nThe proposed method can be summarized in the following steps.\n(1) Identifies critical neurons \n(2) Clusters these neurons into groups\n(3) Investigate the functions and interactions of these groups.", "questions": "Please address or explain the remarks in the above Paper's Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "• The paper introduces a novel way to identify semantic groups by leveraging clustering. By clustering neuron responses into semantic groups, the proposed method can interpret the target model with more specified concepts.\n\n• Interesting applications of the proposed method with easily understandable figures.\n\n•  The paper is well-written and easy to understand, except for a few parts.", "weaknesses": "Although the paper showed novelty and interesting results, I still have some concerns about this paper.\n\nMajor remarks\n\n1. Existing works like [1*] and [2*] investigate neuron groups as fundamental units for explaining the internal workings of deep neural networks. Especially, [1*] proposes a similar idea of identifying critical neurons from different layers with importance scores derived from integrated gradients, which should be compared. Also, what are the major strengths of this paper regarding [1*] and [2*]?\n\n2. This paper aims to enhance the interpretability of deep neural networks (DNNs). However, the proposed method only evaluated on CNNs (ResNet and GoogLeNet). Does the proposed method also work in other architectures like ViTs?\n\nMinor remarks\n\n1. How many random crop patches are needed to guarantee the stable performance of NeurFlow?\n\n2. Various ways of determining the importance score should be evaluated (i.e., various approximations of Shapley value) \n\n3. Existing works like [3*] identify neuron responses with highly activating crop and original images. It also investigates basic groups of neurons for decision-making, which should be referred to in the paper.\n\n4. In Figure 6, why do some layers(i.e., 2,3,4 in resnet and 3b, 4b, 4d in googlenet) have low correlations compared to the previous and subsequent layers? Can you share some insights regarding this result?\n\n5. Shouldn't it be \"knocking out these neurons\" In line 181?\n\nReferences\n\n[1*] Khakzar, Ashkan, et al. \"Neural response interpretation through the lens of critical pathways.\" CVPR 2021.\n\n[2*] Achtibat, Reduan, et al. \"From attribution maps to human-understandable explanations through concept relevance propagation.\" Nature Machine Intelligence 5.9 (2023)\n\n[3*] Kalibhat, Neha, et al. \"Identifying interpretable subspaces in image representations.\" ICML 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)