NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions

Authors: Tue Minh Cao, Nhat Hoang-Xuan, Hieu Pham, Phi Le Nguyen, My T. Thai
Primary area: interpretability and explainable AI
Keywords: Explainable AI, Functional Interactions, Critical Neurons, Concept Circuits

Abstract:
Understanding the inner workings of neural networks is essential for enhancing model performance and interpretability. Current research predominantly focuses on examining the connection between individual neurons and the model's final predictions, which suffers from challenges in interpreting the internal workings of the model, particularly when neurons encode multiple unrelated features. In this paper, we propose a novel framework that transitions the focus from analyzing individual neurons to investigating groups of neurons, shifting the emphasis from neuron-output relationships to the functional interactions between neurons. Our automated framework, NeurFlow, first identifies core neurons and clusters them into groups based on shared functional relationships, enabling a more coherent and interpretable view of the network’s internal processes. This approach facilitates the construction of a hierarchical circuit representing neuron interactions across layers, thus improving interpretability while reducing computational costs. Our extensive empirical studies validate the fidelity of our proposed NeurFlow. Additionally, we showcase its utility in practical applications such as image debugging and automatic concept labeling, thereby highlighting its potential to advance the field of neural network explainability.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730307518653

thread_type:
official_review

<thread_object>
creator: reviewer_kddz

<content>
summary:
The paper proposes a novel framework, NeurFlow, to enhance the interpretability of deep neural networks (DNNs) by focusing on critical neuron groups and their interactions, while existing works focus more on interpreting individual neurons. The framework proposes constructing hierarchical circuits encapsulating critical neuron interactions across layers, providing a more holistic view of model decision-making.

The proposed method can be summarized in the following steps.
(1) Identifies critical neurons 
(2) Clusters these neurons into groups
(3) Investigate the functions and interactions of these groups.

strengths:
• The paper introduces a novel way to identify semantic groups by leveraging clustering. By clustering neuron responses into semantic groups, the proposed method can interpret the target model with more specified concepts.

• Interesting applications of the proposed method with easily understandable figures.

•  The paper is well-written and easy to understand, except for a few parts.

weaknesses:
Although the paper showed novelty and interesting results, I still have some concerns about this paper.

Major remarks

1. Existing works like [1*] and [2*] investigate neuron groups as fundamental units for explaining the internal workings of deep neural networks. Especially, [1*] proposes a similar idea of identifying critical neurons from different layers with importance scores derived from integrated gradients, which should be compared. Also, what are the major strengths of this paper regarding [1*] and [2*]?

2. This paper aims to enhance the interpretability of deep neural networks (DNNs). However, the proposed method only evaluated on CNNs (ResNet and GoogLeNet). Does the proposed method also work in other architectures like ViTs?

Minor remarks

1. How many random crop patches are needed to guarantee the stable performance of NeurFlow?

2. Various ways of determining the importance score should be evaluated (i.e., various approximations of Shapley value) 

3. Existing works like [3*] identify neuron responses with highly activating crop and original images. It also investigates basic groups of neurons for decision-making, which should be referred to in the paper.

4. In Figure 6, why do some layers(i.e., 2,3,4 in resnet and 3b, 4b, 4d in googlenet) have low correlations compared to the previous and subsequent layers? Can you share some insights regarding this result?

5. Shouldn't it be "knocking out these neurons" In line 181?

References

[1*] Khakzar, Ashkan, et al. "Neural response interpretation through the lens of critical pathways." CVPR 2021.

[2*] Achtibat, Reduan, et al. "From attribution maps to human-understandable explanations through concept relevance propagation." Nature Machine Intelligence 5.9 (2023)

[3*] Kalibhat, Neha, et al. "Identifying interpretable subspaces in image representations." ICML 2023.

questions:
Please address or explain the remarks in the above Paper's Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730510853744

thread_type:
official_review

<thread_object>
creator: reviewer_qmvj

<content>
summary:
The paper proposes NeuroFlow, a framework designed to identify the set of connected neurons (i.e., circuit) that most strongly influence the predictions of a target class. The framework begins by clustering neurons based on their semantics and then reconstructs the circuit through interactions between clusters across the network structure. The framework's main steps are as follows:

1. For a given neuron at layer L, the critical neurons for its output at layer L-1 are identified. These critical neurons are the ones whose removal causes the highest change in the top-k patches associated with the target neuron. This set is approximated by an algorithm that sums attributions (calculated using Integrated Gradients) for the identified top-k patches.
2. The Top-k patches activated by the neuron are clustered into semantic groups using agglomerative clustering, where each cluster represents a semantic concept captured by the neuron.
3. The algorithm calculates the contribution of each critical neuron in the preceding layer to each semantic group identified in the previous step.
4. Semantic groups from the entire layer are clustered based on their similarity, and neurons that share at least one clustered semantic group are clustered together.
5. A multimodal LLM labels the clustered semantic groups
6. The concept circuit is generated using the connections and the attributions of the critical neurons within each semantic group.

The proposed framework is evaluated by assessing (1) the approximation accuracy of the heuristic used to compute critical neurons, (2) the impact of critical neurons with respect to a random baseline, and (3) the correlation between attribution scores and accuracy loss when neurons are removed. The paper also demonstrates an application of NeuroFlow to the problem of bias detection.

strengths:
- The paper is well structured, clear, and quite easy to follow (with some caveats noted in the weaknesses section) 
- The polysemantic nature of neurons is often overlooked in the current literature, particularly in works focused on circuits. The paper's attention to this topic, even if only partially addressed (see weaknesses), is highly relevant and of interest to the ICLR community.
- Some of the characteristics of the frameworks are both novel and promising. In particular, step (3), the possibility for a neuron to be “active” in multiple clusters, and the connections between semantic groups are a clever way to deal with polysemantic neurons. 
- The integration of previously explored directions into a unified framework offers a fresh perspective on the topic.

weaknesses:
The main weakness of the paper regards the evaluation and supporting experiments. There is a mismatch between the strong claims and the evidence provided. The current evaluation adequately validates the framework’s design choices. However, it does not place the work in the broader context of the ongoing research on this topic. Given substantial improvements in this direction, I would be willing to increase the score. 

In more detail, several steps of the framework have been explored in prior literature, albeit sometimes with different objectives. Analyzing the connections between the outcomes of these steps and findings from other approaches would greatly strengthen the paper. Below are some specific examples of this point.
- Step 1 identifies critical neurons. The way in which they are identified (in relation to the top-k patches) seems novel to me. However, the concept of critical neurons for a particular prediction or a class has already been studied in the literature. For example, authors of [1]  identify class-specific critical neurons. They found that a small subset of neurons, when removed, leads to significant shifts in prediction. Similarly,  (Vu et al., 2022) explore related concepts.  How do the findings in these papers relate to those in this work? Is there an overlap between their critical neurons and those identified here? If not, what could explain the differences?
- The paper claims to “demonstrate that for a particular task, only a subset of neurons—referred to as critical neurons—have a substantial influence on the model’s performance”.  However, the experiments only show that, on average (but not always), the identified critical neurons have a greater impact than randomly selected neurons. This mismatch should be addressed either by providing additional experiments that prove that no other neurons outside the critical impact the class or by lowering the claim.
- A large body of literature explores the extraction of circuits (like the ones cited in the paper). Although the circuits’ structures may differ, it would be valuable to compare their components. Do they share the same neurons? What important features are missing in prior approaches that the proposed framework captures? This would help clarify the unique contributions of this work.
- The point raised by the authors about “groups of neurons within each layer also collectively encode the same concept” is quite well known. On this topic, a highly related work is [2], which identifies collaborative neurons responsible for the recognition of a concept. Similarly to the previous points, it would be great to see a comparison in terms of finding between this paper and the cited related work.


Other weaknesses that didn’t impact my current score:
- Polysemantic behavior is not limited to the highest activations. Several previous works indicate that neurons can recognize concepts at the lowest activations or across a range of activations [3,4,5]. However, the current framework relies upon and analyzes only the highest activations to compute the “visual features”. Therefore it is not guaranteed that the critical neurons extracted are the only ones contributing to the recognition of those concepts. Note that some alternative approaches [1,  (Vu et al., 2022)] for computing critical neurons are immune to this problem. Given the emphasis on polysemantic behavior, I encourage the authors to acknowledge this aspect as a limitation, a future direction, or to provide a more generalized formulation of the framework applicable to a broader range of settings.
- Some terminology in Section 3.2 could be confusing for practitioners unfamiliar with the domain. Specifically, terms like "neuron concept," "top-kkk patches visual features," and "top-kkk patches" may cause confusion. For example, definition 1 defines the set of top-k patches as the “neuron concept”. That name implies that these patches represent a single concept, whereas, in later sections, they denote a set of visual features (or concepts) recognized by the neuron. Based on the equation and the paper's narrative,"top-k patches" would be a better term here.  The words “visual features” and patches are used interchangeably throughout the text, even if they are not exactly the same. Traditionally, “visual features” refer to the semantic: the same visual feature can be captured by several patches, so there could be five patches that identify one visual feature. For clarity, I suggest using "patches" where possible and reserving "visual features" for references to their semantic meanings. For instance, lines 201-205 would be clearer if the first and last occurrences of "visual features" were replaced with "patches."

[1] Amirata Ghorbani and James Zou. 2020. Neuron shapley: discovering the responsible neurons. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS '20). Curran Associates Inc., Red Hook, NY, USA, Article 497, 5922–5932.

[2] Wang, A., Lee, W., & Qi, X. (2022). HINT: Hierarchical Neuron Concept Explainer. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10244-10254.

[3] Biagio La Rosa, Leilani H. Gilpin, and Roberto Capobianco. 2024. Towards a fuller understanding of neurons with clustered compositional explanations. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 3082, 70333–70354.

[4] https://transformer-circuits.pub/2023/monosemantic-features/index.html

[5] Tuomas Oikarinen and Tsui-Wei Weng. Linear Explanations for Individual Neurons. ICML 2024

questions:
Please check the weaknesses section. Several parts include direct or indirect questions for the authors and I would appreciate their answer and clarification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679316949

thread_type:
official_review

<thread_object>
creator: reviewer_fhh7

<content>
summary:
This paper argues that it is most useful to analyze neural networks through the lens of groups of neurons rather than individual neurons. It describes an algorithm for constructing such groups, which can be used to construct hypertrees of which neuron groups are involved in each class prediction. The algorithm works by creating a dataset of visual features (which I believe are different crops of different dataset classes), then for each neuron constructing the "neuron concept", which is the top K visual features that the neuron activates. The algorithm then looks for the critical input neurons to each neuron, which is defined as ones which would change the neuron concept if removed. (This is actually done via an importance score, since calculating this fully in set form would be too computationally costly). Then, one can construct a hypertree of neurons involved in predicting a class, based on finding the critical neurons of the final logit and then on downwards.

strengths:
- This paper engages with the problem of neuron analysis in a way that seems to pragmatically balance between the purist but unscalable approach of individual neurons, and the combinatoric explosion of looking at every possible circuit 
- The paper did rigorous analysis of both the correctness and the usefulness for interpretability of the neurons they find, as well as calling out explicit useful applications

weaknesses:
The paper defines a lot of terms (critical neuron group, neuron concept, neuron circuit/hypertree, visual feature) that are in colloquial language somewhat similar, and I think it could have benefitted from a glossary figure that explicitly defines all of these with reference to one another, so that a reader can flip to it if needed

questions:
- I was unclear on how the semantic grouping vector was calculated, and which feature maps were explicitly within each visual feature
- Are visual features individual images? Images of a class? Some subset of images within a class? I was unclear on this, and it made it harder to understand the rest of the paper
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717669299

thread_type:
official_review

<thread_object>
creator: reviewer_gfeh

<content>
summary:
Summary:
The presented manuscript  proposes  the framework NeurFlow which offer interpretability to neural networks by focusing on groups of critical neurons. The framework tries to detect critical neurons and clusters them based on shared functionality, constructing a hierarchical circuit to model their interactions. The importance of the selected neurons is validated by comparing their impact on the model’s performance to random groups of neurons. Finally the manuscript also offers potential applications, e.g to debug model biases.

strengths:
Strengths:
•	The identification of groups of critical neurons offers a way to tackle a node’s polysemantic behaviour.

•	The application of MLLMs to annotate group circuits is novel and original. This offers a nice way to explain a model’s decision.

•	The papers is very complete and self-contained. The proposed methodology is tested on different level as the authors don’t only look at the dependence of the model’s performance on the critical neurons, but also assess the optimality of their selection with the additional “Optimality of critical Neurons” experiment. Additionally the authors explore applications of their proposed approach.

weaknesses:
Weaknesses:
•	The formal definition of a  group of critical neurons of is not obvious to me. Heuristically critical neurons are defined as neurons who, if knocked out, would result in a substantial change in performance. The formal definition however defines the critical neurons of neuron a as the group of neurons who, when knocked out, cause the neuron concept of neuron a ($\mathrm{V}^{\mathbb{S}}_a$) to be the most different from the original neuron concept ($\mathrm{V}_a$), as the cardinality of the intersection will then be small. 

•	The description of the experiments are difficult to understand and it misses some important aspects to properly assess the results ( see questions below). The conclusion taken from these experiments are therefore not necessarily supported by the experiments.

questions:
Questions:
1)	How dependent are the results on the choice of the hyperparameter k in the neuron concept $\mathrm{V}_a$ ? 
2)	In the “neuron clustering algorithm based on the semantic groups”, is the concept of the neuron $s_i$, namely $\mathrm{V}_{s_i}$ a subset of the parent neuron’s concept $\mathrm{V}_a$? Intuitively, if the neurons are indeed critical for neuron a this should be the case correct? From Algorithm 3 in supplemental material it seems that the neuron concept $\mathrm{V}__{s_i}$ is identified independently of the parent’s concept. 
3)	In the experiment to detect the optimality of the critical neurons the authors mention: “For each setting we randomly select 50 target neurons (denoted by a_i) from 10 distinct classes”. Does this mean the authors select in total 500 (50 x 10 neurons) or are 5 neuron selected per class ?
4)	In the experiments leading to Figure 4: Could the authors please indicate the confidence intervals as the results are averages over different selections of random groups of neurons ? It would be interesting to see the spread of the loss to note if some random groups of neurons reach near optimality. 
5)	The weight $W(G_i,G_j)$ in Equation 5 seems to be unnormalized with respect to the number of neurons in $\mathbb{S_i}$ and $\mathbb{S_j}$. Doesn’t this make it difficult to interpret these weights over different groups of neurons?
6)	How are the random neurons selected in the masking/retaining experiment? Are the number of random neurons per layer selected in an equivalent amount as the number of critical neurons at that layer? Or is only the total number of random neurons matched to the cardinality of the set $\mathbb{S}_a$ ?
7)	The behaviour of the critical nodes in the multi-layer retaining experiment does not  seem to support the interpretation of critical nodes, as the performance significantly decreases (or fluctuates) with increasing layers.

Additional comments
Line 056: “there” -> “their”
Line 181: “our” -> “out”
Line 250: “its all” should be replaced with “all its”
Line 311: There seems to be a word missing from this sentence to make it coherent.
Figure 4: The current choice of color & marker shape is not very intuitive to grasp the message of the figure. Using a consistent colour for critical/random and varying the marker between different values of $\tau$ might make interpreting the figure more easy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732076171774

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
We would like to thank the AC for securing four reviews with high quality.
We thank **Reviewer fHh7** for the accepting score.
We thank **Reviewers gfeH, qMvj and KddZ** for the detailed questions and very thoughtful comments, which help us highlight better the key contribution of our work.

Our responses to the Reviewers’ main questions are summarized below.
1. We distinguish the definition of *"critical neurons"* in our framework from those in the existing literature to highlight the conceptual novelty of our work **(Reviewers  gfeH, qMvj and KddZ)**.

2. We highlight the uniqueness of the circuit identified by our framework to showcase its contribution **(Reviewers qMvj, KddZ)**.    

3. We perform additional experiments to validate the efficiency of our framework **(Reviewers gfeH, qMvj, KddZ)**.  

4. We offer a more comprehensive explanation of specific algorithms and notations used in our framework, we also provide rationale behind our algorithm **(Reviewers gfeH, fHh7, KddZ)**.  

We welcome any follow-up questions from the reviewers regarding our rebuttal. We hope that, based on our detailed responses, the reviewers will consider increasing their scores if their concerns have been sufficiently addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732733041941

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the revised Manuscript and Supplementary Materials

comment:
Dear AC and Reviewers,
We extend our sincere gratitude to the Reviewers for engaging in thoughtful discussions with us and providing valuable feedback. Your insights have been instrumental in improving our work.

We have carefully addressed all comments and suggestions from the Reviewers, ensuring that their feedback is thoroughly incorporated into both the revised manuscript and the Appendix. 

All changes in the revised manuscript and supplementary materials are highlighted in blue.

We believe the updated submission reflects significant improvements, effectively clarifying and resolving the concerns previously raised. 

&nbsp;

Below, we summarize the key changes made to our revised manuscript and supplementary:

1. **Terminology Update** (Reviewers **gfeH, qMvj, KddZ**):

* We replaced the term **Critical neuron** with **Core concept neuron** to more accurately convey the role of these neurons (i.e., their significance in encoding concepts). This revision also avoids potential confusion with existing "critical neurons" in the literature, emphasizing the conceptual novelty of our work.

2. **Introduction Revision** (Reviewer **qMvj**, **KddZ**):

* The Introduction has been revised to highlight the distinction between our approach and existing methods. In particular, we now emphasize our focus on exploring inter-layer interactions of neuron groups, as opposed to examining the relationship between individual neurons and the model's output.

3. **Additional Experiments:**

We performed several new experiments to better differentiate our approach from prior works, including both quantitative and qualitative evaluations. These experiments include:
* Analyzing the impact of the parameter $\tau$ (**Lines 372-375; 393-399; Appendix D.6**) (Reviewer **gfeH**),
* Examining the influence of the parameter $k$ (**Lines 168-169; Appendix D.7**) (Reviewer **gfeH**),
* Comparing our defined edge weight (by summing) to the one using averaging (**Appendix D.2**) (Reviewers **gfeH**),
* Conducting qualitative comparisons between the core concept neurons identified by NeurFlow and the critical neurons detected by other methods (**Lines 502-508; Appendix D.3**). (Reviewers **qMvj**, **KddZ**) 
* Performing quantitative comparisons of NeurFlow with existing approaches (**Lines 419-431; Appendices D.4, D.5**) (Reviewer **qMvj**, **KddZ**),
* Evaluating the completeness of core concept neurons (**Line 392-399, Appendix D6**) (Reviewer **qMvj**) 
* Adding confidence intervals to the Optimality of Core Concept Neurons experiment results (**Figure 4**) (Reviewer **gfeH**). 
* Adding Shapley-based attribution method in our comparison of attribution method for edge weight (**Figures 10, 11; Appendix D.1**) (Reviewer **KddZ**)

4. **Enhanced Explanations:**

* To improve clarity, we incorporated additional explanations as suggested by all Reviewers (e.g., **Lines 160-161**  (Reviewer **fHh7** ), **Lines 289-290** (Reviewer **fHh7**), **Lines 310-313** (Reviewer **gfeH**), etc.). 
* We also included a table of notations in **Appendix A** (as suggested by Reviewer **fHh7**).

5. **Inclusion of Relevant Works:**
* We have incorporated all relevant studies highlighted by the Reviewers into the revised manuscript (**Lines 094-106**) and provided a clear explanation of how our proposed method differs from existing approaches (**Lines 126-130**).

6. We add a discussion on limitations and future direction as suggested by Reviewers **KddZ**, **qMvj**.

&nbsp;

We hope all Reviewers are satisfied with our revised manuscript and appendix.

We would be deeply grateful if you could kindly reconsider and increase our score, given the substantial improvements we have made in this revised version.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present NeurFlow, a novel framework designed to enhance the interpretability of neural networks by focusing on the interactions between groups of critical neurons. Unlike traditional methods that examine individual neurons, NeurFlow identifies core concept neurons: groups of neurons that collectively encode meaningful concepts and model their interactions across layers in the network. The paper introduces a hierarchical "concept circuit" that illustrates how these neuron groups contribute to the model's decision-making process.

The weaknesses of this paper include complex terminology, writing in the introduction, and insufficient experiments in quantitative and qualitative evaluations. However, during the rebuttal period, the authors actively engaged in discussions with the reviewers, and the final scores were (8, 6, 6, 6). All reviewers rated the paper positively.  Overall, this paper makes a valuable contribution to identifying critical neurons and addressing the polysemantic nature of neurons. It offers a fresh, well-structured perspective on interpretability, with promising, practical applications. So I recommend it for acceptance at ICLR.