Inner Information Analysis Algorithm for Deep Neural Network based on Community

Authors: Guipeng Lan, Shuai Xiao, Meng Xi, Jiabao Wen, Jiachen Yang
Primary area: learning theory
Keywords: inner information analysis, transparency, knowledge mining

Abstract:
Deep learning has achieved advancements across a variety of forefront fields. However, its inherent 'black box' characteristic poses challenges to the comprehension and trustworthiness of the decision-making processes within neural networks. To mitigate these challenges, we introduce InnerSightNet, an inner information analysis algorithm designed to illuminate the inner workings of deep neural networks through the perspectives of community. This approach is aimed at deciphering the intricate patterns of neurons within deep neural networks, thereby shedding light on the networks' information processing and decision-making pathways. InnerSightNet operates in three primary phases, 'neuronization-aggregation-evaluation'. Initially, it transforms learnable units into a structured network of neurons. Subsequently, these neurons are aggregated into distinct communities according to representation attributes. The final phase involves the evaluation of these communities' roles and functionalities, to unpick the information flow and decision-making. By transcending focus on single-layer or individual neuron, InnerSightNet broadens the horizon for deep neural network interpretation. InnerSightNet offers a unique vantage point, enabling insights into the collective behavior of communities within the overarching architecture, thereby enhancing transparency and trust in deep learning systems.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729695601318

thread_type:
official_review

<thread_object>
creator: reviewer_7wnu

<content>
summary:
This paper proposes the InnerSightNet algorithm, which clusters neurons in the same layer of MLPs and convolutional kernels in the same layer of CNNs from a community perspective. Through experiments that mask certain neuron clusters and perform perturbation analysis, it demonstrates that some clusters play a key role in task performance, some clusters can be pruned, and some clusters introduce noise.

strengths:
1. Novelty: The introduction of a community-based perspective for analyzing DNNs is original and provides a fresh viewpoint for understanding the role neurons play.
2. Comprehensive Framework: The multi-phase algorithm (neuronization, aggregation, evaluation) is well-structured, combining theoretical and practical insights.
3. Detailed Evaluation: The paper evaluates the communities formed in DNNs through accuracy drop tests and sensitivity analyses, showcasing the impact of each community on network performance.

weaknesses:
1. Scalability: The paper doesn't address the scalability of InnerSightNet on very large-scale neural networks. It is unclear how well the algorithm would perform or how computationally feasible it is for networks with millions of neurons.
2. Transferability: The experiments are only conducted on image recognition tasks. Can this method be applied to the interpretability of NLP models? The findings are limited to the functions of different clusters which follows conventional research pattern.
3. Usability: The mathematical theory of the algorithm is more solid compared to previous works, but the overall improvement in tasks like noise reduction and pruning is very limited.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730318339059

thread_type:
official_review

<thread_object>
creator: reviewer_rnf1

<content>
summary:
The paper proposes an algorithm designed to enhance the interpretability of deep neural networks by analyzing internal neuron communities. The approach transforms neurons into structured communities, aggregates them based on their functional attributes, and evaluates the information flow across these groups. The study applies the method to both linear and convolutional networks, demonstrating improvements in turning off noisy neurons.

strengths:
1. InnerSightNet introduces a unique perspective by focusing on community-based analysis of neurons, rather than on a single layer or individual neuron.
2. The three primary phases (neuronization, aggregation, evaluation) adaptively ensure the best number of communities.
3. InnerSightNet is shown to enhance interpretability and can be applied to network pruning to reduce model size while maintaining competitive performance.

weaknesses:
1. The paper compares InnerSightNet with methods such as Filan et al. (2021), Hod et al. (2021), and Liu et al. (2023) in the experiments. However, it does not sufficiently explain the methodologies of these baselines. A more detailed introduction to these approaches is needed to help the readers understand how InnerSightNet improves upon or differs from them.
2. While the method performs well on relatively small networks like those used in the MNIST and AFHQ datasets, it remains unclear how the algorithm scales to deeper networks.

questions:
1. What is the runtime of InnerSightNet for different network sizes? How fast the algorithm converges compared to baseline methods?
2. How scalable is InnerSightNet when applied to deeper networks like transformers？
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717454441

thread_type:
official_review

<thread_object>
creator: reviewer_ymgs

<content>
summary:
Review report: Inner Information Analysis Algorithm for Deep Neural Network Based on Community 
The present manuscript converts learnable units in Deep Neural Networks, such as kernels in CNNs, into networks and then groups the neurons into communities based on their representational attributes. To achieve this clustering, the authors develop the InnerSightNet. By subsequently analyzing the communities, the authors gain insights into the inner working of Deep Neural Networks such as relating certain communities to specific features in the underlying test data and detecting irrelevant neurons.

strengths:
Strengths 
-	The main idea of the manuscript, to group learnable units into a network of neurons and then analyze the communities within this network, is very clever and interesting.
-	The general approach to InnerSightNet is described very clearly. In particular, Algorithm 1 is helping in understanding the different steps.  
-	It is intriguing to see that the interdependence of different communities of neurons, such as c_1 and c_2 in the first layer of the CNN as reported in Table 1, is reflected in the 2-dimensional topological representation in Figure 3.

weaknesses:
Weaknesses
-	The different terms in Eq. (1) could be motivated more clearly, especially given the fact that this step is crucial to transform the kernels of a CNN in a structured network. (See also question below) 
-	While the general structure of InnerSightNet is described very clearly, the more detailed description is cumbersome to follow. Especially the aggregation step, which is heavy in notation but also lacks a definition of some variables such as l_d, \tau_{g_k, j} and s. 
-	The section “Community function analysis and finding key communities” is rather descriptive and only offers a shallow discussion regarding the explainability of community structures for MNIST. The discussion here could be greatly improved by more directly linking the observation from panel (a) and (b) together, as has been done with community c_10 in the first layer.

questions:
Questions 
-	In Eq. (1), could you motivate the distinction in positive and negative value regions more clearly as well as the choice to divide by the normalized cosine similarity, which is essentially either +1 or -1? Could you also specify what exactly data X_i refers to? 
-	Are the accuracy drops in closing communities in Figure 1a) and Table 1 reported with or without retraining the network? 
-	In Table 1, why don’t you report the accuracy after closing the corresponding communities for all possible combinations of the communities in the respective layers? For instance, in the first layer, why are c_0 and c_1 not closed together? Could you also close communities together across layers? 
-	Could you elaborate on the practical implications of your results, especially regarding parts of the model that fit to noisy data, cf. lines 458 until 464? 
-	In lines 1109 to 1112 you write “This phenomenon raises a question: in common sense, cat and dog images contain more information than handwritten digital, why is there actually less community division? Our explanation is ‘task-related’. Due to the fact that cat and dog classification is a binary task, the number of effective neurons for binary classification is indeed less than that for ten class tasks.” Would it be possible to simply restrict MNIST to classifying between 0 and 1 to test for this observation?


Additional comments
-	The axis and legend labels in the Figures are barely readable. 
-	In line 53, it should be “one aims” instead of “one aims”. 
-	In line 108, it seems inappropriate to cite a review paper to represent community detection in networks. 
-	In line 146, the citation “David et al. Bau et al. (2017)” seems to contain a typo. 
-	In line 161 and in line 188, the citation “Newman et al. Newman (2006” seems to contain a typo. 
-	In Definition 2, it would be helpful to distinguish 
-	In line 238, the citation “Lange et al. Lange et al. (2022)” contains a typo.
-	In Eq. (11), it is a bit unfortunate to use delta as a variable again as it also appeared in Eq. (1). 
-	In line 264, “sgn is the symbol function” was meant to be sign function? 
-	In line 323, the citation “Wanatabe et al. Wanatabe et al. (2018)” contains a typo. 
-	From line 370 onwards, c_10 should be changed to c_{10} in LaTex
</content>
</thread_object>

</thread>
<thread>
creation date:
1732028958783

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Thank you to PC, SAC, AC, and the reviewers ymgS, Rnf1 and 7WnU for your time and effort in reviewing this manuscript. Your valuable and constructive feedback is beneficial for us to improve the quality of our manuscript. Our author team carefully discussed your manuscript and provided a point-to-point response.

The significant of this problem is that although deep learning has achieved significant success, these networks often operate as black boxes with limited transparency. To address this issue, researchers have been exploring various strategies to uncover the internal workings of DNN, including visualization, model simplification, etc. By studying the community partitioning between neurons, identifying groups of neurons working together, and analyzing their roles throughout the network, InnerSightNet can gain a deeper understanding of how neural networks process information and make decisions. Few studies have explored the behavior and information flow of DNNs from a holistic community perspective. Therefore, it is necessary to further study from the perspective of the overall community. This is of great significance for identifying problems in the network, optimizing performance, and establishing trust in practical applications.

The output of InnerSightNet is neuron partitioning. Our analysis is relied on neural community. We shut down all neurons within a single community and measure the role of it based on changes in results of DNNs. In addition, we also analyze the neuron representation information within the neuron community. Based on these processes, we identify key neurons, analyze the role of neuron communities in the decision-making process, identify noisy neuron communities, and analyze the reasons for prediction errors, etc.

With best regards,

*Submission 36 authors*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732582303434

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised manuscript has been updated

comment:
Dear PC, SAC, AC, and the reviewers ymgS, Rnf1 and 7WnU,

Thank you for reviewing this manuscript and putting in the time and effort to improve it. With your help, the quality of the manuscript has been significantly improved. Due to the approaching deadline for submitting the revised manuscript, we are now submitting a revised version with highlighted changes. If there is any further clarification or explanation needed, we are more than happy to discuss it with you.


With best regards,

*Submission 36 authors*
</content>
</thread_object>

</thread>
<thread>
creation date:
1733295796269

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary our manuscript and rebuttal

comment:
### **Dear PC, SAC, AC,**

Thank you for reviewing this manuscript. As the deadline for the authors' response approaches, we have summarized this manuscript and rebuttal as follows.

***Summary of manuscript:*** InnerSightNet adaptively searches for community associations and explores information flow and decision-making. InnerSightNet is divided into three steps: **'neuronization-aggregation-evaluation.'** InnerSightNet broadens the horizon for deep neural network interpretation and offers a unique vantage point, enabling insights into the collective behavior of communities within the overarching architecture, thereby enhancing transparency and trust in deep learning systems.

***Summary of rebuttal:*** We are pleased to have a positive discussion with everyone regarding the improvement of this manuscript. The overall rating of this manuscript is **6, 6, 6**. After our rebuttal, reviewers ymgS and 7WnU have changed the rating from 5 to 6. Although Reviewer Rnf1 did not respond to our rebuttal, we still appreciate her/his valuable feedback on our manuscript.

We are pleased to receive the following **positive feedback** from the reviewers: The main idea of the manuscript, to group learnable units into a network of neurons and then analyze the communities within this network, is very clever and interesting. - The general approach to InnerSightNet is described very clearly. In particular, Algorithm 1 is helping in understanding the different steps. **(Reviewer ymgS)** -The paper proposes an algorithm designed to enhance the interpretability of deep neural networks by analyzing internal neuron communities.  -InnerSightNet is shown to enhance interpretability and can be applied to network pruning to reduce model size while maintaining competitive performance. **(Reviewer Rnf1)** -Novelty: The introduction of a community-based perspective for analyzing DNNs is original and provides a fresh viewpoint for understanding the role neurons play. -Comprehensive Framework: The multi-phase algorithm (neuronization, aggregation, evaluation) is well-structured, combining theoretical and practical insights. - Detailed Evaluation: The paper evaluates the communities formed in DNNs through accuracy drop tests and sensitivity analyses, showcasing the impact of each community on network performance. **(Reviewer 7WnU)**

In addition, we actively respond to the comments of the reviewers. We have corrected the formatting, font, grammar, and other errors in the manuscript (Proposed by Reviewer ymgS). We conduct new experiments to validate the performance and adaptability of InnerSightNet (Proposed by Reviewer ymgS, Rnf1 and 7WnU). We have added details to the article description to enhance its readability (Proposed by Reviewer ymgS, Rnf1 and 7WnU). **We believe that through revisions to the manuscript and thorough communication with the reviewers, we have effectively addressed their concerns. At the same time, the overall quality of the manuscript has also been significantly improved.**

The above is our summary of this article and rebuttal. Thank you again to PC, SAC, and AC for their time and effort in reviewing this manuscript.

Thanks and Regrads,

*Submission 36 authors*

----
### **Dear reviewers ymgS, Rnf1 and 7WnU,**

Thank you to all the reviewers for your time and effort in reviewing this manuscript. Our author team carefully discussed the suggestions put forward and made revisions accordingly. These valuable suggestions greatly improved the quality of the manuscript. We believe this is a productive academic exchange that has benefited us greatly. Thank you again to the reviewers for their valuable feedback on this manuscript.

Thanks and Regrads,

*Submission 36 authors*
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:
The paper introduces InnerSightNet, an algorithm for analyzing and interpreting the inner workings of deep neural networks through a community-based perspective. The key findings include:
- A three-phase approach  for transforming and analyzing learnable units within neural networks
- Identification of distinct neuron communities with specific functional roles
- Demonstration of how community analysis can improve model interpretability and enable efficient network pruning
- Evidence that certain neuron communities correspond to interpretable features in the input data
- Validation that removing noisy neuron communities can improve model performance

Strengths:
1. Novel Perspective: The community-based approach to analyzing neural networks provides a fresh and insightful way to understand network behavior beyond individual neurons or layers.
2. Well-Structured Framework: The three-phase algorithm combines theoretical foundations with practical implementation in a clear and systematic way.
3. Comprehensive Evaluation: The paper demonstrates the utility of the approach through multiple experiments, including accuracy drop tests and sensitivity analyses.
4. Practical Applications: The method shows promise for network pruning and noise reduction while maintaining performance.
5. Clear Presentation: The methodology is well-explained, particularly through Algorithm 1 and clear visualization of results.

Weaknesses:
1. Scalability Concerns: While the method works well on smaller networks, its computational feasibility for very large networks needs more exploration.
2. Limited Scope: Initial experiments focused mainly on image classification tasks, though authors later demonstrated applicability to other domains.
3. Baseline Comparisons: The paper could have provided more detailed explanations of baseline methodologies for clearer comparison.
4. Runtime Performance: The method shows longer runtime compared to baseline approaches, though this is justified by improved performance.

Reasons for Acceptance:
1. Innovation: The paper presents a novel and well-thought-out approach to neural network interpretation through community analysis.
2. Theoretical Foundation: The mathematical framework is solid and well-justified.
3. Practical Utility: The method demonstrates concrete benefits in terms of network pruning and performance improvement.
4. Comprehensive Response: Authors thoroughly addressed reviewer concerns during the rebuttal period with additional experiments and analyses.
5. Strong Evaluation: The experimental results validate the method's effectiveness across multiple scenarios.