Inner Information Analysis Algorithm for Deep Neural Network based on Community
Authors: Guipeng Lan
Shuai Xiao
Meng Xi
Jiabao Wen
Jiachen Yang
Venue: ICLR 2025 Poster (2025)
Keywords: inner information analysis
transparency
knowledge mining
Primary area: learning theory

Abstract: Deep learning has achieved advancements across a variety of forefront fields. However, its inherent 'black box' characteristic poses challenges to the comprehension and trustworthiness of the decision-making processes within neural networks. To mitigate these challenges, we introduce InnerSightNet, an inner information analysis algorithm designed to illuminate the inner workings of deep neural networks through the perspectives of community. This approach is aimed at deciphering the intricate patterns of neurons within deep neural networks, thereby shedding light on the networks' information processing and decision-making pathways. InnerSightNet operates in three primary phases, 'neuronization-aggregation-evaluation'. Initially, it transforms learnable units into a structured network of neurons. Subsequently, these neurons are aggregated into distinct communities according to representation attributes. The final phase involves the evaluation of these communities' roles and functionalities, to unpick the information flow and decision-making. By transcending focus on single-layer or individual neuron, InnerSightNet broadens the horizon for deep neural network interpretation. InnerSightNet offers a unique vantage point, enabling insights into the collective behavior of communities within the overarching architecture, thereby enhancing transparency and trust in deep learning systems.

Published as a conference paper at ICLR 2025
INNER INFORMATION ANALYSIS ALGORITHM FOR DEEP NEURAL NETWORK BASED ON COMMUNITY
Guipeng Lan, Shuai Xiao; Meng Xi, Jiabao Wen & Jiachen Yang
School of Electrical and Information Engineering Tianjin University Weijin Road Campus: No. 92 Weijin Road, Nankai District, Tianjin, China {IgP, xs611, ximeng, Wen.Jiabao, yangjiachen) en
ABSTRACT
Deep learning has achieved advancements across a variety of forefront fields. However, its inherent 'black box" characteristic poses challenges to the com- prehension and trustworthiness of the decision-making processes within neu- ral networks. To mitigate these challenges, introduce InnerSightNet, ner information analysis algorithm designed to illuminate the inner workings deep neural networks through the perspectives of community. This approach is aimed at deciphering the intricate patterns of neurons within deep neural networks, thereby shedding light the networks' information processing and decision-making pathways. InnerSightNet operates in three primary phases, 'neuronization
in-
aggregattion-evaluatio nneuomizzttoonaggggeatiinnoovv a structured network of neurons. Subsequently, these neurons are aggregated into distinct communities according to representation attributes. The final phase in- volves the evaluation of these communities' roles and functionalities, to unpick the information flow and gecision-makingg By transcending focus on single-layer individual neuron, InnerSightNet broadens the horizon for deep neural network interpretation. InnerSightNet offers a unique vantage point, enabling insights into the collective behavior of communities within the overarching architecture. thereby enhancing transparency and trust in deep learning systems.
Initially, transforms learnable units into
INTRODUCTION
Deep learning has been instrumental in driving advancements across range of domains, such as image recognition He et al.] (2016); Lan et al. (2024), natural language processing [Chowdhary & (2023). Despite their impressive performance on diverse tasks, these networks typically operate as black-boxes with limited trans- parency. To address this issue, researchers have been actively investigating various strategies to demystify the inner workings of deep neural networks. These include visualization, model simpli- fication, attribute specific features, etc. The focus of this work to delve into the micro-structure of deep neural networks, particularly investigating the community partitioning among neurons. By identifying collaborative neural communities and analysing their role throughout the entire network, gain deeper understanding of how neural networks process and make decisions internally.
Chowdhary (2020), and reinforcement learning Moerland
The information flow and decision decision-making are controlled by complex non-linear interactions among parameters. Effective inner information analysis is crucial for identifying problems, correcting rors, and enhancing understanding. Inner information analysis can be defined as a method to de- scribe the computational processes of deep neural networks in way that is understandable to hu- mans. In previous works, researchers have presented theories and tools from various perspectives, including weights, neurons, subnetworks, and latent representations. Regarding weights, ones train weight masks to determine which are important for specific tasks WWrtsman et al. (2020): Csordás al. (20TT. This method commonly used for network pruning Blalock et al. al.] (2020); Frankle & Carbin (2018) to eliminate redundant neurons. Concerning neurons, a common evaluation method involves dataset-based analysis to identify neurons with maximum activation characteristics Zhou
"Corresponding author

---

Published as a conference paper at ICLR 2025
et al. 20 Bau et al. (2017 2018b). Additionally, establishing causal relationships between net- work behaviour and individual neurons by perturbing them a common method Hod et al. (2021), ));
Bau et al. (2018)). In terms of subnetworks, modularity is universal principle that enables mod- els to be understood by their independent parts Watanabe et al. 2018); Ruggeri et al. (2023). By analysing subnetworks,
aims to uncover hierarchical structures. Regarding latent representa- tions, researchers infer the reasoning process of deep neural networks by analysing similarities with group of learned "prototypes' [Alvree Mclis & Jaakkola (0118;; Chen et al. (2019). [engio et al. 2013); Koh et al. (2020) elucidate the behaviour of inner information by decoupling latent repre- sentations. Fong & Vedaldi (2018): Kim et al.) (2018) explain the encoding of inner information by inducing images in conceptual datasets. Although the aforementioned works provide new insights into understanding deep neural networks, few studies have explored the behaviour and information flow from holistic community perspective. This limitation restricts our ability to fully compre- hend the inner mechanisms governing deep neural networks, particularly how these mechanisms contribute to the networks' decision-making processes. Consequently, further research is necessary to investigate the behaviour and information flow from a community wiide perspective.
To enhance understanding of neural networks, we propose InnerSightNet, novel approach that examines networks from a community perspective rather than focusing on individual neurons or layers. InnerSightNet operates in three phases: neuronization (structuring learnable units into neu- rons), aggregation (grouping neurons into communities), and evaluation (analyzing their roles and interactions). This framework reveals intricate patterns of information flow and decision- decision-making, offering deeper insights into the functions and interactions of these communities.
Our contribution We summarize the contribution of this work. Algorithm We propose community-based inner information analysis algorithm for the transparency of deep neural net- works, which broadens the perspective of deep neural network interpretation. We use the informa- tion representation of neurons to explore the community clustering effect in deep neural networks. In addition, this paper provides the mechanisms for analysing the role and function of the commu- nities. Applications We apply InnerSightNet to commonly used structures in deep neural networks: linear neural networks and convolutional neural networks, and prove its effectiveness.
2 RELATED LITERATURE
The quest to demystify deep learning models has led extensive research into various interpretabil- ity methods. We review in the field of deep learning interpretability, internal information analysis, community detection and the unique aspects that set our InnerSightNet apart from existing methods.
Interpretability of Deep Learning: Deep learning has achieved remarkable success across mul- tiple domains; however, their 'black box' nature has prompted researchers to develop techniques aimed at improving their transparency. Visualization methods, such as Grad-CAM /Selarrauu et al. (017)) and saliency maps Adebayy et al. (2018), have been widely used to highlight the regions that contribute most to the network's 's output, thus providing insights into the model's reasoning pro- cess. In additional, model simplification techniques, including pruning and quantization, have been proposed reduce the complexity of neural networks, making them
interpretable. Feature attribution methods like LIME Zhho et al. (202T): )) Marvin et al. (202)) and SHAP SHAP[Lundberg & Lee (2017); Bordt & von Luxburg (2023) further contribute to this effort by assigning importance scores to individual features, thereby elucidating how specific inputs influence the model's 's predictions.
Internal Information Flow and Representation Analysis: In addition visualization and simpli- fication, some work focuses on analyzing the internal information flow of neural networks. This type of analysis typically involves evaluating the importance of weights and neurons to identify the com- ponents that are crucial to performance. Weight masking techniques play a crucial role in network pruning and optimization by training specific weight masks to determine which weights are most important in particular task BBlllokk et al. (2020); Frankle & Carbin (2018). In addition, neuron activation analysis helps to understand which parts have greatest impact on a specific task by de- tecting which neurons in the network exhibit the greatest activation. Latent representation analysis provides an understanding of how neural networks process and store information by studying the la- tent variables and eigenvectors within the network Bengio al. (2013): Koh et al. (020). Prototype inference helps reveal the classification and decision- becision-making basis of the network by comparing its output with the prototype of known concepts Alvarez Melis & Jaakkola (2018); Chen al. (2019).
2

---

Published as a conference paper at ICLR 2025
Community detection: is concept in network science Bedi & Sharma (2016). Elali & Rachid (023), Goodley et al.J (2024). aimed at identifying connected subgroups in a network, where the connections within these subgroups are denser than those with external nodes. Recently, commu- nity detection has been introduced into the analysis of GNN, becoming an innovative means of understanding the internal structure [Sun al. (202T); y [Su et al. (2022). These methods provide new perspective for understanding the modularity of graph by identifying closely related groups of nodes that work together achieve specific functions. Traditional community detection, such as modular optimization Que et al. (2015); Traag et al. (2019) or clustering methods Li et al. (2021); Van Lierde et al. (2019), has been improved and adjusted to adapt high- digh-siienni,nal, nonlinear features. The information exchange between neurons within neural network can be conceptual- ized as a specialized unidirectional graph, where neurons serve as nodes and the information flow serve as directed edges. This inherent structure makes the application of community detection in the internal analysis of neural networks both logical and effective.
In summary, the existing methods advance understanding of neural networks, they fail to capture the dynamic interactions and functional roles within neuron communities. InnerSightNet addresses these limitations, and enable more nuanced exploration of community and information flow.
3 INNERSIGHTNET
InnerSightNet adaptively searches for community associations and explores information flow and decision-making. To achieve this, InnerSightNet neuronizes learnable units and aggregates neu- rons with similar roles as communities based on the input-output representations. InnerSight- Net represents an iterative algorithm that finds the best community allocation through continuous "aggregation-evaluttio" InnerSightNet is divided into three steps: 'neuronization-aggeeaalll evaluation." The details are as follows. (We provide Theoretical background Appendix: A.2)
3.1 INNERSIGHTNET: NEURONIZATION
We emphasize that structure plays pivotal role in information flow and decision-making, We classify the fundamental structures into two categories: learnable units (convolutional layer, linear layers, etc.) and invariant non-linear units (normalization layers, pooling layers, activation functions, etc.). The invariant non-linear units are treated as consistent non-linear function, unaffected by any alterations in the weights. Consequently, our study focus on the learnable units.
linear layer consists of multiple hidden nodes, functioning as idealized neurons. In contrast, a convolutional layer, composed of kernels, outputs 2D representations, with each kernel analogous to neuron. Based on the continuity of convolutional kernel outputs revealed by Bau et al. (2017), we quantify the correlation between the p-th kernel in layer d, nig, and the q-th kernel in layer + 1, re d++1 as shown in Eq.
N KL, (nttit) + KLi ( (d4+1)) .ig
1
.Kg (re nd41 )/ ( (nef, (d+1)
8(p. g)
N
where i represents the calculation under data X,, x and K- respectively represent the positive and negative value regions in the extracted information, 0 represent the cosine similarity, and KL represents the caleulation of KL divergence between two representative information.
3.2 INNERSIGHTNET: AGGREGATION
The community aggregation algorithm is predicated on following foundational principles: Newman (006) argue that using probabilistic mixed models and expectation maximization algorithm can detect a wide range of structural types without prior knowledge.
Preliminaries: Let's establish some foundational concepts firstly.
Definition 1: A connection weight between neurons can be categorized as an activation connection if its value is positive, inhibition connection if its value is negative.
3

---

Published as a conference paper at ICLR 2025
Definition 2: A connection is considered valid if its activation weight exceeds a certain threshold E1, or if its inhibition weight is below a certain threshold E2. We denote valid connections by the value 1. Conversely, connection is considered invalid if its activation weight is less than E3. its inhibition weight is greater than ς1. Invalid connections are denoted by the value 0.
Let A 1.d (1 denote the activation (inhibition) connection between the k-th neuron in layer and the i-th neuron layer 1. Similarly, let k, 1..4+1 (I denote the activation (inhibition) connection between the k-th neuron in layer and the j-th neuron in layer d + 1. Therefore, we define four connection matrices: A 1,d 1a-1,d Ad.dt,, and [d,4+1 For clarity, these denoted A, I, A', and I' respectively.
Definittion The probability of connection refers to the likelihood that a specific connection be- tween neurons is valid. We define Th and TH,5 as the probabilities of the activation and inhibition connections between the k-th neuron in layer and the i-th neuron in layer d-1 being valid, respec-
tively. Similarly, Thi and Th. denote the probabilities of the activation and inhibition connections between the k-th neuron in layer d and the j-th neuron in layer d + being valid.
Definition 4: We define ne the probability that neuron belongs the c-th community within the network, where Σ Te =
T5, and T/ as the probabilities of the activation and inhibition connections between the k-th neuron of are the probabilities the activation and TA and Tk.j
layer d and the i-th neuron in layer inhibition connections between the k-th neuron layer and the j-th neuron in layer +
The
Newman 2006) pointed out that the standard framework for fitting such models to given dataset is likelihood maximization. To address this issues, we enter g = {9t) for calculating the expected log likelihood estimation. 9& represents the community assignment of the k-th neuron in the d-th layer. The probabilities of A, I, and g can be expressed by Eq.
Pr(A,T,A ,913, 7
-P A I
9, a/T ot a sxr)
Pr(gIT,
where the two factors in Eq shown in Eq
and Eq 4
Pr(A,1,,, I, T 7 ( AAi (1 -i.ga )1-Asi (Tige )ss Fi 11-1
H&l 1.92
(3)
HAT (TA Ahs - A - (T) ge.)
Joi g6.5
1 11-1e
gg.
Pr(gIT, AA r', T (
-I
(4)
The expected log likelihood estimation Eg on = {gx} can be obtained by Eq.
1 In Ta
- - Er-gp
(5)
As.c represents the probability of that the k-th neuron is assigned to c-th community. la is the number of neurons in d-th layer. According to Bayesian, can conclude that Ok.e is represented as Eq. 6
-l T rdd x(t) Pr(A,T, A', I' ,
Pr(A,I,A
(6)
gk.c
Through Eq.
Eq. we maximize Cg solve for the best assignment of neurons. Using the Lagrangian multiplier method, get ok,es Tea T The Th, and 7 Eq.
4

---

Published as a conference paper at ICLR 2025
Pk.c
Ex tkee
,
Σx As..
4k,c
Es Pk.c
Te
ld
Σx Iike
En A..9
En I
7/
Σn Ds.0
Σx qhe
Σx Dk.c
where Pk.c is shown as Eq.
8 is equal to e or
( A Aus - -1pp = 1 (T fea ( - T (1pp)~1b00))
Aus
Phop Fee
(8)
11 o (1- Alr
1 (T/ Frns (1 7/ = [1-fa
See appendix A.3 for detailed inferential proof process.
3.3 INNERSIGHTNEN EVALUATION
To confirm the best number of communities, we introduce Q. Q determines whether the division is the most reasonable by measuring the consistency of input-output and structure-related metric.
Similarity based on input sensitivity: Inspired by Lange et al. (2022). we measure the similarity in Jacobian matrix. where e, is the input, and is the layer index. The i-th and j-th column in the Jd. -d T Rexx is the similarity measure of sensitivity between the i-th and j-th neurons in layer d when the input sample z,, The sensitivity similarity between the i-th and j-th neurons
input sensitivity of individual neurons in community. Let J.. be the
(i,( =
1 EE K
Jd.
Já (6.2)
where K is the sample number of test set. The similarity of neurons within community can be recorded as: S.n = Σ (inii.)/nnn - 1), Vi,j E c, and i i jj.
Similarity based output representation: To measure the consistency of outputs within com- munity, we introduce a new statistic called normalization consistency score (ncs). Firstly, we caleu- late the average output representation Fout of each neuron. Then, calculates the output Fout of each sample after input into the deep neural network, and get the standard deviation s.
K Σ K k= morm(Fo... Fout 2
(10)
$
where norm a normalization process, in order to ensure that nes only measures the consistency, and is not affected by the values. 1/(s+1) converts the nes a consistency score between 0 and 1. The sign consistency metric I is introduced to consider the directionality of numerical deviations.
r ElFout - Fouul sigm(Fout Fout)
(11)
El Fout!
where sign is the sign function. Sout
nnc.
[T| ncs.
Score based on Structure: havior within a community and differences between communities. generates four connection ma- trices using varying weight thresholds. A function u(c;, 0j) calculates the number of shared connec- tions between & and ej. measuring intra-community similarity (when = j) and inter-community differences (when i + j). Subsequently, we iterate over each pair of communities (c,, C;), calculate and construct two connection matrices Uact and Uinh, which the size is i x é. Each element in the matrix represents the common connection between the corresponding communities. Sytru needs to represent the activation and inhibition connections of weights, Satry = Suct + Stinh where the factors are calculated as shown in Eq. 12
Sytrus modularity variant, evaluates the consistency of neuronal be-

---

Published as a conference paper at ICLR 2025
jactjinhhC cq) Vacs/inn +
E, Jactt/iih (Cs, og)
gactf stru
-
)2)
(12)
E Wee/finh +
where n the number of communities, /n is the sum of all elements. ( 10-6 avoid the denominator to 0. The evaluation indicator Q can be calculated by weighting Sin- Sout and Sstru-
Q -c (aS + + + 3S
(13)
where we; = len(c))/ Σ len(c;) is the weight. len(c;) represents the number of neurons in ç₁- and B the hyper-parameters trade-off the Sin. Sout and Sxtra. We demonstrate the three steps of the InnerSighiNet: 'neuronization', "aggregation', and 'evaluation', as shown in Algorithm 1.
Algorithm Inner Information Analysis Algorithm for Deep Neural Network based on Community
1: Input: deep neural network N
2: Output: optimal neuron community partition
3: if N is convolutional layer then Neuronize(N)
4:
b According to Eq.
5: end if
6: Initialize optimal evaluation metric Qupt <- -00
7: Initialize optimal community count Topt
8: for 4 to 20 do
9:
for iter to 200 do
10:
Perform EM algorithm to update community partition Randomly initialize model parameters Te, Tot Tes Tof Tej"
and
According Eq.
Initialize probability matrix (kc(k,, c)
According Eq.
13:
Compute probability gko(k,
14:
Update model parameters based on (ko(k, e) end for
15:
16:
Q +
Calculate evaluation metric Q value
According to Eq. 13
17:
if Q Qum then
18:
Qopt + Q
19:
Topt r-
20:
Gkcopt + Qkc end if
22:
end for
return community partition for Topts 4keopt
EXPERIMENTS & APPLICATIONS
We use differential output analysis and perturbation statistical analysis analyze the results of InnerSightNet. For more details, please refer to appendix A.5.
4.1 NNNERSIGHTNET IN LINEAR LAYER AND CONVOLUTIONALL LAYER
We use InnerSightNet to analyse two typical structures: linear layer and convolutional layer. We differential output analysis and perturbation statistical analysis (an improved method based Watanabe et al. (2088) to explore the representation of community.
4.1.1 INNERSIGHTNET IN LINEAR LAYER
Overview: During our investigation into the InnerSightNet on linear layers, we focus on a classic benchmark: the MNIST recognition task, problem of classifying into 10 categories. We design a linear neural network Rosenblati (1958) with three hidden layers, with 128, 64, and 32 hidden nodes respectively, and the output layer is 10 vector. The MNIST dataset Letas et al. (1998)

---

Published as a conference paper at ICLR 2025
Jayer
layer 2
layer 3
Digits
Communities
(4) Accuracy drop after closing corresponding community
Communities
(b) Perturbattion-statiiiiaa analysis results
Figure 1: (a) Accuracy drop after closing corresponding community. (b) Perturbation-statistiin analysis. Note: to clearly show the attention of each community, we do not normalize the colorbar.
consists of handwritten digits, comprised of 60,000 samples within the train-set and an additional 10,000 samples in the test-set. Each sample takes the form of a 28 x 28 pixel grayscale image, where each pixel's value, ranging from 0 to 255, signifies varying levels of color intensity.
We conduct InnerSightNet on the well-trained linear neural network. Due to the hidden nodes natu- rally play the role of neurons, directly adopt the 'Aggregation' to identify various communities. We determine the ideal number of communities for the three hidden layers of the network as 14, 15, and 10 based on value. To clarify the specific impact of each community on output accuracy, we use differential output analysis. Specifically, we shut down all neurons in the community at once and record the decrease in accuracy, shown in Fig. (a). We also use perturbation-statistee analysis to visualize the sensitivity of each community to input, as shown in Fig. (b).
Community function analysis and finding key communities: According to Fig. (a), we observe Cg in the first layer is crucial for identifying digit 3. When it is removed (without re- re-training), the accuracy decreases by 32.57%, and the impacts on digit and are significant, with accuracy decreases by 5.37% and 5.72%. This indicates may be responsible for extracting certain shared geometric features, such as curves and angles. Ci in the second layer crucial for recognizing digit O, with an accuracy decrease of 15.92% C1 captures global closed shapes such as circles. From (b), it can be seen that C1 in second layer is more sensitive to the edges such structures.
In terms of specificity analysis, in the second layer, see that oş has significant impact on identifying digit 1, 2 and 3 (with a decrease of 2.73%, 3.00%, and 5.35%). This indicates 0g can capture the vertical diagonal features of these digits. The 0o of the second layer has significant impact on digit 7 and 8, with a decrease of 3.89% and 1.03%. This may indicate 02 has a high sensitivity to the combined shape of vertical lines and curves. The Cio in the first layer has the greatest impact on digit 5 with an accuracy decrease of 7.6%, and the impact on digit 6 is second, with an accuracy decrease of 5.53%. The accuracy of digit 8 has decreased by 3.39%, and the accuracy of the digit 3 has decreased by 2.77%. This indicates that Cio may have captured common specific features of the digit 5, 6, 8 and 3, such as the curve the lower right area. From the perturbation-statisiici
analysis in Fig. Fig. (b). it can be seen that C10 in the first layer is more sensitive to the curve in the lower right region. In the second layer, cp has a significant impact on the digit 1, 2, and 3. After closing the community, the accuracy decreases by 2.73%. 3.00%, and 5.35%. This indicates that 6G captures the common feature of digit 1, 2, and 3, which is the vertical line segment (digit is entirely composed of vertical lines, the top and bottom of digit 2 are usually connected by a vertical line, and the upper and lower arcs of digit 3 visually form an implicit connection through the vertical symmetry in the middle). From the perturbation statistical analysis
7

---

Published as conference paper at ICLR 2025
in Fig. (b), it can be seen that C5 is more sensitive vertical line segments. In terms of redundancy analysis, for digit 5, the impact of Cio and C7 in the first layer is significant (decreased by 6.50% and 5.16%). This may indicate these two communities capture the curve combination features from different perspectives, and there may be redundant feature extraction to improve the network's fault tolerance.
4.1.2 INNERSIGHTNET IN CONVOLUTIONAL LAYER
Overview: During investigating the InnerSightNet within convolutional neural networks LeCun al. (1989), we select the task of cat and dog classification. We design network architecture that consists of three convolutional layers (consisting of 32, 64, and 128 kernels). This is followed by three linear layers binary classification. The AFHQ dataset Choi et al.) (2200) has been chosen. Specifically, the train-set consists of 5,153 images of cats and 4,739 images of dogs, while the test- set includes 500 images from each category. Through the InnerSightNet, we perform community detection the well-trained convolutional layer. Based the Q-value, determine that the most ideal number of communities for the three convolutional layers convolutional layer is 3, 3, and 4. We visualize the sensitivity of each community to input data using certurbation-statisti analysis, C₁ and C2 in the first convolutional layer, с₁ and cs in the second convolutional layer, cp. C22 and cg in the third convolutional layer are defined as key communities for this task, as shown in Fig. 2 (b).
analysis, as shown in Fig. 2 (a). According to the cntturationn-ttaiitiii
Table 1: Acc after closing corresponding community.
26
3
Co
C1
layer
Ace (%)
92
Co
C1
C2
C3
98.5
Co
C1
Cz
98.1
97.9
37
82
6
95.8
layer 2
Acc (%)
Numbers refer number newroms the community.
98.7
94.2
4
C2
98.1
(a) Perturbattion-statittial analysis results
(b) Key communities
69.9
layer 3
Ace (%)
96.9
Figure 2: (a) Perturbation-stttiistcaal analysis results. (b) Key communities. The green communities representing key neurons and gray representing non-key neurons. Numbers refer to the number neurons in the community.
97.5
98.8(+)
97.7
55.5
Community function analysis and finding key communities: To further investigate the role of these key communities, we close each community one by one (i.c., setting the convolutional kernel output of the corresponding index within the community to 0), and record the impact on the accuracy. The results are shown in Table 1. If all the communities are in an open state, the accuracy is 98.7%. In Table 1, the ' indicates that the community closed, and the 'o" indicates that there no C3 community in the first and second convolutional layers.
According table 1, in the first convolutional layer, the co community contains relatively less information, which has little impact on the accuracy. In contrast, the eı and C2 communities contain more information. When both the e₁ and 62 communities are closed simultaneously, the accuracy decreases significantly, indicating that the information in the C₁ and 02 communities has certain degree of complementarity in recognition. The same logic also applies to the 2-nd layer. In the 3-rd layer, the ei community contains less recognition feature information compared to others, while the information the Co, C22 and C3 communities together form a complementary recognition feature.
In addition, observed that when in the third layer is closed, the accuracy is actually improved. Based on the perturbation statistical analysis results, as shown in 2 (a), we can confirm that ç mainly contains features unrelated the recognition task (which define 'noise"). Similarly, from
8

---

Published as a conference paper at ICLR 2025
(n)
(c)
Prediction crrer samples
laver
Inver
layer 3
(b)
layer
Oy ILRER
Og
layer
layer
layer 2
layer 3
6.13
3
cy:81944
Figure 3: (a) The representation distribution in the linear layers. (b) The representation distribution in the convolutional layers. (c) The distribution of community activation levels convolutional layers and the analysis of the error prediction sample.
(a), we can identify that the in the first layer and the co in the second layer are both 'noise'. During the training process the model, the convolutional layers inevitably fit some noisy data. Closing these neurons during the testing phase reduces overfitting and enhances robustness. This processing makes model more accurate in capturing core features to improve the generalization ability.
From Fig. 2 (a), it can be seen that the convolution kernels located within the key community mainly focus on the key feature areas for cat and dog classification, while those in non-key communitics focus on the non recognition feature areas have insufficient attention to the recognition features. We attempt to close the non-key communities the three convolutional layers (Co in the first layer, co in the second layer, and с₁ in the third layer). The result shows that although the accuracy of cat and dog classification decreased to 93.6% (a decrease of 5. 5.1%), we achieve a high recognition accuracy with only 70 neurons. Compared with 224 neurons using the entire convolutional layers, the number of neurons used decreased by 68.75 78,7, providing new perspective for network pruning.
4.2 VISUALIIZING THE REPRESENTATION OF NEURONS WITHIN THE COMMUNITY
To more intuitively reveal the inherent consistency of neuron representations within the same com- munity, we visualize the representations of neurons. We traverse the test-set and calculate the av- erage representation of each neuron benchmark. We fed the samples from test-set one by one and record the output values each neuron. For neurons within a specific community, we use the output value corresponding to their index as their representation. For neurons that do not belong to the community, we use the average as their representation. We collect representation data of specific community neurons and record the community index which these representations belong. We use T-SNE to reduce the dimensionality of these representations, as shown in Fig. 3 (a) and (b).
From Fig. 3(a). neurons within the same community cluster due similar functions and response patterns, reflecting shared feature preferences formed during training. Fig. 3(b) highlights com- plementarity and differences in key communities. In the first convolutional layer, ej and e2 share similarities in feature distribution, explaining their minor individual impact on accuracy but sig- nificant drop when both are closed. Their distinct feature differences, marked in red boxes, justify their assignment to separate communities.
4.3 COMMUNITY ACTIVATION LEVEL AND ANALYSIS OF ERROR PREDICTION SAMPLES
To accurately evaluate the activity level of neurons within the community, adopt the following statistical measurement. We define 'community activation level': refers to the average level of activation values of all neurons within the community. We record the activation outputs of each

---

Published as a conference paper at ICLR 2025
community in each convolutional layer when the imput image is a cat. Similarly, also record the situation when the imput image is dog. We quantitatively describe the distribution of community activation levels using violin plots, as shown in Fig. (c). In cat and dog recognition task, we focus on those error prediction samples and analyze the community activation levels, aiming to explore the reasons behind classification errors. In Fig. (c). we present some prediction errors. Taking the first image as a case, we demonstrate the activation level of each community in convolutional layer when the misclassified sample is fed. The activation values in the first layer suggest that the image tends to activate patterns associated with dogs, which is consistent with observations in the second layer. However, in the third layer, the community activation level of e₁ is closer to the distribution of cat, while in co. c2. and c3 are again biased towards dogs. Importantly, co. C2 and ca in the third layer are the key communities responsible for key identification, while ej the non-key community.
4.4 COMPARE TO OTHER METHODS
Searching for Noise Neuron Communities: In Table 2, we demonstrate that turning off noisy neu- rons in the last layer can improve performance. This is because the noise neuron community focuses more on non-task features, and deleting these neurons makes the networks pay more attention fea- tures. To demonstrate the effectiveness of InnerSightNet in searching for noisy neuron communities, we compare InnerSightNet with Filan et al. (2021), ) Hod et al. (2021). ). and et al. (2023) to search for noisy neurons in the last layer and test the improvement in final accuracy. Filan et al. investigate the concept of 'clusterability'. focusing on dividing neurons into groups with strong internal con- nectivity and weak external connectivity. Hod et al. focus on quantifying the local specialization of neural networks, where clusters of neurons are linked to comprehensible sub-tasks. Liu et al. pro- pose
Modular Training, which enhances network modularity and interpretability by embedding neurons in geometric space, penalizing connection lengths during training. We choose MNIST and AFHQ datasets, and select linear and convolutional layers to be tested, respectively.
Table 2: The results of searching for noise neuron commu- nities.
Table 3:
The results of network
pruning based on key neurons.
MNIST
AFHQ
num
Acc
Filan et al.
98.5%%=0.135%% 92.68% 9..01%%0.011% 9..6%%%0..020% 92.69%+0.008%
Filan el al.
105
94.2%
Hod et al.
98.60%=0.11799
Hod et al.
81
93.5%
Liu et al. InnerSightNet
98.68%=0.075%
Liu et al.
75
93.4%
9..38%=0.033%%
InnerSightNet
70
93.6%
Network Pruning Based on Core Neuron Community: To verify the superiority of InnerSightNet
in locating key neurons, we choose the convolutional layers trained on AFHQ as the network to be tested. Meanwhile, using Filan et al. (202T), [odd et al. (002T) and Liu et al. (023))
baseline methods to search for the key neurons in the neural network. From Table 3, it can be seen that InnerSightNet has significant advantages in the search of key neurons. Although the accuracy of the Filan et al. 2202 method is 0.6% higher than that of InnerSightNet, it uses 15.625% more neurons than Inner' SightNe..... Overall, InnerSightNet performs better in searching for key neurons. This is mainly due to the fact that InnerSightNet considers the connection strength and probability between different neurons and layers, rather than focuses not only on single layer or individual neuron. InnerSightNet is not only suitable for network pruning, but can also be applied to other fields.
5 CONCLUSION AND FUTURE WORK
In this work, we use the inherent characteristics of neurons in Icarnable units to partition neurons into communities. InnerSightNet adaptively searches for the best number of communities and dis- plays the sensitive areas of concern to the community based on roles and functionalities analysis. We analyze the inference process of neural networks from the community perspective, avoiding the limitations of only analyzing single layers or individual neurons. Many future works follow. According to our algorithm, community-basee analysis methods can be potentially applied to the analysis of other tasks, such analyzing the flow of abstract concepts during image generation from generative networks, dynamic problems during network training, etc. Our future work also is based on community analysis improve our understanding in deep neural networks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps, Advances in neural information processing systems, 31, 2018.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 2018.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. arXiv preprint arXiv: 1811.01 157, 2018a.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
David Bau, Jan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works. arXiv preprint 181 2018b.
Punam Bedi and Chhavi Sharma. Community detection in social networks. Wiley interdisciplinary reviews: Data mining and knowledge discovery, 6(3):115-135, 2016.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pallern analysis and machine intelligence, 35(8): 2013.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learming and systems, 2: 2:129-146, 2020.
Sebastian Bordt and Ulrike von Luxburg. From shapley values to generalized additive models and back. In International Conference on Artificial Intelligence and Statistics, pp. 709-745. PMLR, 2023.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188 8197, 2020.
KR1442 Chowdhary and KR Chowdbary. Natural language processing. Fundamentals of artificial intelligence, pp. 603-649 2020.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? in- specting functional modularity through differentiable weight masks. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.
Faisal R Elali and Leena N Rachid. Ai-generated research paper fabrication and plagiarism in the scientific community. Patterns, 4(3), 2023.
Daniel Filan, Stephen Casper, Shlomi Hod, Cody Wild, Andrew Critch, and Stuart Russell. Cluster- ability in neural networks. arXiv preprint arXiv:2103.085 03386, 2021.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8730-8733 2018
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803.03635 2018.

---

Published as a conference paper at ICLR 2025
Patrick Goodley, Haval Balata, Alberto Alonso, Christopher Brockelsby, Matthew Conroy, Nicola Cooper-Moss, Christopher Craig, Matthew Evison, Kath Hewitt, Coral Higgins, et al. Invitation strategies and participation in community-basee lung cancer screening programme located in areas of high socioeconomic deprivation. thorax, 79(1):8-667, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Shlomi Hod, Daniel Filan, Stephen Casper, Andrew Critch, and Stuart Russell. Quantifying local specialization in deep neural networks. arXiv preprint arXiv: 10.08058, 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338-5348. PMLR, 2020.
Alex Krizhevsky, lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Guipeng Lan, Shuai Xiao, Jiachen Yang, and Jiabao Wen. Generative model perception rectification algorithm for trade-off between diversity and quality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13328-13336. 2024.
Richard D Lange, David S Rolnick, and Konrad P Kording. Clustering units in neural networks: upstream vs downstream information. arXiv preprint arXiv:2203. 118/5, 2022.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-555 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 2324, 1998.
Shudong Li, Laiyuan Jiang, Xiaobo Wu, Weihong Han, Dawei Zhao, and Zhen Wang. A weighted network community detection algorithm based on deep learning. Applied Mathematics and Com- putation, 401:126012, 2021.
Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, 26(1) 26(1):41, 2023.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Ggaliwango Marvin, Daudi Jjingo, Joyce Nakbbndba-Naaendee and Md Golam Rabiul Alam. Local interpretable model-agnostic explanations for online maternal healthcare. In 2023 2nd Interna- tional Conference on Smart Technologies and Systems for NexI Generation Computing (ICSTSN), pp. 1-6. IEEE, 2023.
Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based rein- forcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
Mark EJ Newman. Modularity and community structure in networks. Proceedings of the national academy of sciences, 103(23):85778882 2006.
Xinyu Que Fabio Checconi, Fabrizio Petrini, and John A Gunnels. Scalable community detec- tion with the louvain algorithm. In 2015 IEEE international parallel and distributed processing symposium, pp. 28-37. IEEE, 2015.
12

---

Published as a conference paper at ICLR 2025
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
Nicolò Ruggeri, Martina Contisciani, Federico Battiston, and Caterina De Bacco. Community de- tection in large hypergraphs. Science Advances, 9(28) 2023.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Xing Su, Shan Xue, Fanzhen Liu, Jia Wu, Jian Yang, Chuan Zhou, Wenbin Hu, Cecile Paris, Surya Nepal, Di Jin, et al. comprehensive survey on community detection with deep learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Jianyong Sun, Wei Zheng, Qingfu Zhang, and Zongben Xu. Graph neural network encoding for community detection in attribute networks. IEEE Transactions on Cybernetics, 5298)7771,77844 2021.
Vincent A Traag, Ludo Waltman, and Nees Jan Van Eck. From louvain to leiden: guaranteeing well-connected communities. Scientific reports, 911::1-12, 2019.
Hadrien Van Lierde, Tommy WS Chow, and Guanrong Chen. Scalable spectral clustering for over-
lapping community detection in large-scale networks. JEEE Transactions on Knowledge and Data Engineering, 32(4):754-767, 2019.
Chihiro Watanabe, Kaoru Hiramatsu, and Kunio Kashino. Modular representation of layered neural networks. Neural Networks, 97:62-73, 2018.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi. Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184, 2020.
Xingyu Zhao, Wei Huang. Xiaowei Huang, Valentin Robu, and David Flynn. Baylime: Bayesian local interpretable model-agnostic explanations. In Uncertainty in artificial intelligence, pp. 887- 896. PMLR, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv: 1412.6856, 2014.
A APPENDIX
A.1 APPENDIX ABSTRACT
In this work, InnerSightNer can be divided into three primary phases: Initially, it executes process of "neuronization' transforming learnable units into a structured network neurons. Subsequently, these neurons clustered into distinct communities according to representation attributes. The final stage involves the examination of these communities' roles and functionalities to make sure the best community partitioning. In additional, statistical method to unpick the neural tapestry of decision-making. In the appendix, we elaborate on the theoretical background (A.2) and supplement the detailed derivation process of the formulas (A.3) cited in the main text. In addition, we also provide specific details of the algorithm implemen- tation (A.4), methods of roles and functionalities analysis InnerSightNet (A.5), attach extensive experimental results (A.6) and limitations (A.7).
differential output analysis and perturbation-
A.2 THEORETICAL BACKGROUND
In analyzing complex data structures, especially those containing unobserved or implicit variables, probability models demonstrate their powerful ability to effectively reveal the hidden structures behind the data. This type model introduces probability distribution to describe the process of data generation, which not only handle inherent uncertainty properly, but also statistical methods
13

---

Published as a conference paper at ICLR 2025
to accurately estimate model parameters. The advantage of probability models lies in their ability to use parameterization to characterize the interdependence between variables. Especially in classification clustering problems, the Expectation Maximization (EM) algorithm is often used to infer the parameters of these probability models. This algorithm optimizes parameter estimation through iterative process to adapt to the statistical characteristics of observed data.
Theorem: EM parameter estimation for probabilistic models: Given a set of observation data Ad-1,d 7d-1,d Ad,d+1 and pd,d+1 (To facility a cleaner description, we use A, 7, A", and install.),
we consider probability model that takes into account the parameters { Ti, Ti, Tkys Tky) describe the process of result generation. The goal of the model is to maximize the likelihood function of the observed data, which typically involves marginalization of implicit variables. In this context, we can describe the estimation method of model parameters through the following thcorem:
Let represents the prior probability of class (Te TÉ Tk. Th. Hl== represent the condi- tional probability of given class c, respectively. For each observation h and class c, define Os,es where is the posterior probability that observation belongs class c. The parameters A ",j" Teiy (Tes Tikk Tikk
can be iteratively estimated through the following expectation maximiza-
tion steps:
(Step E) Estimate the posterior probability hased on the current parameters:
CR.c - Pk.c/ E. Pk.c:
(14)
where Pk.c is shown Eq.
15 yp is equal
(T Aii (1 1 1 ALa (lipp) Trip)) I--e....
Ph. The
(15)
A T... 1 (Ti Ils 1 (
(Step M) Update the model parameters to maximize the likelihood function of the observed data:
Ee AcKtle ,' Σx 9k,c
Ti Σx 9t.c
Tec
La
(16)
Σx Akgaia
Ex Tn.y9hhe
A
Ex Dr.c
By adopting this theorem, explain how to use the expectation maximization algorithm esti- mate the parameters of probability model under known observation data conditions. This method provides the mathematical foundation for revealing the hidden category structure in the data.
When applying this theorem for parameter estimation, we initially determine the probability of each data belonging different classes through Step E, that is, implementing "soft clustering'. Subsequently, in Step M, we adjust the model parameters to enhance the overall likelihood of these probability distributions. Through repeated iterations, the algorithm will eventually converge and obtain the optimal estimate of parameters, thereby revealing the implicit structure within the data.
A.3 PROOF OF SECTION 2.2
The description Eq. Eq. in main text is the basis of the InnerSightNet: aggregation. Here, we provide the detailed derivation process for Eq. 2 to Eq. 8. According to Bayesian theorem:
Pr(A, A I glz,
( -P(A,,,, 9. T T In (T(). Pr(gIT, A A. (
(17)
14

---

Published as a conference paper at ICLR 2025
Based on the connection matrix and joint probability, can calculate the two factors in Eq. 17 Th
separately. For factor Pr(A, /, A ,9, - 7 , ), where {Ti Ti A "hij is used as
the condition, calculate the probability distribution of Pr(A, I, A. .9.
P, GT ,T TT ;²)
I&
1,90 A (1- - To )
11- 1--Ani - ) (1 Tieee)" )
(18)
( )
Ae (1 T )² - (T ) Ty (
kigs This is used as the condition, calcu- For factor Pr(g P , , where { Tiee Thpyy on o
late the probability distribution of Pr(g)
Pr(gIn, T AA, ,()
I Rg
(19)
Due to the fact that the probability distribution of Pr(A,L, g , is in the form of continuous multiplication, is very friendly for logarithmic calculations. Therefore, the logarithmic likelihood function of the probability distribution is:
In Pr g. , ,+))
la A.P
A ,') Pr(gIT, - A (')))
T
Ta In(Pr(A,L,A,,
,g, _ A i }} + Ta In(Pr(g 7 A T ,7
1 Ta
In IIAIL
Tgg 11- A-A (T 11 (1 - ggg 11-11.13
(20)
) Aes (1 - Ne.t Nia:(To (To1 /
f1.x (1 -/y T Seis } + la
IL ILAII, (LLLI,,TThh)
In oo
1 in ExE, (Ai Tra + +1 A1, In() -iine) + I In Tho + (1 Te,k)
-ta
In(1 =. }} + E, (4 In Tnii + (1 Akr) In(1 Tais)) Lij In Trist
(1- k)) In(1 - + Inf H
where la is the member of the neurons in d-th layer. Since the variable is unknown in Eq. calculate the expected value of the likelihood function on the implicit variable set = {gx}
20
Cg = Pr(gT A, A.I,A
(21)
Substitute Eq.
into
into Eq. 21 we can get,
1
Cg
11 En.lk ne + E, (Ai,k In T.. + (1 (1-Ar,)) An, -p)) + (, In
Ii,k] In(1 - +1,1)) + E, Akk In Tkj +1 A (n.,) In - + E, (Tk. In
+ (1 - ( In(1 - ,((
(22)
Qt.e. represents the probability that the k-th neuron assigned to c-th community. According to Bayesian formula, conclude that ds.c represented Eq. 23
15

---

Published as a conference paper at ICLR 2025
Pr(A, 9h - , ad ,()) (23)
Ok.c
Pr(A, A', ,
For the numerator of gk,os
Pr(A,I, 9k c|T, , ,()
1E .. ΣHa, Prl,,oPr(AA gT A r Td ,t') donate as factor B 92
(24)
i.e, Pr(A A, 1, A 1 9h = c/ T , , ,² = factor B D.=c. x factor gatt
where,
(i) factor =e= -Te
(TA
T - 14 (oo (T(.))
- =(ee) [(e)) hl-
M') - Aes (Tes) Fieo Te.) 1
I1,
Taj
(25)
iii factor B gufe
= I1 Σ
110 Ai. (1 T - (T(.)) 1 (1 =(oo)) 1-h
(T Ans Ans - Toj) 1- (tt - -ttns
For the denominator of ls.er
Pr(A, 1, I, n. - PA T ,gl,, TA A r st()
#1
96
II T. (I Ar. (1 Ts,e) 1 (Te.g) )lio (1 T(e))
(26)
I1 (1 Aos( - Téd) A'ss (To tli -
Therefore, we can get Ok.e as follow,
Pk.c
Dk.c
En Pk.c
(27)
where is shown as Eq. 28 &p is equal to or
Phopp The (T Aus (1 T(p)) 1--- (Teip) (Tep) (1 - Txpp)
1 11
(28)
II (TA aa -a'
(T) ths - 1- [
[p..
Currently, we have the likelihood function C, and the constraint Σ. To 1. For this type of optimization problem with multiple variables and constraints, it can be transformed into a problem with set of equations and can be solved through the Lagrange multiplier method. We define new function å as = mathbf Cg a. Re. where a a constant, and for function h, the best value exists if the following conditions met.
h= 0, h=0
(29)
16

---

Published as a conference paper at ICLR 2025
where ( Tha Ta) Taj, Tagh are the Lagrange multipliers.
For Vnch 0, we can get,
VT (Lgg Ec =0
Va b a
1
va Ek, At.ek Tec + (As, In I TiA + (1- - A1,)) (n(- + E, In In
(1 T(,k) In(1 - x),,)) + E, (Ak. In RR.j
A'
+ (1 - Akk) In(1 - - + E, In Innks (30)
+ (1 - -ka)) In(1 )) = a
1 1
1
1 E Engk,== Eeghi (8.t.,, a= 1)
Exgrie
a Tee l. a
ld Te
la
For T h = 0, we can get,
VrA (C.g a a. To) = 0
Via Lg- =0
1
Ek (ITTe + (Ar, 7 + (1 (n (()- + Ei
+ Vr Ta
(1 11,4 In(1 - ) + E, (Ak. In + (1 - A In (n(- + Cns lurks
(31)
+(1 - (1-1k,)) In(1
[k1 =0
Aie
- Aic ( Aio)= =0
+ Ex
-.r
1 1,c
1,
+ Exh Σ
Ar,c
T/e
= 0 + A
An
A
(1 [,,))
Tie
Ee Ik.e
For V.! 1 1=00 0, we can get,
Vrz. (Lg a. Te) =0
7 Vr Cg =0
+ Vr. Ta
-
1
E
Mk. TI + (A In + (1 AI,x) In(1 - + E, Ln
(1 11,44 In(1
k1 + Σ, (Ak, In In Tkj + (1 - A In + E, (K.. In
(32)
+ +(1 (-hi)))1 In(1 } =
Ihe
- 1,e =0
+) En
T
10
1 x/=
I1, T/_
Ex Tin
E,,,. Σ
= 0 + T/
Tia"" (1
Σx Ik.0
For V v,gch-
h =0= 0, we can get,
17

---

Published as a conference paper at ICLR 2025
V T4' (Cg a. Te) =0
ra' =0
=0
Lko + (A lu + (1- A In(1 T(k)) + Ei(1ish In In
TH' the ld
(1 4., In(1
)) + Σ, (Ak,, In kj + (1 ) In -b5))
+ ΣS (4kss In Fk.j
(33)
(1 (k5))
In
Tk. }}} =0
1
1
A o.
Exgre 2
oj
=0
A
1 1 c/
c
A - TA
Σx Aske
= EEgke E
A'
(1 [
0
Tcj
Σx ts.c
1.0
For V v/y0 h = 0. we can get,
V (Lg 0 3. Te) =0
V o by =0
v ld
Ek (In To + (Ar, i + (1- (1-A,)) 1n (((- + Ei +
k Th.
(1 (1-4,,) In(1 x1,,) + Σ, (Ak, In T + (1 - A In T())) + (ks, In
+11 -1.5) In(1
T/e }}} 0
(34)
A ed
- A
+ E
=0
A
1 1- Tej
[e)
[o. Tej
Σx Jalle
+ >E I Σ
-0 Top
To. (1 71')
Σn Di.0
Therefore, we get Ck.or Tee Tike They Thiy and This as Eq. 35
Pk.c
Σx 9k.e TA
Σx As,
ck.c
Eg Pk,
Tec
EB Th.c
(35)
Σx Tinggr Σx De.e
Σn Angghi
Σx In.
Etghe 9r.c
Σ+ 9k.c
A.4 DETAILS OF EM ALGORITHM IN INNERSIGHTNET
In this study, we propose model based on the EM algorithm aimed discovering potential commu- nity structures in the data. The EM algorithm is an iterative algorithm used for parameter estimation and inference of potential data structures, particularly suitable when the model contains unobserv- able hidden variables, as described in appendix A.2.
Firstly, we defined the function E Step for the expected step (E step). In step E, based on the current model parameter estimation, calculate the expected value of the latent variable Qr.c- The logarithmic probability form used in the calculation to avoid numerical instability when dealing
with extremely small values. Specifically, the model parameters include: {meS 7,, The T +(
18

---

Published as a conference paper at ICLR 2025
Subsequently, we implemented the function M step for maximizing step (M step). In step M. update the model parameters based on the expected values of the latent variables obtained in step E, in order to maximize the logarithmic likelihood function of the observed data.
As is well known, the EM algorithm has instability and is prone to getting stuck in local optima. In the process of executing the EM algorithm, in order to alleviate the above problems, we have adopted three main strategies: random initialization, multiple start strategy, and converting multiplication operations into logarithmic operations. The following will provide detailed description of the implementation methods and their purpose and role of these strategies.
(a) Random initialization: Random initialization refers randomly assigning model parameters before the EM algorithm starts. This is because the EM algorithm, as gradient based optimiza- tion method, relies heavily on the initial values of the parameters find the final solution. If the parameters are initialized properly, the algorithm is more likely to converge to the global optimal solution or a better local optimal solution. On the contrary, improper initialization may lead to slow convergence speed or suboptimal solutions for the algorithm. Through random initialization, we can explore the parameter space from multiple different starting points, increasing the chances of finding better solutions.
(b) Multiple start strategy: Multiple start strategy refers to repeatedly executing the algorithm multiple times, each time using different random initialization parameters. This strategy is based on the assumption that by independently starting optimization from different initial points multiple times, we can select the best local optimal solution from multiple found ones, thereby reducing the risk of the algorithm falling into poor local optimal solutions. In this study, we set the number of multiple starts to 100.
(c) Convert multiplication to logarithmic operation: In step E of the EM algorithm, it is necessary to calculate the product of probabilities, which are often very small. Direct multiplication can lead to numerical underflow. meaning that the computer cannot represent such small values. To avoid this situation, we adopt the method of converting multiplication operations to logarithmic operations. Specifically, by utilizing the properties of logarithmic functions, multiplication can be transformed into addition: taking the logarithm of the probability, adding up. and finally converting it back to the original probability space through exponential transformation. This conversion not only prevents numerical problems, but also improves the numerical stability of the entire calculation process due to the more stable addition operation.
A.5 DETAILS OF ROLES AND FUNCTIONALITIES ANALYSIS INNERSIGHTNET
In the communities of deep neural networks, their roles and functions are more intuitively reflected in input and output. InnerSightNet provides quantitative analyses from the input-output perspective.
Differential outcome analysis: In order to quantitatively analyze the impact of communities on output, we adopt a differential outcome analysis. The differential outcome analysis are statistically analyzed to determine the changes in the output of the neural network between corresponding com- munity is not closed and closed after inputting the same data.
Perturbation-statisiice analysis: order to quantitatively analyze the impact of input data on the community, we adopt a perturbation-statiities analysis. By perturbing the input data and recording the response changes the community, this method allows us to calculate the sensitivity of each community towards changes in input data. We define the sensitivity of the community S. I X²), where N is the number of samples in the test set. f is function that evaluates the difference in feature representation between the original input X; and the perturbed input X"-
When starting perturbation analysis, are not limited the perturbation of independent pixels, but extend it to a 5x5-pixes neighbourhood. This operation takes into account the correlation between adjacent pixels in the image. We define neighborhood perturbation function Per(X;, 2r, 8) that sets the pixels of an image at position (x. y) and its neighborhood to 0, i.e., Per( ( xr, y) = x where X (u, v) = 0 for (u, v) E N(x, y). To measure the impact of perturbations on the neuronal community, we calculated the mean squared error (MSE) of feature representations between the original and perturbed samples. The response of community samples pairs (X,, X²) is:
19

---

Published as a conference paper at ICLR 2025
f(X,, = MSEX, ExT (hc(X.)) - hc((;))
(36)
M
where M is the number of neurons. hx,) and ho(X,) represent the feature representations of X, and X Then we calculate the root of ((X,x;) Perturbation-statiitcaal analysis traverses the input image. Each pixel represents the overall response of the ipput data to community c at point (x, y). We obtain cntturabttiosssttitteete analysis results that are consistent with the size of the input data.
We provide a detailed introduction to cntuuraationsttatstiiiee analysis here. Perturbation-statisse analysis measures which regions of the input data are sensitive to community in deep neural network. The sensitivity of community to imput information is directly related the flow of infor- mation and decision-making processes in deep neural networks. We obtain perturbation-sstaiiiical analysis results that are consistent with the size of the input data.
A.6 MORE DETAILS OF SECTION 4.4
To further demonstrate the significance of InnerSightNet, use the Wilcoxon test to determine the differences between the results of the methods are statistically significant. The Wilcoxon signed- rank test is non-parametric statistical hypothesis test used either to test the location of population based on sample of data, or to compare the locations of two populations using two matched sam- ples, which be applied in statical significance tests. We use MATLAB perform Wilcoxon rank sum test.
Zero hypothesis Ho: Two sets of data come from the same distribution, meaning that there is no significant difference between the two sets of data overall.
Alternative hypothesis H1: The two sets of data come from different distributions, meaning there is a significant difference between the two sets of data.
We list the results of wilcoxon rank sum test between InnerSightNet and the baselines in the follow- ing table.
Table 4: Tthe results of wilcoxon rank sum test between InnerSightNet and the baselines.
baselines
p-value (MNIST)
Statistic (MNIST)
p-value (AFHQ)
Statistic (AFHQ)
Filan el al.
0.000212
-3.704051
0.000381
-3.552866
Hod et al. 0.000157
-3.779644
0.004071
-2.872529
Liu et al.
0.001490
-3.174901
0.001498
-3.1749015
where Statistic represents the magnitude and direction of the difference in rank sum between two samples. The negative statistic indicates that the rank of the first set data is generally lower than that of the second set of data. This means that the values of the first set of data generally smaller than those of the second set. p-value represents the probability of observing extreme or even more extreme results under the assumption that the Hg is true. Usually, when the p-value less than the significance level (such as 0.05 or 0.01), we reject the Ho.
From the above table, it can be seen that the p-values are all less than 0.01. We can reject the Hu and conclude that the two sets of data are statistically significantly different. The Statistic are negative values that further indicates the performances of baselines are generally lower than those of InnerSightNet.
A.7 LIMITATIONS
Although InnerSightNet has demonstrated its unique advantages in partitioning communities based on the input-output representations of neurons, determining the best number of communities, and conducting in-depth analysis of information flow and docision-making processes in deep neural networks, the algorithm still faces two significant limitations.
20

---

Published as a conference paper at ICLR 2025
Firstly, the time consumption of algorithms cannot be ignored. The core of InnerSightNet is based on the EM algorithm, which is an iterative optimization process. Its iterative nature itself means an increase in time cost. In order to avoid the risk of EM algorithm getting stuck in local optima, introduce the multiple start strategy. Although this strategy improves the algorithm's 's global search ability, it further exacerbates the burden of computation time.
The complexity analysis of InnerSightNet reveals the root cause of its time consumption. Inner- SightNet initialization stage involves setting model parameters and initializing the optimal loga- rithmic likelihood value, with a constant level of complexity and minimal impact on the overall performance. However, the main body of the algorithm consists of two layers of loops: the outer loop is responsible for the algorithm restart mechanism, executing R times; The inner loop is re- sponsible for iteratively optimizing the model parameters, with a maximum of T iterations executed per restart. The complexity of these two layers of loops is O R and O(T), respectively. In the inner loop, the algorithm needs to perform probability calculations and parameter updates on each of the K' samples and C clusters, with complexity of O(CC). Due to these operations being executed in each iteration, the overall complexity is proportional to the number of iterations T, i.e. O(TКC). Taking into account the R restarts of the outer loop, the overall complexity of the entire algorithm is (RT
In order to reduce time consumption, adopt multiple strategies. Firstly, migrate the com- putation process to the GPU and use Cupy instead of Numpy to improve computational efficiency. Secondly, we pre calculated the average feature value, average Jacobian matrix, etc., to reduce the evaluation time for each community partition. Although these measures have to some extent alle- viated the time pressure, the computation time of InnerSightNet still relatively long. When the number of single-layer neurons is 128, the computation time for InnerSightNet in processing linear and convolutional neural networks approximately 4 hours and hours, respectively. Therefore, how to further optimize the algorithm to reduce time consumption becomes the focus of our future research.
Secondly, the issue of concentration in community partitioning is also worth paying attention to. When applying InnerSightNet in convolutional neural networks, we find that community partitioning is too centralized, which is in stark contrast to the situation where linear neural network analysis can partition more than 10 communities. This phenomenon raises a question: in common sense, cal and dog images contain more information than handwritten digit, why is there actually less community division? Our explanation is 'taskerelated.. Due to the fact that cat and dog classification is a binary task, the number of effective neurons for binary classification is indeed less than that for ten class tasks. In addition, our community partitioning method is based on classification results, which may lead to bias towards classfication-sseecfee features rather than common features during the parti- tioning process. Therefore, developing evaluation methods suitable for non-classification networks focus community partitioning more on detailed features, such as neurons within a community specifically responsible for recognizing cat eyes, is our future research direction and one of the ways extend InnerSightNet to generative models.
21


Reviews:
[{"review_id": "wVK90RmRKF", "reviewer": "Reviewer_ymgS", "summary": "Review report: Inner Information Analysis Algorithm for Deep Neural Network Based on Community \nThe present manuscript converts learnable units in Deep Neural Networks, such as kernels in CNNs, into networks and then groups the neurons into communities based on their representational attributes. To achieve this clustering, the authors develop the InnerSightNet. By subsequently analyzing the communities, the authors gain insights into the inner working of Deep Neural Networks such as relating certain communities to specific features in the underlying test data and detecting irrelevant neurons.", "questions": "Questions \n-\tIn Eq. (1), could you motivate the distinction in positive and negative value regions more clearly as well as the choice to divide by the normalized cosine similarity, which is essentially either +1 or -1? Could you also specify what exactly data X_i refers to? \n-\tAre the accuracy drops in closing communities in Figure 1a) and Table 1 reported with or without retraining the network? \n-\tIn Table 1, why don’t you report the accuracy after closing the corresponding communities for all possible combinations of the communities in the respective layers? For instance, in the first layer, why are c_0 and c_1 not closed together? Could you also close communities together across layers? \n-\tCould you elaborate on the practical implications of your results, especially regarding parts of the model that fit to noisy data, cf. lines 458 until 464? \n-\tIn lines 1109 to 1112 you write “This phenomenon raises a question: in common sense, cat and dog images contain more information than handwritten digital, why is there actually less community division? Our explanation is ‘task-related’. Due to the fact that cat and dog classification is a binary task, the number of effective neurons for binary classification is indeed less than that for ten class tasks.” Would it be possible to simply restrict MNIST to classifying between 0 and 1 to test for this observation?\n\n\nAdditional comments\n-\tThe axis and legend labels in the Figures are barely readable. \n-\tIn line 53, it should be “one aims” instead of “one aims”. \n-\tIn line 108, it seems inappropriate to cite a review paper to represent community detection in networks. \n-\tIn line 146, the citation “David et al. Bau et al. (2017)” seems to contain a typo. \n-\tIn line 161 and in line 188, the citation “Newman et al. Newman (2006” seems to contain a typo. \n-\tIn Definition 2, it would be helpful to distinguish \n-\tIn line 238, the citation “Lange et al. Lange et al. (2022)” contains a typo.\n-\tIn Eq. (11), it is a bit unfortunate to use delta as a variable again as it also appeared in Eq. (1). \n-\tIn line 264, “sgn is the symbol function” was meant to be sign function? \n-\tIn line 323, the citation “Wanatabe et al. Wanatabe et al. (2018)” contains a typo. \n-\tFrom line 370 onwards, c_10 should be changed to c_{10} in LaTex", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths \n-\tThe main idea of the manuscript, to group learnable units into a network of neurons and then analyze the communities within this network, is very clever and interesting.\n-\tThe general approach to InnerSightNet is described very clearly. In particular, Algorithm 1 is helping in understanding the different steps.  \n-\tIt is intriguing to see that the interdependence of different communities of neurons, such as c_1 and c_2 in the first layer of the CNN as reported in Table 1, is reflected in the 2-dimensional topological representation in Figure 3.", "weaknesses": "Weaknesses\n-\tThe different terms in Eq. (1) could be motivated more clearly, especially given the fact that this step is crucial to transform the kernels of a CNN in a structured network. (See also question below) \n-\tWhile the general structure of InnerSightNet is described very clearly, the more detailed description is cumbersome to follow. Especially the aggregation step, which is heavy in notation but also lacks a definition of some variables such as l_d, \\tau_{g_k, j} and s. \n-\tThe section “Community function analysis and finding key communities” is rather descriptive and only offers a shallow discussion regarding the explainability of community structures for MNIST. The discussion here could be greatly improved by more directly linking the observation from panel (a) and (b) together, as has been done with community c_10 in the first layer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NMrDeZzgpF", "reviewer": "Reviewer_Rnf1", "summary": "The paper proposes an algorithm designed to enhance the interpretability of deep neural networks by analyzing internal neuron communities. The approach transforms neurons into structured communities, aggregates them based on their functional attributes, and evaluates the information flow across these groups. The study applies the method to both linear and convolutional networks, demonstrating improvements in turning off noisy neurons.", "questions": "1. What is the runtime of InnerSightNet for different network sizes? How fast the algorithm converges compared to baseline methods?\n2. How scalable is InnerSightNet when applied to deeper networks like transformers？", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. InnerSightNet introduces a unique perspective by focusing on community-based analysis of neurons, rather than on a single layer or individual neuron.\n2. The three primary phases (neuronization, aggregation, evaluation) adaptively ensure the best number of communities.\n3. InnerSightNet is shown to enhance interpretability and can be applied to network pruning to reduce model size while maintaining competitive performance.", "weaknesses": "1. The paper compares InnerSightNet with methods such as Filan et al. (2021), Hod et al. (2021), and Liu et al. (2023) in the experiments. However, it does not sufficiently explain the methodologies of these baselines. A more detailed introduction to these approaches is needed to help the readers understand how InnerSightNet improves upon or differs from them.\n2. While the method performs well on relatively small networks like those used in the MNIST and AFHQ datasets, it remains unclear how the algorithm scales to deeper networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lzx6UVJ2d6", "reviewer": "Reviewer_7WnU", "summary": "This paper proposes the InnerSightNet algorithm, which clusters neurons in the same layer of MLPs and convolutional kernels in the same layer of CNNs from a community perspective. Through experiments that mask certain neuron clusters and perform perturbation analysis, it demonstrates that some clusters play a key role in task performance, some clusters can be pruned, and some clusters introduce noise.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novelty: The introduction of a community-based perspective for analyzing DNNs is original and provides a fresh viewpoint for understanding the role neurons play.\n2. Comprehensive Framework: The multi-phase algorithm (neuronization, aggregation, evaluation) is well-structured, combining theoretical and practical insights.\n3. Detailed Evaluation: The paper evaluates the communities formed in DNNs through accuracy drop tests and sensitivity analyses, showcasing the impact of each community on network performance.", "weaknesses": "1. Scalability: The paper doesn't address the scalability of InnerSightNet on very large-scale neural networks. It is unclear how well the algorithm would perform or how computationally feasible it is for networks with millions of neurons.\n2. Transferability: The experiments are only conducted on image recognition tasks. Can this method be applied to the interpretability of NLP models? The findings are limited to the functions of different clusters which follows conventional research pattern.\n3. Usability: The mathematical theory of the algorithm is more solid compared to previous works, but the overall improvement in tasks like noise reduction and pruning is very limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)