# Chain Graph Explanation of Neural Network Based on Feature-Level Class Confusion
## 3 Proposed Method
### Extract Key Channels for Class Confusion Based on Unit Channel Suppression
For example, 0% selection on violation graph means that none of the channels in the graph got zero-values, and 90% attack means that we choose 14 \(\times\) 0.9 = 12.6 \(\approx\) 12 channels in the graph to be zeroed out and leave the remaining two channels with their own values. Thus, the larger \(x\) is, the more channels are deleted. We also compare the results with random channel deletion for the same amount of deletion. Figure 3 shows the results of random channel selection on the violation, correction, and total confusion graphs, respectively. The blue line in each figure indicates performance with \(x\%\) selection of each graph, and the orange line is that of random channel deletion with the same number of channels. From each figure it can be observed that even with fewer channels in the path, there is more performance drop than for random channel selection. Further, even if only about 50% of channels are selected in each path, it can cause a greater performance degradation than the random off accuracy in Table 1. This confirms that the channels found by the proposed method operate at the boundaries of the class feature, which is the weak point of the model. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** &
\begin{tabular}{c} **Number of** \\ **Removed Channels** \\ \end{tabular} & **Accuracy (Test Set Acc.)** \\ \hline \multirow{2}{*}{NShap [28]} & 10 & 38\% (74\%) \\ \cline{2-3}  & 20 & 8\% (74\%) \\ \hline Violation Graph & **16** & **0.756\% (76\%)** \\ \hline Correction Graph & **18** & **2.0\% (76\%)** \\ \hline Confusion Graph & **27** & **0.164\% (76\%)** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The number of removed channels for Inception-V3 in [28] and the proposed graphs. Accuracy corresponds to ImageNet test part with the absence of channels and full-featured test set accuracy within brackets. Efficiency not only indicates how each graph corruption affects model accuracy but also includes the time until graph creation. Therefore, we also analyzed the computational details of the proposed algorithm. To measure the importance of each neuron in the model's performance, NShap formulates the problem into Multi-Armed Bandit (MAB) problem and applies early truncation to speed up the single iteration of its algorithm. However, it iterates repeatedly until the algorithm converges. For example, it took 21 h for their algorithm to converge, requiring a total of 3000 iterations to get the Shapley value of Inception-V3 with 100 parallelized computing machines. This indicates that it needs a lot of computing power to converge their algorithm. Meanwhile, the proposed method does not have the convergence problem, so it can be done with a single computing machine with a single iteration. It took 23 h 48 m with a single computing machine to figure out the neurons for our graph with a single computing machine. Although it took about three more hours to obtain the graph than [28], we used only one computing machine, much less than 100, which implies it is better for real-world applications. It took 8 h 51 m to collect the confusion dictionary for VGG16 in a single machine, and it can be further reduced by half if we used multiple machines. Once the confusion dictionary is built, then it only takes 31 s to put it together to form the proposed confusion graph without GPU support. The total number of forward-pass is another option for comparing computation complexity. According to [28], their algorithm can approximate the top 100 import neurons without observing 1500 neurons in one iteration among 17,216 neurons. Thus, they search about 15,000 neurons per batch and repeat 3000 times, and it requires about \(4.5\times 10^{7}\) forward-passes. However, in such a case, the proposed method needs only 17,216 forward-pass for each batch, which is far less than the state-of-the-art method. Figure 3: Model accuracy according to \(x\%\) random selection on each type of graph. (**a**) violation confusion, (**b**) correction confusion and (**c**) total confusion graph.