# Chain Graph Explanation of Neural Network Based on Feature-Level Class Confusion
## 3 Proposed Method
### Extract Key Channels for Class Confusion Based on Unit Channel Suppression
To maintain fairness between the random and proposed graph, the number of channels turned off for each layer in random channel deletion is the same as the number of channels turned off in the same layer in the proposed confusion graph. We repeat this procedure five times to maintain the randomness of the experiment and use the average value. Table 1 shows the performance drop of the model with each type of graph and the full-featured model. The total number of channels in our selected layer was 9152, and the classification accuracy was 91.82% on the evaluation data. The first column shows the accuracy of the model according to each state. The full-featured state indicates a state in which no zero-out manipulation is applied to the model, and violation, correction, and total confusion are the results of measuring the accuracy of the model after deleting each type of graph from the full-featured state. The second column shows the model accuracy after zero-out by randomly selecting the same number of channels as the proposed graph for each layer. The last column indicates the total number of channels dropped by each graph state. The result shows that when the total number of channels in confusion graphs, 54, is less than 1% of the total number of channels, the performance decreased by 58.7%, dropping to less than one-third of the original accuracy. Furthermore, compared to the random channel deletion, the selected channels play a similar role in that they show 47.11% \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Accuracy** & **Random-Off** & **Number of Removed Channels** \\ \hline Full Feature & 91.82 & - & 0 \\ \hline Violation & **78.03** & 89.67 & 14 \\ \hline Correction & **51.04** & 80.36 & 44 \\ \hline Total Confusion & **33.12** & 80.23 & 54 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Accuracy (%) of VGG16 without any information loss (full feature) and with information loss after deleting each type of graph in the AwA2 dataset and the number of removed channels in each case. Random-off is the accuracy when the same number of channels is randomly turned off. Figure 2: (**a**) Violation confusion graph of VGG16 and (**b**) correction confusion graph of VGG16 that are built from the proposed algorithm. The final confusion graph would be the union of the two graphs. lower performance compared to the random deletion when 54 channels are deleted. This suggests that the proposed graph points to the weak channels of the model, which should be modified to improve the robustness of the model in the future. Table 2 shows the performance drop of the proposed graph and the Neuron Shapley (NShap) [28]. According to [28], 25,000 ImageNet test sets they used showed 74% accuracy for the pretrained Inception-V3 and dropping 10 filters lowered the model's performance by 38% and dropping 20 could lower it by as much as 8%. Since we also separate the ImageNet validation set into two parts, one for graph construction and another for test, we should report the performance of pretrained Inception-V3 on our test part. We get 76% accuracy, and it is indicated in parentheses of the last column of Table 2. The observed model performance by deleting the proposed graphs is shown in the 3 rd to 5 th rows of Table 2. Both the violation graph and the correction graph delete less than 20 neurons, and the violation lowers the accuracy by 0.75% and the correction by 2%. It infers that the proposed method can find more efficient neurons than [28]. Moreover, even though the confusion graph uses a larger number than [28], it can completely destroy the model in terms of reducing the accuracy by 0.1%. The time taken to obtain the graph will be covered in Section 4.2. ### Efficiency of Proposed Confusion Graph

Additionally, we checked the effect of each graph by performing a random selection on each type of confusion graph. We randomly turn off as many as \(x\%\) of channels in each graph and leave (\(100-x\))% of channels in each graph alone.