# Chain Graph Explanation of Neural Network Based on Feature-Level Class Confusion
## 3 Proposed Method
### Extract Key Channels for Class Confusion Based on Unit Channel Suppression
If \(a\) channels in \(A_{i}\) and \(b\) channels in \(A_{i+1}\) had the same confusion relation, the cartesian product of two lists of channels is performed, and the element corresponding to the cartesian coordinate in the neighbor matrix \(N\) increases in value by 1, indicating that the channels constituting the coordinate play a common role. Thus, if we find \(c\) common confusions, the maximum value that each neighbor matrix element can have should be \(c\). This means that the two channels generate all the common confusion, indicating that they are playing similar roles in class confusion. Similarly, the larger weight on the neighbor matrix indicates that the two connected nodes take similar roles in class confusion. Therefore, when we extract the violation confusion graph, we extract the edges that have the highest weight in the neighbor matrix. In the case of the correction confusion graph construction, all procedures are identical to the steps for generating a violation confusion graph. The only difference is that the confusion information to be collected is no longer a violation but a correction. The obtained violation graph and correction graph each show the connection relationships of the most confusion-related channels in non-overlapped confusion directions. Therefore, by merging these two graphs, we can obtain a chain graph of the network based on feature-level class confusions. Algorithm 1 is the pseudo code of our violation/correction confusion graph construction and Algorithm 2 is the pseudo code for the total confusion graph. ## 4 Experiments

For the following experiments, we use VGG16 [21] initialized with the ImageNet [22] pre-trained model and fine-tuned on the Animals with Attributes 2 (AwA2) dataset [23]. AwA2 has 37,322 images of 50 animal classes with pre-extracted feature representations for each image. To avoid confusion based on imbalanced numbers of classes, we excluded the classes with less than 500 images and constructed the entire dataset by randomly sampling 500 images each for the remaining 25 classes. The training and evaluation data were composed by dividing the entire dataset into an 8:2 ratio, and the graph analysis was performed on the evaluation data through unit channel suppression. The model was optimized with Stochastic Gradient Descent (SGD) with a learning rate of 0.01, which achieves 91.82% accuracy on the validation data. Our experiment was conducted on NVIDIA Titan Xp 12 GB. We investigated the output of each convolution block and classification layers in VGG16. Moreover, we compare our work with [28], which is the state-of-the-art method to approximate the importance of neurons in the model, on ImageNet validation data. For fair comparison, we followed the same settings as [28] by dividing the released ImageNet validation set into half (25,000 images each) and using one part as graph construction and another part as test sets. Since the proposed method does not require hyperparameters except batch size, we set the batch size as 128, which is the same as [28]. ### Confusion Graphs Based on Feature-Level Confusion and How It Affects the Model's Decision

In this section, we describe the effect of our proposed graph on the model's performance. Figure 2 shows the violation and correction confusion graphs of VGG16. As mentioned above, we select the last convolution of each convolution block and classification layers as our target layers. Compared to other studies that are complicated to visualize,our proposed graph selects a relatively small number of channels, less than 5% of the total number of channels, so the key channels can be drawn as a single figure. To observe whether the proposed graph, which collects channels that behaved identically for a common confusion relationship, can be a weak point in the model, we observed the effect of each graph on the model performance. We compared the accuracy of the model after deleting the channels that consist of the confusion graph and the random channel deletion.