# Chain Graph Explanation of Neural Network Based on Feature-Level Class Confusion
## 3 Proposed Method
### Extract Key Channels for Class Confusion Based on Unit Channel Suppression
### Observation of Confusion Relations between Two Classes

In addition to the graph, since the proposed method is concentrated on the role of each neuron, the proposed confusion information dictionary can provide clues to understand feature-level class confusions. When building the confusion matrix with image classification results, we cannot consider where each image is at the feature level but only reflect the output for that image. However, based on the confusion information dictionary, which contains what kind of images caused class confusion when modification occurred at the feature level with unit channel suppression, it is possible to distinguish the images that can be changed to another class at the feature level from those that cannot. This can reveal a closer confusion relationship between the two specific classes from confusion relation based on channel deletion. We experimented with ox and cow as an example. Figure 4 shows the ox-cow confusion relationship that can be seen through a simple confusion matrix, and Figure 5 shows the ox-cow confusion relationship drawn through the confusion information dictionary. In both figures, the blue frames represent samples from the cow class, and the red frames indicate samples from the ox class. Model decisions of 'ox' are placed on the left side of the figures, and decisions of 'cow' are placed on the right side of the figures. Then, in Figure 4, we can observe the samples where the label was 'ox', but the model prediction was 'cow' and vice versa. However, in Figure 5, we can observe the classification of the samples in greater detail. In this case, the red arrows indicate cases where the decision changed from cow to ox during unit channel suppression, and the blue arrows indicate the opposite. At this time, cases with dark brown or black objects with cow predictions show red arrows regardless of the presence or absence of horns. Moreover, if a cow is incorrectly predicted as an ox, the decision does not change at the feature level if the color of the object is dark brown or black. Similarly, even in the case of the blue arrow, if the object is a relatively pale color between white and brown, regardless of the presence or absence of horns, confusion occurs at the feature level between ox and cow. Furthermore, in the case of a completely white ox, even if it is incorrectly predicted as cow, the decision does not change at all during the channel suppression. Figure 4: The ox-cow confusion relationship in image-level classification confusion. In addition to explaining the cause of confusion for the two classes in Figure 5, we also checked the concept in ACE. It should be noted here that ACE is mainly used to reveal which concept was used to determine a specific class, but it is not possible to identify the confusion between two specific classes. Therefore, after identifying the confusion relationship with the proposed method, it can be additionally used to find the confused patches for confusion classes. Figure 6 presents concept patches from the ACE algorithm with the purpose of distinguishing between ox and cow in our evaluation set. The collected patches are the subset of concepts that has more than 90% accuracy on cow and ox classification and \(p\)-value \(<\) 0.05. Except for the patches that are related to the background, which has a green or sky-blue color, the color range of ox patches is darker than that of cows. However, the pattern of concept is too noisy, so concepts from ACE are also difficult to translate into simple human language. Thus, we investigate the difference between ox and cow that was observed in the pre-defined predicate provided by AwA2. The color-related parts (e.g., brown, black, white) are marked with the common points of ox and cow in the predicates. Therefore, it becomes a factor that can confuse the two classes. This indicates that class-discriminative characteristics can be inferred by observing the sample distribution, as shown in Figure 6, through the confusion dictionary of the proposed method.