# Chain Graph Explanation of Neural Network Based on Feature-Level Class Confusion
## 3 Proposed Method
### Extract Key Channels for Class Confusion Based on Unit Channel Suppression  
Our assumption for channel suppression is to make the target channel zero-valued. This is because we want to observe the information loss that comes from the absence of the targeted channel. Figure 1 shows how unit channel suppression works on the \(i\)-th layer of a CNN model. To measure the effect of an individual channel on classification confusion, it is necessary to define the types of decision conversion that can be observed by removing channels. There are three cases in which the class changes. 'Violation Confusion' occurs when the model makes correct predictions when the full feature is used but incorrect decisions when the unit channel is deleted. 'Correction Confusion' refers to the opposite case, in which the model makes the wrong prediction when the full feature is used, but the decision becomes correct when the unit channel is deleted. In the final confusion case, the model makes the wrong prediction when the full feature is used and also when channel deletion is performed, but the two predictions are different. Based on these observations, we collect the confusion information for the 'violation' and 'correction' cases to single out the channels that the model was sensitive to in terms of correct and incorrect predictions. Suppose that we have a fixed target model \(f(. )\) and input \(X\), label \(y\). We can select \(k\) layers for which we want to observe the key channel in \(f(. )\), and the set of \(k\) activations are defined as \(A\,:=\,(A_{1},\,\dots,\,A_{k})=(a_{0},\,\dots,\,a_{C-1})\), where \(C\) indicates the total number of channels in \(k\) activations, \(\sum_{k}c_{k}\). For example, if we choose 3 layers with 64, 128, and 256 channels, then the set \(A\) consists of \((a_{0},\,\dots,\,a_{447})\) that \((a_{0},\,\dots,\,a_{63})\) belongs to the layer with 64 channels and the channels \((a_{64},\,\dots,\,a_{191})\) compose the layer with 128 channels and the remainders are for the final layer. The Algorithm 1 shows the pseudo code of confusion graph construction with confusion dictionary collection. First, we collect the violation confusion information from each layer. The confusion information should contain the three items below:  
* Channel information with the layers, which causes confusion. * The class prediction of the model before and after the confusion. * Images that show confusion while the unit channel is suppressed. Figure 1: Unit channel suppression on the i-th layer, c-th channel. Activation map of c-th channel had been turned down to zero and passed to consecutive weights to provide \((i+1)\)-th activation. ``` Input: Fixed model \(f(. )\), Input \(X:=(X_{1},\ \ldots,\ X_{l})\), Label \(y\), Model prediction \(\overline{y}\), The channels of the model \(A\ =\ (a_{0},\ \ldots,\ a_{\text{C}-1})\), Neighbor matrix \(\text{N}:=\ (N_{1},\ \ldots,\ N_{k-1})\); Output: \(Dict_{violation}\); Violation Confusion Information Dictionary \(G_{violation}\); Violation Confusion Graph Step (1) Confusion Relation Collection \(Dict_{violation}\leftarrow\ \{i:\{\ \}\ for\ i\ in\ \text{range}(\text{C})\}\);  For \(X_{l}\) in \(X\):  For (i, a) in enumerate (\(A\)): \(\ \ \ \ \ \ \ \ \ \ \ \



the confusions. The selected channels are key channels since they cause the common confusion for each layer. The Algorithm 2 shows the pseudo code of total confusion graph construction based on violation and correction confusion graph. ### Graph Construction Based on Key Channel Neighboring

From the acquired key channels among \(k\) layers, neighboring is performed by constructing a neighbor matrix between the upper and lower layers to mine channels with the same role. Thus, we need \((k-1)\) neighbor matrixes to make consecutive connections between \(k\) layers. Suppose that the zero-initialized neighbor matrix of \(A_{i}\) with \(m\) channels and \(A_{i+1}\) with \(n\) channels is expressed as \(N\in R^{m\times n}\), which means no relations between the two layers.