A self-explanatory method for the black box problem on discrimination part of CNN
Authors: Jinwei Zhao
Qizhou Wang
Wanli Qiu
Guo Xie
Wei Wang
Xinhong Hei
Deyu Meng
Venue:  (2021)
Keywords: Convolution neural network
Interpretability performance
Markov random field

Abstract: Recently, for finding inherent causality implied in CNN, the black box problem of its discrimination part, which is composed of all fully connected layers of the CNN, has been studied by different scientific communities. Many methods were proposed, which can extract various interpretable models from the optimal discrimination part based on inputs and outputs of the part for finding the inherent causality implied in the part. However, the inherent causality cannot readily be found. We think that the problem could be solved by shrinking an interpretable distance which can evaluate the degree for the discrimination part to be easily explained by an interpretable model. This paper proposes a lightweight interpretable model, Deep Cognitive Learning Model(DCLM). And then, a game method between the DCLM and the discrimination part is implemented for shrinking the interpretation distance. Finally, the proposed self-explanatory method was evaluated by some contrastive experiments with certain baseline methods on some standard image processing benchmarks. These experiments indicate that the proposed method can effectively find the inherent causality implied in the discrimination part of the CNN without largely reducing its generalization performance. Moreover, the generalization performance of the DCLM also can be improved.

Under review as a conference paper at ICLR 2021
A SELF-EXPLANATORY METHOD FOR THE BLACK BOX PROBLEM ON DISCRIMINATION PART OF CNN
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, for finding inherent causality implied in CNN, the black box problem of its discrimination part, which is composed of all fully connected layers of the CNN, has been studied by different scientific communities. Many methods were proposed, which can extract various interpretable models from the optimal dis- crimination part based on inputs and outputs of the part for finding the inherent causality implied in the part. However, the inherent causality cannot readily be found. We think that the problem could be solved by shrinking an interpretable distance which can evaluate the degree for the discrimination part be easily ex- plained by an interpretable model. This paper proposes a lightweight interpretable model, Deep Cognitive Learning MDdLM And then, a game method be- tween the DCLM and the discrimination part is implemented for shrinking the interpretation distance. Finally, the proposed self-explanatory method was eval- uated by some contrastive experiments with certain baseline methods on some standard image processing benchmarks. These experiments indicate that the pro- posed method can effectively find the inherent causality implied in the discrimi- nation part of the CNN without largely reducing its generalization performance. Moreover, the generalization performance of the DCLM also can be improved.
INTRODUCTION
Convolution neural netW)rk(CNN) has surpassed human abilities in some specific tasks such as computer game and computer vision etc. However, they are considered difficult to understand and explain(Brandon, 2017), which leads to many problems in aspects of privacy leaking, reliability and robustness. Explanation technology is of immense help for companies to create safer, more trustable products, and to better manage any possible liability of them (Riccardo et al., 2018). Recently, for finding inherent causality implied in the CNN, the unexplainable problem of CNN, especially con- cerning the discrimination part which is composed of the fully connected layers of the CNN, has been studied by different scientific communities. Many methods were proposed, which extract various interpretable models from the optimal discrimination part based on inputs and outputs of the part for expressing the inherent causality implied in the part. However, because of data bias and noisy data in the training data set, the inherent causality cannot readily be found because the part is difficult be approximated by any interpretable model. We think that the problem could be solved by the following procedure. Firstly, a lightweight interpretable model is designed which can be easily understood by human. And then, the model is initiatively extracted from the discrimina- tion part by solving Maximum SatisfiabiliyyMMAXSSSS the neurons in the first layer and the output layer of the part. An new distance proposed which can evaluate the degree to which the discrimination part is easily explained, namely as interpretabil- ity performance or interpretable distance. For shrinking the interpretable distance, a game process between the interpretable model and the discrimination part is implemented. Finally, the optimal interpretable model can be obtained, which can express inherent causality implied the discrim- ination part. Moreover, based on the procedure, it is also possible to monitor the evolution of the inherent causality implied in the part in the game process.
problem based on the activated states of
Main contributions of this paper can be summarized as follows:
An interpretable model. Deep Cognitive Learning Model(DCLM), is proposed to express the inherent causality implied in the discrimination part, and greedy method is given

---

Under review as a conference paper at ICLR 2021
for initiatively extracting the DCLM from the discrimination part by solving its Maximum Satisfiabiltt(MAA SAT) Problem.
A new game method is proposed improve the interpretability performance of the discrim- ination part without largely reducing its generalization performance by iteratively shrinking the interpretable distance between DCLM and the discrimination part.
A new distance is proposed to evaluate the degree to which the discrimination part is easily explained, namely as interpretability performance or interpretable distance.
RELATED WORK
There are usually two types of methods for the unexplainable problem of the discrimination part, such post-hoc method and ante-hoc method (Holzinger et al., 2019). However, because ante-hoc method is transparent modeling method(Arriet)) et al., 2020), it can not obtain an explanation about the discrimination part. So, the post-hoe method will be reviewed.
Early post-hoe method can obtain global explanations for a neural network by extracting an in- terpretable model. Some references(Craven & Shavlik, 1999; Krishnan et al., 1999; Boz, 2002; Johansson & Niklasson, 2009) proposed few methods that can find decision tree for explaining a neural network by maximizing the gain ratio and an estimation of the current model fidelity. Other references (Craven & Shavlik, 1994; Johansson & Niklasson, 2003; Augasta & Kathirvalavakuaar, 2012; Sebastian et al., 2015; Zilke et al., 2016) proposed rule extraction methods for searching op- timal interpretable rules from neural network.
Recently, some feature relevance methods have become progressively more popular. Montavon et al.oMontvvon et al., 2017) proposed decomposition method from network classification deci- sion into contributions of its input elements based on deep Taylor decomposition. Shrikumar et al. et al., 2016) proposed DeepLIFT which compute importance scores in multi- layer neural network by explaining the difference of the output from some reference output in terms of differences of the inputs from their reference inputs.
Some other works make complex black box model simpler. Che et al. et al., 2017) proposed a simple distillation method called Interpretable Mimic Learning for extracting an interpretable sim- ple model by gradient boosting trees. Thiagarajan et al. (Tihaggaaaaann et al., 2016) build a Treeview representation of the complex model by hierarchical partitioning of the feature space. In addition, some references (Hinton et al., 2015; Bucila et al., 2006; Frosst & Hinton, 2017; Traore et al., 2019) proposed the distillation method of knowledge from an ensemble of models into single model. Wu et aM. Wu, 2018) proposed a tree regularization method via knowledge distillation to represent the output feature space of RNN based on Multilayered perception. However, these methods can only solve the unexplainable problem of trained neural network or trained deep neural networks with explicit input characteristics. Wan et al. (Wan et al., 2020) constructed a decision tree using the last fully connection layer of the discrimination part of CNN based on prior structure.
In the paper, our goal is to find the inherent causality implied in the discrimination part CNN, which is composed of all fully connected layers of the CNN without hurting its generalization per- formance by initiatively extracting its logic relationships with no prior structure and finally obtain its explanation by these logic relationships.
3 DEEP COGNITIVE LEARNING MODEL
For expressing the causal relationship between these neurons in the discrimination part, a new inter- pretable model is designed in the section. As we all known, CNN includes feature extractor and a discrimination part. The feature extractor composes some convolution layers and some pooling layers. The outputs from the feature extractor the inputs of the discrimination part of the CNN, namely feature maps, T1, T2,..., Tk where l is the number of feature maps. All these feature maps form feature set l.
We suppose that the discrimination part should better be explained by the logic relationships of the activated states of the neurons in its first layer and its output layer. This is because the relationships
2

---

Under review as a conference paper at ICLR 2021
the inherent property of the part. In order to express the relationships, deep cognitive learning model (DCLM)is proposed, shown Fig.
Taut S Faad She uad dierr Sae
X003
putt
ay achonetic
Narow Eaper
Narw No tiibe Lipe
Yowwn Dhepue Eugee
Sake Lapal
oilllnteiieation
(biitepp Cegathhr wurnin
Figure 1: Discrimination part and DCLM of CNN(A predication Z; indicates activated state of a neuron 1 of the discrimination part of the CNN and a predication D, indicates an activated state of neuron O₃)
The DCLM consists of three layers:feature predicate layer, disjunction layer, and decision layer. The top layer is feature predicate layer which consists of many nodes. Every node has a predicate (T') that expresses a positive action or negative action of features which the jth neuron the first fully connected layer of the discrimination part captures. The predicate Z, (T) is defined as follows:
1, null, O,
Σ & > -b and T3 r, Σ Ts W = - and T3 E r,
(1)
Z1(1) =
(1')
otherwise.
(1")
where j E 1, N,, N is the number of the input neurons of the first fully connection layer of the discrimination part of the CNN. Wi, a weight vector between the ith feature map and the jth neuron, by is the bias of the jth neuron, and "*" is convolution operation. "1" and "0" denote positively activated state and negatively activated state of the neuron respectively. "null" denotes an inactivated state.
The bottom layer a decision layer which includes nodes used for decision. Every node has predicate which expresses an activated state of an output neuron of the discrimination part. It is
defined as follows:
{ 1,
91 >0
D(y))
0, otherwise.
(2')
where i 1,,,..,,C,C, C,C' is number of the output neurons the CNN, y; is the output value of äth output neuron of the discrimination part. All nodes the feature predicate layer and every node on the decision layer are connected to one or more nodes on the middle layer, namely as disjunction layer, with true or false edges. Every node represents a disjunction relation, which is expressed by disjunctive normal form. is worth mentioning: if node is connected to a node on the disjunction layer by false edge, its predicate follows after non-operator in the disjunetive normal form.
The potential function of a disjunctive normal form can be obtained by using the Lukasiewicz method(Giles, 1975).
(c().) =
(3)
3

---

Under review as a conference paper at ICLR 2021
- (1)-aa) + (ay + 1y)( and N is the number of the nodes the feature predicate layer. If aj = 1, there is a false edge. Otherwise, there is true edge.
where T[I, 34)
The conditional probability distribution that ground DCLM including all disjunctive normal forms true is
p(y, D) = exp( Dis Accbalvi)
(4)
Ei=1 X
where is the number of all ground formulas, = Erer exp( Ee Aidra(le)) is a partition func- Σ tion, y = (y) (3/13/2, "/,,,,,,,................. MU)) yi is an output value of the CNN and X; is weighted value of the ith ground formula.
By maximizing its likelihood function, the optimal o; and X the DCLM can be obtained.
Σ, (. log(=))
C(T) = arg max[log p(y, 1')] arg max
(5)
Σ,>
For extracting a optimal DCLM, a Maximum A Posterior(MAP) algorithm the Maximum Satis- fiability Problem (MAX-SAT) was designed. Using the disjunction normal form with the greatest weighted value in the optimal DCLM, a prediction of input image can be obtained.
EVALUATION OF INTERPRETABILITY PERFORMANCE
We consider that if the discrimination part of CNN has similar shape of function curve with its optimal interpretable model, the former can be easily explained by the latter. Therefore, the inter- pretable performance of the discrimination part be measured by the shape similarity between it and its optimal interpretable model. We posit that given the same input data set, the similarity may be measured by variance of differences between outputs of the both models. It can be named inter- pretation distance. It is easily proved that the smaller the interpretation distance is, the more similar their shapes are, and the better the interpretability performance of the discrimination part would be.
Definition 1 If X is compact metric space and // is a Borel measure in X, such as Lebesgue measure or marginal measures, in (X)), square integrable function space on X, the interpretation distance, d2( P*, f), between discrimination part f(z) and its optimal DCLM P* (x) is
Pa(P" f) = / (f(x) P'(x)) p df
(6)
where
p ( - (f(x) P
GAME BETWEEN A DCLM AND THE DISCRIMINATION PART OF CNN
As discussed above, when the shapes of the discrimination part of a CNN and its optimal interpretable model are enough similar, the discrimination part has well interpretability performance. However, its generalization performance will tend decrease. This is mainly attributed the fact that because of data bias and noisy data in training data set, the sufficient and necessary condition for the consistent 0f is the optimal prediction model), is
convergence of the two performances, Φa(P". {P* f difficult to be guaranteed. Therefore, for the tradeoff problem, in training process, extracting an DCLM from the discrimination part and then iteratively reducing the interpretation distance between the two models may be feasible solution. A detailed discussion about the problem can be found in the appendix.
To avoid reducing the generalization performance, the maximum probability p(w X,y,) should be guaranteed, where X is a training sample, w is the parameter set the CNN and y, is the target vector of X.
p(w] X,y.) p(w X)p(y, w. ply, (X)
& p(w)p(y, w, X)
(8)
4

---

Under review as a conference paper at ICLR 2021
Algorithm Game between DCLM and the discrimination part of a CNN((tss time complexity is
O(N M) where (NN) is a time complexity of training CNN,O(M) is a time complexity of construction of Logic Net.)
Input: Data_set
Output: DCLM
Repeat
CNN = CNN Train Dala_sel, Adam, loss CrossEntropp) for data,label in Data set do
Feature_map C. www.eeeeeueeeeerrrrroooooo
disjunctive normal form
Disjunction( Feature map, Fun, W Rule1
1q Rule2 Eq.2) DCDaaaaDCLMMii (dsssjunnttieeerrrnll... end for
Updata = Eq.5)
for i- to do
for data,label in Data set do
Feature_map = C'N CNNN_feetuuee.czzrrotrrCC ym DCLM (F (Feature map)
CNN DCLM (data, ym, loss Eg.14)
end for end for Until the interpretation distance and accuracy converge
where p(y, w. X) = f(y,, w,,, f(f(f ydeims w, X))(P ww X. Jdy delm of and ydctm is prediction of DCLM.
When the DCLM is known, Vantm is its optimal prediction and p(y W, X) = 1. Then
fOLF Ydelme w, X ())() w. X.dy delm = p(f Ydelm X)
(9)
Similarly, known the input X and w, frn is the optimal solution of the CNN.
ply, w. X) ply. frn w, WyD))p(m Ydalm: X.)
(10)
If and X given and the loss function (y san) =- probability distribution function
- E. Y1 -fnn 12 the conditional
ply, Jan, w, x)= exp(or (Yer Snn))
(11)
E1
Meanwhile,
pmf Ydetm. w, X) exp( - Oa deim fnn))
(12)
E2
where E₁ and Eg are partition functions. Then by maximizing a likelihood function of p(w X, y,)
the optimal w can be obtained. In particular, assuming that follows Gaussian distribution, we get:
Cx,,) = arg max
a
2 +o. (yes fun) - log(E,) @d(ydetm fun) - log(Ez)] (13)
2
where a is meta-parameter determined by the variance of the selected Gaussian distributions. Turn into minimization problem:
Ow (X, y1) arg min| a || - or fnn) + log(En) + dalYantm: Imn) + log (E2) (14)
2
The iterative optimization algorithm is shown as follows:
EXPERIMENTAL VERIFICATION
We designed two experiments to verify the effectiveness the proposed method. The first exper- iment verified whether the self-explanatory method could improve the interpretability performance
5

---

Under review as a conference paper at ICLR 2021
of the CNN without sacrificing its generalization performance. The second experiment verified whether the proposed method can tend towards stability and convergence in the game process.
In the experiments, CNN3(includes 3 convolution layers,3 MaxPooling layers, 3 fully connect lay- crs(FCLs) and 1 output CNNS convolution layers.5 MaxPooling layers, 3 FCLs and 1 output layer), and CNN8(include 8 convolution layers,8 MaxPooling layers, 3 FCLs and output layer) were used. Traditional training methods on the three types CNN were named as CNN3- Trad, CNNS-Trad and CNN8-Tra -rad respectively. By contrast, our proposed methods on these CNNs were named as CNN3-DCLM, CNNS-DCLM and CNN8-DCLM respectively. All experiments used Mnist(Lecun et al., 1998), Fushion 2017), and Cifar- 10(Krizhessky, 2009) bench- mark data sets. All algorithms were implemented in Python using the Pytorch library(Paszke et al., 2019). All experiments ran on server with Intel Xeon 41 41102.11GH Silver Processor, 20GB RAM and Nvidia Telsa T4.
Experiment 1: Performance verification of the proposed method on CNN. We replaced the dis- crimination part of three traditionally trained CNNs with soft decision tree(SDD)(Frosstt & Hinton, 2017) and designated these methods as CNN3-Trad-SDD CNN5-Trad-SDD and CNN8-Trad-SDT respectively. The accuracy of CNN, accuracy of STD or DCLM, and interpretation distance corre- sponding to all methods were shown in Table 1. Some values are " "_", which indicates that these results do not exist.
is observed in Table I that the accuracies of all CNN trained by the proposed method are higher than those of the two interpretable models, such SDT and DCLM, on all benchmark data sets and are around 1.4 percentage points lower than those of all CNN trained by the traditional training method. But it is worth noticing that on the most of data sets the interpretation distances of these CNNs trained by the proposed method are lower than interpretation distances of CNNs obtained by the traditional method. These might prove that that the self-explanatory method can improve the interpretability performance of the discrimination part of CNN without largely reducing its generalization performance. The accuracies of the DCLMs are higher than those of the SDTs except CNN3-DCLM on FashionMnist data set and CNN3-DCLM on Mnist. These results might prove that the proposed method can find more excellent interpretable model than the traditional method.
Experiment 2: Convergence test of the proposed method We designed the experiments to demon- strate convergence of the proposed method. CNN3-Trad, CNN5-Trad and CNN8-Trad were used for comparing with CNN3-DCLM, CNN5-DCLM, and CNN8-DCLM respectively. Every training works out 25 epochs. Experiment results were measured at every epoch and shown in four figures, Fig.2, Fig.3, Fig.5 and Fig.4. Every figure includes nine subplots The three subplots column were shown for the experiment results on Cifar- data set. These subplots on the middle column were for Fashion data set and these subplots on the right column were for Mnist data set.
the left
Poot JALM
OVL JOAN
deL Sne
OS
OM
ON
Talio 240
OML JML
Chi 3M
Seato
OPFICA
1839
Figure 2: Accuracies of DCLMs and CNNs obtained Figure 3: Interpretation distance of CNNs from the by the proposed method
tradition training method and the proposed method re- spectively

---

ANN8-TRAD-SDDY Under review as a conference paper at ICLR 2021
Table Classification accuracies and interpretation distances.
DATA SET: MNIST
METHOD
ACCURACY
ACCURACY OF SDT OR DCLM
INTERPRETATION DISTANCE
CNN3-TRAD
0.97
0.005#0.0039
CNN3-TRAD-SDD
0.970 0.051
CNN3-DCLM
0.993:100006
0.004410.0010
TRAD
0.993=000006
0.005
CNNS-TRAD-SDT
0.961
CNNS-DCLM
0.993+0.007
0.973#0.014
0.003+0.0010
CNN8-TRAD
0.984+0.007
0.003++000099
CNN8-TRAD-SDT
0.978=0.012
CNN8-DCLM
0.992+0.009
0.989=0.018
0.001+0.0010
DATA SET: FASHIONMNIST
METHOD
ACCURACY
ACCURACY SDT DCLM
INTERPRETATION DISTANCE
CNRA-TRAD CNN3-TRAD-SN
0.921#0.021
0.083+0.0355
0.842=0.041 0.82410.032
CNN3-DCLM
0.920.100024
0.01410.0030
CNNS-TRAD
0.923#0.024
0.064+0.011
TRAD-SDT
0.792=0.041
CNNS-DCLM
0.921 0.0022
0.87310.036
0.009:10.0020
0.931
0.1224:.0339
0.785 0.905=0.032
CNN8-DCLMM
0.914+0.020
0.005+000000
DATA SET: CIFAR-10
METHOD
ACCURACY
ACCURACY OF SDT OR DCLM
INTERPRETATION DISTANCE
CNN3-TRAD
0.68110.040
0.0444100000
CNN3-TRAD-SDI
0.538
CNN3-DCLM
0.643:10.038
0.022+0.0030
CNNS-TRAD
0.744+00004
0.036+
CNNS-TRAD-SDTT
0.609
CNNS-DCLMM
0.729+0.037
0.684+0.039
0.027+000000
CNN8-TRAD
0.754+0.003
0.040+1000000
0.647=0.035
CNN8-DCLM
0.682+0.038
0.661=0.036
0.01440.0030
In 2, the accuracies of the DCLMs and the CNNs of every epoch in the game process were shown. From these figures, is obvious that accuracies of the DCLMS and these CNNs steadily increase in the early stage. In the next stage, these accuracies tend to be stable. This reflects that the game method do not affect the improvement the generalization performances of these DCLMs and these CNNs. We also find that the accuracy gap obtain by CNN3-DCLM for the FashionMnist data set is much more than other two data sets. Meanwhile, it can be found that the DCLM convergence to stable state. The main reason is that for CNN3, in the later stage of the game process, inherent causality implied in the part can not readily be found by the FashionMnist data set. Even so, the proposed method also improves the accuracies of the DCLMS and these CNN-DCLMS steadily. We also find that the gaps between these accuracies of the DCLMs and the CNNs obtained by the CNN8 become very small the later epochs. This reflects that CNN8 can extract more effective features by which the DCLMs can find more accurate causality implied in the discrimination part of the CNNs.
Fig.3 shows the interpretation distances of the CNNs trained by the traditional method and the CNNs trained by the proposed method, As seen from these subplots, the interpretation distances of CNNs trained by the traditional method are greater than those the other CNNs at the most the epochs, especially by the end of the game. The results indicate that the game method can effectively improve the interpretability performance CNNs. From these subplots in Fig. 2and Fig.3, we also
7

---

Under review as a conference paper at ICLR 2021
030, 241
Spc
090
Figure 4: Information Entropy of DCLM
Figure 5: Accuracies of CNNs obtained by the tradition training method and the proposed method
can see that after the 15th epoch, interpretation distances of CNNs from the proposed method tend to converge. The phenomenon indicates that the discrimination part of the CNNs can be explained by its DCLMs at every epoch after the fifteenth epoch.
From Fig.4, is evident that all DCLMs from CNN3-DCLMS, CNN5-DCLMs, and CNN8-DCLMs have been found to have stable information entropies at the end of the which calculate the diversity of disjunction normal forms in the DCLMs obtained by the game method. On Mnist data set. the entropies finally converge 135, 13d 51.3 respectively. On FashionMnist data set, the entropies finally converge 144,132,and 65. On Cifar-10 data set, the entropies finally converge 65,73,and 57. More complicated the extract part of CNN, more small the information entropies of DCLM obtained by the proposed method. The results indicate that the game algorithm can ensure that the diversity of disjunction normal forms of the DCLMs converges to stable state. The game with the CNN with complex structure can obtain the more robust DCLMs than with the CNN with simply sturcture. The main reason is that the features captured by the CNN with the complex structure is so sparse and robust that the disjunction normal forms of the DCLMs is sparse and robust.
In 5,h accuracies of the CNNs trained by the traditional method and the CNNs trained by the proposed method at every epoch were shown. From these subplots, can be seen that its accuracies steadily increase in the early stage. But in the following stage, their accuracies tend to be stable and consistent. The main reason is that in the early stage, a tradeoff problem between the generalization performance and interpretability performance of the discrimination part of CNN inevitably reduces its generalization performance in order to increase its interpretability performance. Though the proposed game method effectively reduce the gap between two performances, it do not increase the gap between the accuracies of the CNNs trained by the traditional method and the CNNs trained by the proposed method. This reflects that the proposed method effective for the tradeoff problem.
CONCLUSION
The performance of the proposed method was demonstrated by experiments benchmark data sets. The proposed method showed prominent advantages over traditional learning algorithm on CNN for improving the generalization performance and the interpretability performance of the discrimination part of the CNN.
In practical engineering, the proposed method may provide new learning paradigm. The method can not only predict an relatively accurate result for new input data but also provide a reasonable causal interpretation for the prediction of the discrimination part of a CNN. We suppose that it can solve the black box problem in the discrimination part. We believe that the proposed method provide way to understand the discrimination part.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
A.B. Arrietaa, N. Diaz-Rodriguezh, J.D. Sera, A. Bennetoth, S. Tabikg, A. Barbadoh, S. Garciag, S. Gil-Lopeza, D. Molinag, R. Benjaminsh, Chatilaf, and F. Herrerag- Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion, 58:82-115, 2020.
M. Gethsiyal Augasta and T. Kathirvalavakumar. Reverse engineering the neural networks for rule extraction classification problems. Neural Process. Lett, 35(2):131-150, 2012.
O. Boz. Extracting decision trees from trained neural networks. pp. 456C461, the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002.
J. Brandon, An ai god will emerge by 2042 and write its own bible. will you worship it?, 2017. https: / /twptteeettteetttteeee/eeeaeaa.e com.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceed- ings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, 2006.
Z.P. Che. S. Purushotham, R. Khemani, and Y. Liu. Interpretable deep models for icu outcome prediction. Amia Annu Symp Proc, 2016:371-380, 2017.
Mark W. Craven and Jude W. Shavlik. Using sampling and queries extract rules from trained neural networks. Machine Learning Proceedings, pp. 37-45, 1994.
Mark W. Craven and Jude W. Shavlik. Extracting tree-structured representations of trained networks. Advances in Neural Information Processing Systems, 8:24-30, 1999.
N. Frosst and G. Hinton. Distilling neural network into soft decision tree, 2017. arXiv preprint arXiv: 171.0977 (2017).
R. Giles. Lukasiewiez logic and fuzzy set theory. International Journal of Man-Machine Studies, 8 (3):313-327, 1975.
G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. Computer Science, 2015.
A. Holzinger, G. Langs, H. Denk, K. Zatloukal, and H. Mller. Causability and explainability of arti- ficial intelligence medicine. Causability and explainability of artificial intelligence in medicine, 944) 2019.
U. Johansson and L.. Niklasson. Rule extraction from trained neural networks using genetic pro- gramming. Int.conf.neraal Inform.processingg 2003.
U. Johansson and L. Niklasson. Evolving decision trees using oracle guides. Computational Intelligence and Data Mining, 2009. CIDM '09. IEEE Symposium on, 2009.
R. Krishnan, G. Sivakumar, and P. Bhattacharya. Extracting decision trees from trained neural networks. Pattern Recognition, 92(1)):199920099 1999.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 36(11)2228-2224, 1998.
S. Parbhoo M. Zazzi V. Roth andF. Delei-Velez M. Wu, M. C. Hughes. Beyond sparsity: Tree regularization of deep models for interpretability. AAAI, 2018.
Grgoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus Robert Mller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211-222, 2017.

---

Under review as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035.
Curran Associates, Inc., 2019.
G. Riccardo, M. Anna, R. Salvatore, T. Franco, G. Fosca, and P. Dino. A survey of methods for explaining black box models. ACM Computing Surveys, 51(5):1-42, 2018.
B. Sebastian, Alexander, M. Grgoire, K. Frederick, M. Klaus-Robert, S. Wojciech, and S.O. Deniz. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. Plos One, 10(7):e0130140; 2015.
A. Shrikumar, Greenside, A. Shcherbina, and A. Kundaje. Not just a black box: Learning impor- tant features through propagating activation differences, 2016. arXiv preprint arXiv:1605.011 (2016).
J.J. Thiagarajan, B. Kailkhura, P. Sattigeri, and K. N. Ramamurthy. Treeview: Peeking into deep neural networks via feature-space partitioning, 2016. arXiv preprint arXIv:1611.0742292016
R. Traore, H. Caselles-Dupre, T. Lesort, T. Sun, G. Cai, N. D. Rodriguez, and D. Fil- liat. Discorl:coninnul reinforcement learning via policy distillation, 2019. arXiv preprint arXiv:1907.000 www.1990755885(011)
A. Wan, I.. Dunlap, D. Ho, J. Yin, S. Lee, Jin, Petryk, S. Bargal, and E. Gonzalez. Nbdt: Neural-backed decision trees, 2020.
S.E. Zalando. Fashion mnist data set, 2017. http::/ww..tt..//////a///oo//aaa httt:p//hhhaa...mmmaaam..eeeeeet
J.R. Zilke, E.L. Mencia, and F. Janssen. Deepred-rule extraction from deep neural networks. In International Conference on Discovery Science, Springer, pp. 457-473, 2016.
A APPENDIX: SUFFICIENT AND NECESSARY CONDITION FOR CONSISTENT CONVERGENCE OF THE GENERALIZATION PERFORMANCE AND THE INTERPRETABILITY PERFORMANCE
If the minimization of loss function of CNN can guarantee the minimum of its interpretability per- formance, learning algorithm of the CNN can improve its interpretability performance. If not, the tradcoff problem between the generalization performance and the interpretability performance will exit. For proving the existence of the problem, we focus on neuron of CNN. From the foot we may judge of Hercules. If an input channel fx) of the neuron is seen as kernel function K ((x, ww is weight vector including a bias of the neuron), it will span kernel Hilbert space HK = (f()) E (X) f(x) = K ((x,, w) = Ex= (x))(<)) for the neuron. Hk can be regarded as linear function set on (3
a solution space of the neuron. The necessary and sufficient conditions for consistent convergence between the generalization performance and the interpretabil-
ity performance are discussed below in 23 based on the following lemmas.
Lemma 1.
Continuous linear functional set on separable Hilbert space X is nowhere dense in square integrable function space (X).
Lemma 2. Continuous nonlinear functional set of the separable Hilbert space X is everywhere dense in (X).
When the optimal input channel (xx approximate linear functional in Hx while the optimal in- terpretable model p* (ax) don't approximate any linear functional or do not exist in Hk. traditional training process cannot guarantee í' (x:) approximates p*(x) according Lemma 1. From Lemma 2, the approximate will cannot converge until Pa (æ) approximate f (xx). Here, we define the approx- imation the similarity between the shapes of function curve of f*:)) and P" the sufficient and necessary condition for the consistent convergence of the two performances is P For the discrimination part of the CNN, the sufficient and necessary condition is still true. However
O.
10

---

Under review as a conference paper at ICLR 2021
because of data bias and noisy data in training data the condition is difficult to be ensured in the majority engineering applications. The tradeoff problem always exists between the two performances of the discrimination part.
According to the above conclusion, in order to completely solve the tradcoff problem, the da(P", f (Here (x) is the optimal discrimination part) should be reduced. However p and (x) are unknown. Therefore, in training process, extracting a interpretable model P(œ) from discrimination part (x) and then iteratively reducing @a(P, f) may be convenient method.


Reviews:
[{"review_id": "uLSEy5LJyT5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper Summary\nThe authors proposed Deep Cognitive Learning Model (DCLM) as a surrogate of \"discrimination part of CNN\" which is the fully connected layers after the convolution layers. The authors claimed that DCLM can explain the causal relationship between the features and the output result of the discrimination part of CNN.\n\n### Quality & Clarity\nI found the paper very hard to read. I tried my best to understand the paper. The major contributions of this paper are as follows.\n1. The authors proposed to use DCLM to approximates the fully connected layers (FC), i.e. $\\mathrm{FC}(x) \\approx \\mathrm{DCLM}(x)$ for any $x$. Because DCLM is in a logical formula, we can interpret the \"discrimination part of CNN\" approximately by interpreting DCLM.\n2. The authors further proposed to use DCML to \"regularize\" FC. That is, train CNN so that its FC to be close to DCLM.\nThe first contribution is described in Sec3, while the second contribution is described in Sec4 and Sec5.\n\nBelow, I list the points that makes the paper hard to read.\n* The definition of the term \"discrimination part of CNN\" is described at the beginning of Sec3, although the term is frequently used in Sec1. I could not understand what the \"discrimination part of CNN\" means when I first read the paper. This problem is fatal because the readers cannot understand the problem the authors want to solve in this study.\n* The paper does not describe why interpreting \"discrimination part of CNN\" is important. Thus, the paper failed to motivate the problem. Although the authors cited some related studies, such as [Wan+20], the importance of the problem need to be described in the paper. This problem is fatal because the readers cannot understand why the problem is important.\n* Algorithm1 contains undefined notations. For example, the inputs and the outputs of $\\mathrm{CN}$ and $LN$ are not defined. There are no descriptions what $FMs$ and $f_{nn}$ are. Moreover, the second $\\mathrm{CN}$ in Algorithm1 outputs $FCMs$ in addition to $FMs$ and $f_{nn}$. This problem is fatal because the readers cannot understand how the proposed method operates.\n* The authors merely compared the accuracies of DCLM and its approximation performance. There is no demonstration nor discussion that DCLM can resolve the problem of interpretability of the \"discrimination part of CNN\", which should be the primal goal of this study. This problem is fatal because the readers cannot understand whether the problem considered in this study is solved.\n\nOverall, I think the paper needs major rewriting so that the main message of the paper to be clear: what the problem is; why it is important; how the authors solved the problem; and how the authors confirmed the problem is solved.\n\nIn addition to the readability, I also found some technical errors.\n* In Eq.(5), the partition function $\\log \\Xi$ should appear because it is a function of $y_{dclm}$ and $a$. The authors somehow ignored it.\n* In Eq.(13) and Eq.(14), the partition functions $\\log \\Xi_1$ and $\\log \\Xi_2$ should appear because they are functions of $w$. The authors somehow ignored them.\nThere should be some justifications why one can ignore the partition functions. Or, the experimental results need to be updated based on the objective functions with the partition functions.\n\n### Originality & Significance\nThe use of decision tree for approximating the \"discrimination part of CNN\" is considered by [Wan+20]. This paper proposes using a logical formula instead of the decision tree. The idea seems to be straightforward, and the innovation made in this paper is marginal.\n\n### Pros & Cons\n[Pros]\n* I could not find anything positive about this paper.\n\n[Cons]\n* The paper is very hard to read. The major rewriting is needed.\n* There are some technical errors.\n* The improvement over [Wan+20] seems to be marginal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qZb6h-deJy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting framework for training interpretable CNNs, similar to distillation methods. The authors propose a probabilistic model to approximate CNN predictions (specifically the discriminatory part i.e. fully connected network, and a procedure for training CNN+ DCLM as a game. Results show interesting performance over benchmark datasets in comparison to existing distillation baselines. \n\nMajor concerns:\n1. While the motivation is easy to understand, notation is continuously introduced in the paper without clear description and definitions. This makes it very challenging to parse the technical correctness of the paper. \n\n2. What is the iterative optimization procedure described in Page 5? Highly unclear how the CNN model is actually updated in Algorithm 1 to do the minimization of Eq (14)\n\n3. Can the authors elaborate why the accuracy gap for more challenging datasets like FashionMNIST is much more than simpler datasets for the DCLM model? I also suggest authors to include other datasets instead of focusing on MNIST only.\n\n4. The entropy analysis is interesting but I would've liked to see more insight into why entropy behavior across different datasets is so variable.\n\n5. Way too many typos in the paper. Please proof-read and correct, I only highlighted a few:\n 1. Update_CNN_Gradient in Algorithm 1\n 2. Information Entropy in Figure 5 caption\n 3. Clarify what \"discrimination part\" of CNN means, very early on in the introduction and abstract.\n\n6. Did the authors do a qualitative analysis of where exactly the approximate DCLM has lower accuracy compared to the CNN model and why?", "extra_scores": {}, "extra_text": {}}, {"review_id": "BAVYJ09zazC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I found this paper very difficult to follow as it has many grammatical and syntactic errors. I believe it needs a significant amount of editing in order for the paper to be published in english. In particular careful attention should be made to omitted particles and pluralization.  This alone is a barrier to publication. \n\nSetting aside the grammatical errors, I believe there are some issues in the content of the paper that would need to be addressed as well in order to make this paper ready for publication. \n\nIf I understand the paper correctly, which I am not sure I do, I believe the authors propose a method for extracting an interpretable model which replaces the fully connected layers of a Convolutional Image Network. They present results on toy datasets, MNIST, Emnist, and FashionMNIST.   \n\nMy primary issue with the paper is that it attempts to provide an 'explainable alternative' to a CNN but this explainable model still relies on the features extracted from the convolutional section of a CNN. The paper does not put forward a convincing argument to justify the focus on the fully connected layers. It is interesting to extract an interpretable model from a fully connected network, but if this is the goal of the paper, then the authors should focus on datasets in which a fully connected network outperforms standard explainable models such as logistic regression but interpretability would still be desirable, such as the MIMIC medical dataset. \n\nThe paper would also be significantly improved if more realistic datasets would be explored. The only datasets used are variants of MNIST, in which good performance can be achieved with traditional explainable models.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors work on improving the interpretability of CNNs following distillation methods. The paper is written in such a convoluted way and with many changes in the notation that makes it hard to understand what they are proposing and what for. This is a major impediment for the paper going forward. But also, the reviewers point out some clear aspects of the paper and the interpretability that it provides for CNNs, for example, the entropy analysis and its variability or referring to the discrimination part of the CNN or mentioning an explainable alternative to CNNs, while still relying on the convolutional filters. The authors need to dedicate more time to explain this paper carefully before it can be properly reviewed and published at any major conference.
Decision: Reject