A self-explanatory method for the black box problem on discrimination part of CNN

1 INTRODUCTION . Convolution neural network ( CNN ) has surpassed human abilities in some specific tasks such as computer game and computer vision etc . However , they are considered difficult to understand and explain ( Brandon , 2017 ) , which leads to many problems in aspects of privacy leaking , reliability and robustness . Explanation technology is of immense help for companies to create safer , more trustable products , and to better manage any possible liability of them ( Riccardo et al. , 2018 ) . Recently , for finding inherent causality implied in the CNN , the unexplainable problem of CNN , especially concerning the discrimination part which is composed of the fully connected layers of the CNN , has been studied by different scientific communities . Many methods were proposed , which can extract various interpretable models from the optimal discrimination part based on inputs and outputs of the part for expressing the inherent causality implied in the part . However , because of data bias and noisy data in the training data set , the inherent causality can not readily be found because the part is difficult to be approximated by any interpretable model . We think that the problem could be solved by the following procedure . Firstly , a lightweight interpretable model is designed which can be easily understood by human . And then , the model is initiatively extracted from the discrimination part by solving a Maximum Satisfiability ( MAX-SAT ) problem based on the activated states of the neurons in the first layer and the output layer of the part . An new distance is proposed which can evaluate the degree to which the discrimination part is easily explained , namely as interpretability performance or interpretable distance . For shrinking the interpretable distance , a game process between the interpretable model and the discrimination part is implemented . Finally , the optimal interpretable model can be obtained , which can express inherent causality implied in the discrimination part . Moreover , based on the procedure , it is also possible to monitor the evolution of the inherent causality implied in the part in the game process . Main contributions of this paper can be summarized as follows : • An interpretable model , Deep Cognitive Learning Model ( DCLM ) , is proposed to express the inherent causality implied in the discrimination part , and a greedy method is given for initiatively extracting the DCLM from the discrimination part by solving its Maximum Satisfiability ( MAX-SAT ) Problem . • A new game method is proposed to improve the interpretability performance of the discrimination part without largely reducing its generalization performance by iteratively shrinking the interpretable distance between DCLM and the discrimination part . • A new distance is proposed to evaluate the degree to which the discrimination part is easily explained , namely as interpretability performance or interpretable distance . 2 RELATED WORK . There are usually two types of methods for the unexplainable problem of the discrimination part , such as post-hoc method and ante-hoc method ( Holzinger et al. , 2019 ) . However , because ante-hoc method is a transparent modeling method ( Arrietaa et al. , 2020 ) , it can not obtain an explanation about the discrimination part . So , the post-hoc method will be reviewed . Early post-hoc method can obtain global explanations for a neural network by extracting an interpretable model . Some references ( Craven & Shavlik , 1999 ; Krishnan et al. , 1999 ; Boz , 2002 ; Johansson & Niklasson , 2009 ) proposed a few methods that can find a decision tree for explaining a neural network by maximizing the gain ratio and an estimation of the current model fidelity . Other references ( Craven & Shavlik , 1994 ; Johansson & Niklasson , 2003 ; Augasta & Kathirvalavakumar , 2012 ; Sebastian et al. , 2015 ; Zilke et al. , 2016 ) proposed rule extraction methods for searching optimal interpretable rules from a neural network . Recently , some feature relevance methods have become progressively more popular . Montavon et al . ( Montavon et al. , 2017 ) proposed a decomposition method from a network classification decision into contributions of its input elements based on deep Taylor decomposition . Shrikumar et al . ( Shrikumar et al. , 2016 ) proposed DeepLIFT which can compute importance scores in a multilayer neural network by explaining the difference of the output from some reference output in terms of differences of the inputs from their reference inputs . Some other works make complex black box model simpler . Che et al . ( Che et al. , 2017 ) proposed a simple distillation method called Interpretable Mimic Learning for extracting an interpretable simple model by gradient boosting trees . Thiagarajan et al . ( Thiagarajan et al. , 2016 ) build a Treeview representation of the complex model by hierarchical partitioning of the feature space . In addition , some references ( Hinton et al. , 2015 ; Bucila et al. , 2006 ; Frosst & Hinton , 2017 ; Traore et al. , 2019 ) proposed the distillation method of knowledge from an ensemble of models into a single model . Wu et al. ( M . Wu , 2018 ) proposed a tree regularization method via knowledge distillation to represent the output feature space of a RNN based on a Multilayered perception . However , these methods can only solve the unexplainable problem of trained neural network or trained deep neural networks with explicit input characteristics . Wan et al . ( Wan et al. , 2020 ) constructed a decision tree using the last fully connection layer of the discrimination part of a CNN based on a prior structure . In the paper , our goal is to find the inherent causality implied in the discrimination part of CNN , which is composed of all fully connected layers of the CNN without hurting its generalization performance by initiatively extracting its logic relationships with no prior structure and finally obtain its explanation by these logic relationships . 3 DEEP COGNITIVE LEARNING MODEL . For expressing the causal relationship between these neurons in the discrimination part , a new interpretable model is designed in the section . As we all known , a CNN includes a feature extractor and a discrimination part . The feature extractor composes of some convolution layers and some pooling layers . The outputs from the feature extractor are the inputs of the discrimination part of the CNN , namely feature maps , τ1 , τ2 , ... , τk where k is the number of feature maps . All these feature maps form a feature set Γ . We suppose that the discrimination part should better be explained by the logic relationships of the activated states of the neurons in its first layer and its output layer . This is because the relationships are the inherent property of the part . In order to express the relationships , a deep cognitive learning model ( DCLM ) is proposed , shown in Fig.1 ( b ) . The DCLM consists of three layers : feature predicate layer , disjunction layer , and decision layer . The top layer is feature predicate layer which consists of many nodes . Every node has a predicate Zj ( Γ ) that expresses a positive action or negative action of features which the jth neuron in the first fully connected layer of the discrimination part captures . The predicate Zj ( Γ ) is defined as follows : Z1 ( Γ ) = 1 , ∑k i=1 τi ∗ wi,1 > −b and τi ∈ Γ , ( 1 ) null , ∑k i=1 τi ∗ wi,1 = −b and τi ∈ Γ , ( 1′ ) 0 , otherwise . ( 1′′ ) where j ∈ 1 , 2 , ... , N , N is the number of the input neurons of the first fully connection layer of the discrimination part of the CNN . wi , j is a weight vector between the ith feature map and the jth neuron , bj is the bias of the jth neuron , and ” ∗ ” is a convolution operation . ” 1 ” and ” 0 ” denote a positively activated state and a negatively activated state of the neuron respectively . ” null ” denotes an inactivated state . The bottom layer is a decision layer which includes all nodes used for decision . Every node has a predicate which expresses an activated state of an output neuron of the discrimination part . It is defined as follows : D ( y1 ) = { 1 , y1 > 0 , ( 2 ) 0 , otherwise . ( 2′ ) where i ∈ 1 , 2 , ... , C , C is number of the output neurons of the CNN , yi is the output value of the ith output neuron of the discrimination part . All nodes on the feature predicate layer and every node on the decision layer are connected to one or more nodes on the middle layer , namely as disjunction layer , with true or false edges . Every node represents a disjunction relation , which is expressed by a disjunctive normal form . It is worth mentioning : if a node is connected to a node on the disjunction layer by a false edge , its predicate follows after a non-operator in the disjunctive normal form . The potential function of a disjunctive normal form can be obtained by using the Lukasiewicz method ( Giles , 1975 ) . φc ( yi ) = min ( 1 , T ( Γ , yi ) ) ( 3 ) where T ( Γ , yi ) = ∑N j=1 { aj [ 1− Zj ( Γ ) ] + ( 1− aj ) Zj ( Γ ) } + ( aN + 1 ) D ( yi ) and N is the number of the nodes on the feature predicate layer . If aj = 1 , there is a false edge . Otherwise , there is a true edge . The conditional probability distribution that a ground DCLM including all disjunctive normal forms is true is p ( y , Γ ) = 1 Ξ exp ( ∑G i=1 λiφci ( yi ) ∑G i=1 λi ) ( 4 ) where G is the number of all ground formulas , Ξ = ∑ Γ∈F exp ( ∑G i=1 λiφci ( yi ) ∑G i=1 λi ) is a partition function , y = ( y1 , y2 , ... , yG ) , yi is an output value of the CNN and λi is a weighted value of the ith ground formula . By maximizing its likelihood function , the optimal ai and λi in the DCLM can be obtained . C ( Γ ) = arg max ai , λi [ log p ( y , Γ ) ] = arg max ai , λi ( ∑ i λiφci ( yi ) ∑ i λi − log ( Ξ ) ) ( 5 ) For extracting a optimal DCLM , a Maximum A Posterior ( MAP ) algorithm on the Maximum Satisfiability Problem ( MAX-SAT ) was designed . Using the disjunction normal form with the greatest weighted value in the optimal DCLM , a prediction of an input image can be obtained . 4 EVALUATION OF INTERPRETABILITY PERFORMANCE . We consider that if the discrimination part of a CNN has a similar shape of function curve with its optimal interpretable model , the former can be easily explained by the latter . Therefore , the interpretable performance of the discrimination part can be measured by the shape similarity between it and its optimal interpretable model . We posit that given the same input data set , the similarity may be measured by variance of differences between outputs of the both models . It can be named interpretation distance . It is easily proved that the smaller the interpretation distance is , the more similar their shapes are , and the better the interpretability performance of the discrimination part would be . Definition 1 If X is a compact metric space and ν is a Borel measure in X , such as Lebesgue measure or marginal measures , inL2ν ( X ) , a square integrable function space on X , the interpretation distance , φd ( P ∗ , f ) , between a discrimination part f ( x ) and its optimal DCLM P ∗ ( x ) is φd ( P ∗ , f ) = ∫ Z ( f ( x ) − P ∗ ( x ) − µP∗ ( f ) ) 2dν ( 6 ) where µP ∗ ( f ) = ∫ Z ( f ( x ) − P ∗ ( x ) ) dν ( 7 )

Summary: This paper proposes an interesting framework for training interpretable CNNs, similar to distillation methods. The authors propose a probabilistic model to approximate CNN predictions (specifically the discriminatory part i.e. fully connected network, and a procedure for training CNN+ DCLM as a game. Results show interesting performance over benchmark datasets in comparison to existing distillation baselines.