# A Simple Convolutional Neural Network with Rule Extraction
## 2 Related Work
### Explanation of CNN Classifications  
On object recognition problems, [PERSON] et al. explained CNN classifications by training a second network that learns to produce sentences as an explanation feature [23]. Specifically, it learned to generate sentences that realized a global sentence property by reinforcement learning. [PERSON] et al. proposed to generate a heat map from a CNN to localize important regions of the image for classification [24]. The key idea behind this approach is the use of the Kullback-Leibler divergence gradient. [PERSON] et al. introduced Layer-wise Relevance Propagation (LRP) to determine the inherent reasoning of deep neural networks [25]. In practice, this technique generated heatmaps of relevant areas contributing to the final classification. Furthermore, the authors characterized the importance of the context, with respect to targeted objects in images. For medical images, [PERSON] et al. used very deep CNN architectures with residuals [3]. To understand the internal structure of the trained networks, they proposed to replace the majority of convolutional layers by AM-FM components (Amplitude Modulation-Frequency Modulation) and to retrain the upper network layers. Overall, convolution layers were visualized through their frequency coverage.  
[PERSON] and [PERSON] presented DeconvNet [26], in which strong activations are propagated backward to determine parts of the image causing these activations. [PERSON] and [PERSON] proposed an optimization method based on image priors to invert a CNN [27]. As a result, visualizations yielded insight on the information represented at each layer. [PERSON] and [PERSON] proposed to analyze which data is preserved by a feature representation and which data is discarded [28]. Specifically, from a feature vector a neural network was trained to predict the expected pre-image, corresponding to the average image producing the given feature vector.  
Several authors proposed to interpret CNNs in the neighborhood of the instances. As an example, [PERSON] et al. presented LIME (Locally Interpretable Model Agnostic Explanations), whose purpose is to learn an interpretable model in the local region close to an input instance [11]. A first disadvantage of this approach is that the data must be transformed into binary format. Secondly, local explanations were only provided through linear models and their features relevance. Similarly [PERSON] et al. introduced the Model Explanation System [29]. Basically, by using a Monte Carlo technique, they derived a scoring system for finding the best explanation. [PERSON] et al. determined the training instances that are the most important for the prediction [30]. Finally, LORE (Local Rule Based Explanations) learns a local decision tree on a synthetic neighborhood generated through a genetic algorithm [13]; then, rules are generated to explain the outcome, as well as counterfactual rules.  
[PERSON] and [PERSON] proposed to use a CNN to train a particular decision tree that imitates the input-output associations found out by the trained CNN [31]. Nevertheless, the decision tree did not explain the network logic, clearly. As an improvement, [PERSON] et al. proposed a decision tree to explain CNN predictions at the semantic level [32]. Specifically, the decision determines with respect to image recognition tasks, which filters are activated by objects parts and how much they contribute to the final classification. Overall, their technique provided an approximate explanation of CNN classifications.  
The Explanation of CNNs responses in text classification has been rarely explored, as stated in [33]. The authors show that one-dimensional convolution filters can be associated to different semantic classes of n-grams. Moreover, the max-pooling layer is able to separate important n-grams. In [34] the Layer-wise Relevance Propagation (LRP) estimates which individual words are relevant to the overall classification decision.