# A Simple Convolutional Neural Network with Rule Extraction
## 4 Experimental Results
### General Results  
A CNN including a convolutional layer with 120 kernels of size one, two and three was defined empirically (cf. Section 3.1.3). Note that our purpose was not to find the best possible CNN architecture, but rather to create an acceptable CNN in terms of performance and then to generate rules to explain classifications. Training was performed with Lasagne libraries, version 0.2 [41]. The training parameters were:  
* learning rate: 0.02;
* momentum: 0.9;
* dropout = 0.2;  
To illustrate the results of rule extraction, we applied the CNN architecture defined above to a well-known binary classification problem related to \"tweets\" of movie reviews [42]. The characteristics of the dataset are:  
* number of samples: 10,662;
* maximal number of words in a \"tweet\": 59;
* positive sentiment samples: 5331;
* negative sentiment samples: 5331;
* single words: 21,426;
* bigrams: 111,590;
* trigrams: 174,628. The positive and negative subsets have been divided into 10 folds, in order to carry out ten-fold cross validation trials. Moreover, a randomly selected subset of the training set representing 10% of it was extracted as a tuning set for early-stopping [43], which is useful to avoid over-training. Table 3 illustrates the results. The first row of this Table is related to the original CNN, while the other ones provide results obtained by interpretable CNNs, which depend on the number of stairs in the staircase activation function. Columns from left to right designate:  
* average train accuracy;
* average predictive accuracy on the testing set;
* average fidelity, which is the degree of matching between generated rules and the CNN;
* average predictive accuracy of the rules;
* average predictive accuracy of the rules when rules and CNN agree;
* average number of extracted rules;
* average number of rule antecedents. The average predictive accuracy of the rules is slightly lower than that obtained by the CNN (73.7% versus 74.1%, for the best result). Moreover, average fidelity on the testing set is above 95%, meaning that rules explain CNN responses in a large majority of cases. Finally, the best average predictive accuracy of the rules when rules and model agree is higher than that obtained by the CNN (75.1% versus 74.1%). \begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline  & **Tr. Acc.** & **Tst. Acc.** & **Fid.** & **Rul. Acc. (1)** & **Rul. Acc. (2)** & **\#Rul.** & **\#Ant.** \\ \hline CNN & 82.1 (1.3) & 74.1 (1.1) & – & – & – & – & – \\ CNN (\(\Theta=100\)) & 82.1 (1.3) & 74.1 (1.1) & 95.3 (0.6) & **73.7** (1.0) & **75.1** (1.0) & 651.7 (53.1) & 5368.9 (379.3) \\ CNN (\(\Theta=200\)) & 82.2 (1.4) & 74.1 (1.0) & **95.8** (0.4) & 73.7 (1.0) & 74.9 (1.0) & 573.8 (49.0) & 5040.7 (354.0) \\ CNN (\(\Theta=500\)) & 82.1 (1.4) & 74.1 (1.1) & 95.6 (0.8) & 73.6 (1.0) & 75.0 (0.9) & **566.1** (28.0) & **4980.7** (364.5) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Average results based on cross-validation. The models are CNNs and interpretable CNN approximations with varying numbers of stairs in the staircase activation function. Standard deviations are given between parentheses. As a baseline, a simple pedagogical technique aiming at explaining CNN predictions is represented by a DT that learns training samples with CNN prediction classes, instead of true labels. Table 4 illustrates the results obtained by cross-validation. The \(\lambda\) parameter, which is the minimum number of samples required to be at a leaf node makes it possible to control the size of the trees. The number of nodes of a tree is in turn related to the proportion of learned training samples. It is worth noting that the fidelity on the training samples decreases when \(\lambda\) is increased. Moreover, the average predictive accuracy is never above 60%, the average fidelity being always below 63%. On one hand, this performance is substantially lower than that obtained by interpretable CNNs with DIMLP subnetworks (see Table 3). On the other hand, with \(\lambda\geq 10\), a lower number of rules are generated.