# A Simple Convolutional Neural Network with Rule Extraction
## 4 Experimental Results
### General Results
We may wonder about replacing the DIMLP subnetwork in interpretable CNNs by Decision Trees, from which rules can be extracted. Note that a decision tree can be viewed as a set of rules, with each rule represented by a path going from the root to a leaf. Table 5 illustrates the results with respect to the extracted rules, by varying the \(\lambda\) parameter. The best average predictive accuracy is equal to 68.6%, which is much lower that that obtained by CNNs with the DIMLP subnetwork (68.6% versus 73.7%). An intuitive reason explaining this result is that the fully connected layer of the original CNN is better approximated by DIMLPs than DTs. However, DTs generate a significant smaller number of rules, on average: 201.8 versus 573.8. Support Vector Machines are usually very competitive with very highly dimensional classification problems. Here we have 17,700 (\(59*300\)) input variables for each sample; thus, a natural question is \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **Tr. Acc.** & **Tst. Acc.** & **Tr. Fid.** & **Tst Fid.** & **\#RuL.** & **\#Ant.** \\ \hline DT (\(\lambda=1\)) & **82.1** (1.3) & 57.5 (1.4) & **100.0** (0.0) & 60.9 (0.8) & 797.9 (16.9) & 10,354.3 (704.6) \\ DT (\(\lambda=5\)) & 78.6 (1.1) & 57.9 (0.9) & 94.3 (0.2) & 61.0 (1.0) & 615.7 (6.4) & 7134.4 (304.6) \\ DT (\(\lambda=10\)) & 75.6 (0.8) & 58.4 (1.3) & 89.1 (0.2) & 61.4 (1.3) & 456.8 (7.4) & 4863.7 (149.3) \\ DT (\(\lambda=15\)) & 73.3 (0.7) & 58.5 (1.4) & 85.4 (0.4) & 61.6 (0.8) & 362.2 (5.6) & 3666.2 (116.4) \\ DT (\(\lambda=20\)) & 71.9 (0.6) & 58.5 (1.5) & 82.9 (0.3) & 61.5 (2.2) & 299.5 (4.5) & 2904.5 (86.4) \\ DT (\(\lambda=25\)) & 70.6 (0.6) & 58.6 (1.6) & 81.1 (0.4) & 61.4 (1.8) & 253.9 (3.5) & 2363.1 (71.6) \\ DT (\(\lambda=30\)) & 69.8 (0.6) & 59.3 (1.0) & 79.8 (0.4) & 62.1 (1.8) & 222.7 (4.4) & 2015.9 (52.1) \\ DT (\(\lambda=35\)) & 69.0 (1.0) & 59.3 (1.6) & 78.5 (0.4) & 62.5 (1.5) & 194.4 (4.2) & 1707.9 (64.1) \\ DT (\(\lambda=40\)) & 68.5 (0.7) & **59.7** (1.8) & 77.4 (0.3) & 62.3 (1.4) & 173.9 (3.8) & 1482.8 (42.4) \\ DT (\(\lambda=50\)) & 67.6 (0.6) & 59.6 (1.1) & 75.9 (0.4) & **62.7** (1.2) & **143.8** (2.7) & **1168.4** (31.5) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Average results obtained by Decision Trees (DTs) trained to learn datasets with CNN targets, instead of true labels. Columns from left to right represent average results on: Training accuracy; predictive accuracy; fidelity on the training set; fidelity on the testing set; number of extracted rules; and number of antecedents in the rules. In the rows, the \(\lambda\) parameter controlling the size of the trees varies. \begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **Tr. Acc.** & **Tst. Acc.** & **\#RuL.** & **\#Ant.** \\ \hline DT (\(\lambda=1\)) & 100.0 (0.0) & 66.0 (1.6) & 943.2 (23.6) & 11,160.8 (367.7) \\ DT (\(\lambda=5\)) & 93.4 (0.2) & 66.8 (1.4) & 640.0 (16.3) & 6710.8 (191.2) \\ DT (\(\lambda=10\)) & 89.0 (0.3) & 67.0 (1.4) & 450.4 (9.3) & 4414.5 (97.7) \\ DT (\(\lambda=15\)) & 86.5 (0.4) & 67.6 (1.1) & 342.6 (5.6) & 3198.0 (60.5) \\ DT (\(\lambda=20\)) & 84.8 (0.4) & 68.0 (1.0) & 276.8 (7.4) & 2483.2 (69.9) \\ DT (\(\lambda=25\)) & 83.6 (0.4) & 67.9 (1.3) & 234.6 (4.7) & 2043.1 (38.4) \\ DT (\(\lambda=30\)) & 82.7 (0.4) & **68.6** (0.9) & 201.8 (4.2) & 1704.5 (38.1) \\ DT (\(\lambda=35\)) & 82.0 (0.5) & 68.3 (1.1) & 178.0 (3.5) & 1468.0 (40.1) \\ DT (\(\lambda=40\)) & 81.3 (0.5) & 68.3 (1.2) & **159.1** (3.5) & **1281.6** (29.0) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Average results obtained by the rules generated from Decision Trees that replace the fully connected layer of CNNs. Columns from left to right represent average results on: Training accuracy; testing accuracy; number of extracted rules; number of antecedents in the rules. In the rows, the \(\lambda\) parameter controlling the size of the trees varies. whether SVMs perform better than CNNs. Due to the high input dimensionality it is recommended to use linear SVMs.