# Fibers of Failure: Classifying Errors in Predictive Processes
## 3 Results and Discussion
### MNIST Model
#### 3.1.2 Qualitative  
For the qualitative analysis, we chose to focus on four groups with digit five as the ground truth digit: group 50, which is not one of the failure mode groups, and groups 30, 40, and 47, all part of the total 39 failure mode groups. The location of each group in the Mapper graph is shown in Figure 6. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Dataset** **(Number of Images)** & **5-Fold-Test (4000)** & **C-MNIST-Eval (10,000)** \\ \hline CNN & 69.40\% & 41.14\% \\ CNN+LR & 75.45\% & 59.57\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance of the CNN as compared to CNN with FiFA-driven improvements both on the average of the five folds of test data (5-fold-test) and on entirely corrupted test data (C-MNIST-eval). The improvements by the classifier ensemble are for the best performing parameters. The FiFA-driven improvement produces an 18.43%pt increase in accuracy on the C-MNIST-eval dataset, which consists of only corrupt MNIST images. In addition, the percentage of clean images of the adjusted predictions by the correction layer was only 0.21%. The distributions of predicted probabilities for each digit in the three failure mode groups are shown in Figure 7: group 30 is the group with the highest probability to the digit five, while 40 and 47 are more focused on eight, two, and three. All three groups favor digit eight, as their mean probabilities are between 0.5 and 0.9. We compared these three failure modes with the non-failure group 50 and extracted the five activations with the highest KS-values from the Dense-128 layer; see Figure 4. To illustrate the differences between the three failure modes regarding the activations, we have provided a selection of saliency maps [79] for all images considered as true members of each of the three failure mode groups. These were all produced using the keras-vis Python package. Figure 6: One of the five Mapper graphs created by the activations from the CNN model on four of the five folds, i.e., 16,000 images (5-fold-training), as explained in Section 2.8.1. The graph is colored with the probability of predicting the ground truth digit. The colorbar is for interpreting the values of the coloring. The circled nodes and edges are the groups 30, group 40, 47, and 50. The other four Mapper graphs are shown in the Supplementary Material [78]. Figure 7: The failure modes for a ground truth of five. We see the distributions of predictions for the three failure modes: only group 30 attaches any significant likelihood to the digit 5 at all, while all three favor eight. For group 40, the digits two and three are also commonly suggested; this happens somewhat more rarely in groups 30 and 47. Figure 8 shows a selection of noisy images and their saliency maps for some of the activations' highest KS-values within the Dense-128 layer. The two leftmost image pairs were selected based on visual clear saliency maps with respect to digits. The two rightmost were selected based on most unclear/noisy saliency maps. The full collection of saliency maps for these groups can be found in our Supplementary Material [78]. Figure 8: Examples of noisy images and saliency maps for activations in the penultimate dense layer for the three main failure modes identified for noisy 5s. The two leftmost images were chosen as the most clear saliency maps with respect to digits. The two rightmost were selected based on unclear/noisy saliency maps. All saliency maps are from images classified as members of the respective failure mode group. All saliency maps can be found in our Supplementary Material [78]. The activations 24 and 81, present in all three groups, display activity that is consistent with an activation detecting features of the digit five. The activations 89 and 99 correspond closer to an activation for the digit three and 119; 122 and 124 correspond to activations for the digit eight. In particular, in the last three groups, noise that closes loops in a written five tend to have high saliency.