# EmmDocClassifier: Efficient Multimodal Document Image Classifier for Scarce Data
## 5 Discussion and Evaluation of the Results  
In this section, we will evaluate the results in detail. We will walk through some examples where the model behaved differently than expected, for example, incorrectly classified to a different class. We will also discuss possible future directions for improvements. Figure 9 shows some examples of correctly and wrongly classified documents. First, as we train the model initially on RVL-CDIP dataset for the first experiment, we set all hyper-parameters. Then, once we receive good results, we proceed with transfer learning. As we can see in Figure 10, although all the classes are classified correctly to a greater extent, some of the classes are confused with other classes ranging from 3 to 4%. This high misclassification is due to more overlapping features between them. For example, the two classes, Scientific Publication and Scientific Report, are mutually misclassified in almost 3% of the cases. Furthermore, if we look carefully into a few images of the \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**Imagenet + RVL-CDIP**} & \multicolumn{2}{c}{**Imagenet + No RVL-CDIP**} \\ \hline
**Authors** & **Image** & **Both Modalities** & **Image** & **Both Modalities** \\ \hline [PERSON] et al. (2014) [55] & - & - & 43.8 & - \\ \hline [PERSON] et al. (2014) [56] & - & - & 65.37 & - \\ \hline [PERSON] et al. (2015) [1] & - & - & 77.6 & - \\ \hline [PERSON] et al. (2015) [8] & 89.8 & - & 79.9 & - \\ \hline [PERSON] et al. (2016) [12] & - & - & - & 79.8 \\ \hline [PERSON] et al. (2017) [7] & 91.13 & - & - & - \\ \hline [PERSON] et al. (2018) [9] & 92.21 & - & - & - \\ \hline [PERSON] et al. (2019) [23] & - & - & 84.5 & 87.8 \\ \hline [PERSON] et al. (2019) [10] & 93.2 & 95.8 & - & - \\ \hline [PERSON] et al. (2020) [11] & 91.3 & 96.94 & - & - \\ \hline [PERSON] et al. (2020) [13] & 94.04 & 94.9 & 85.99 & 89.47 \\ \hline Our approach & **94.04** & **95.7** & **86.0** & **90.3** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Accuracy comparison of the proposed method to the former methods on Tobacco-3482 dataset. Our work surpasses all previous performance compared to only those approaches that do not train on the bigger dataset. We have highlighted best performing method in bold. classes Scientific report and Presentation, we can see many similarities in visual and textual features. That explains the failure for around 4% of the cases in the confusion matrix. Furthermore, if we look at the classes E-mail and Memo, we see that they have minimal overlap with any other classes, which is justified by the precise boundary of the features from other class features. As depicted in Figure 11, the overall behavior of the model can be understood while being tested on the Tobacco-3482 dataset. Whether or not the model is pre-trained with the RVL-CDIP dataset, the model can easily classify certain classes such as E-mail, Form, Memo, etc. As we can see in Figure 1, classes such as E-mail, Form, and Memo have apparent visual and textual features completely independent and non-overlapping with other classes. Hence, both visual and textual streams deliver the features that help the classifier categorize them into their corresponding classes. However, this section focuses on those scenarios where the classifier failed to classify the input with its respective class. As we can see from the Confusion Matrix Figure 11, the model with no pretraining on the RVL-CDIP dataset performs better on most of the  
Figure 10: The figure depicts the confusion matrix of the model on RVL-CDIP test data. Figure 9: This figure is a collection of 10 different images taken randomly from the dataset, in which 5 of them are successfully classified, while the other 5 were incorrectly classified. The potential reasons for failure in identifying the classes of these input images are discussed in detail in Section 5.  
classes except for News, Note, and Scientific. On a higher level, the reason is that these classes have a high overlapping number of features with other classes.