# Text Line Extraction in Historical Documents Using Mask R-CNN
## 7 Results and Discussion
### Experimental Results
#### 7.2.1 VML-AHTE and DIVA Datasets  
For DIVA and the VML-AHTE datasets, the model was trained and tested using binary images. Since the refinement stage assumes straight horizontal lines, weakening the generality of the method, we evaluate the method twice, with and without the refinement stage (denoted as MRCNN+ and MRCNN, respectively). In order to evaluate the results using the ICDAR2013 metrics described in Section 6, we have extracted pixel level segmentation by labeling each foreground pixel (text pixels) in the binarized image with its corresponding label in the bounding polygon prediction generated by the proposed method (see Figure 10). We compared the proposed method with the methods in task-3 of ICDAR 2017 competition on layout analysis for medieval manuscripts [27]. The scope of task-3 is limited to the main text lines and not the interlinear glosses. We removed these glosses prior to all our experiments using the ground truth. It should be noticed that task-3 participants also removed these glosses using their own algorithms. Table 1 presents the results of our method against those from ICDAR 2017 competition on layout analysis for challenging medieval manuscripts for text line extraction (task-3). We can see from the Table 1, the refinement stage almost always improves the accuracy of the MRCNN method. With refinement, the Mask R-CNN method (MRCNN+) outperformed all of the competition participants in Pixel IU. MRCNN without refinement performs similarly to the four best preforming methods in the competition. Furthermore, Table 2 shows the ICDAR 2013 evaluation metrics results of the method on the Diva dataset. \begin{table}
\begin{tabular}{l c c c c c c} \hline  & \multicolumn{2}{c}{**CB55**} & \multicolumn{2}{c}{**CSG18**} & \multicolumn{2}{c}{**CSG863**} \\ \hline  & **LIU** & **PIU** & **LIU** & **PIU** & **LIU** & **PIU** \\ \hline System-2 (BYU) & 84.29 & 80.23 & 69.57 & 75.31 & 90.64 & 93.68 \\ System-6 (IAIS) & 5.67 & 30.53 & 39.17 & 54.52 & 25.96 & 46.09 \\ System-8 (CIT-lab) & **99.33** & 93.75 & 94.90 & 94.47 & 96.75 & 90.81 \\ System-9 + 4.1 (DIVA + MG1) & 98.04 & 96.67 & 96.91 & 96.93 & **98.62** & 97.54 \\ \hline MRCNN & 94.13 & 95.17 & 96.97 & 96.90 & 97.55 & 97.24 \\ MRCNN+ & 95.42 & **98.71** & **98.39** & **98.65** & 97.24 & **97.55** \\ \hline \end{tabular}
\end{table} 
Table 1: The results of the ICDAR2017 competition on layout analysis for challenging medieval manuscripts ([27]). Line IU and Pixel IU results of Task 3 of all competition participants and the proposed method. LIU denotes Line IU, and PIU denotes Pixel IU. Figure 10: Mask R-CNN segmentation results: (**a**) an input page, (**b**) blobs prediction, and (**c**) pixel level prediction. Table 3 shows the proposed method performance on the VML-AHTE dataset. It lists the evaluation measures for the polygons and pixel level results, calculated using DIVA and ICDAR2013 line segmentation evaluator tools, respectively. In the above experiment, DR values are higher than RA values. Since the number of ground truth elements is constant, high DR values mean a high number of one to one matches. Consequently, low RA values in the presence of high number of one-to-one matches are caused by the high number of false predicted lines. This observation directly leads to the question why the Line IU values does not reflect the low RA values. The answer follows from the fact that ICDAR2013 matching score is 90% while the DIVA evaluator matching score is 75%, meaning that the evaluators use different threshold values. Another observation is the fluctuating pattern of the RA values compared to the RA values, while the flow of Line IU and Pixel IU values is parallel. Such counter-intuitive behaviour of a metric is not preferable as it reduces the interpretability of the experimental results. Such behaviour is a direct result of the fact that the DIVA evaluator cannot handle text lines with multiple disconnected polygons.