# Automated Dataset-Creation and Evaluation Pipeline for NER in Russian Literary Heritage
## 4 Automated Pipeline Description for Dataset Creation and Evaluation
The goal is to elect high-quality and trusted entities for further steps depending on the majority of the votes. This leads to the following:  
\[E_{k}=\{x\in R_{k}\mid\sum_{i=1}^{N}S_{i,k}(x)\geq T\}\text{ where, }T\in\{1,2,3,4\} \tag{4}\]  
We experimented on a test set with various values of \(T\), as shown in Figure 6. The results show the change in the overall precision, recall, and F1-score by changing the minimal number of voted models to accept a given entity in the final nominated entities. It is clear that by increasing \(T\) the trust in the accuracy of the nominated entities increases. To validate and verify the proposed approach, we tested each of the pre-trained models on the test set and the results are shown in Table 5. We also tested the voting system (by considering the most trusted threshold \(T=4\)) on the test set, and the results are shown in Table 6. The results show high precision for all of the entity types nominated by the voting system compared to relying on one model's prediction. Also, the overall precision of the system is higher by 28% than using only the SOTA model (DeepPavlov) for auto-annotating. This leads us to trust the predictions auto-annotated by the system. Since it guarantees a high probability of accurately identifying entity spans and types within the dataset, this enhancement can significantly improve the quality of the final dataset after checking by specialists, reduce the time spent verifying, and facilitate the process of creating the dataset by reducing the human interaction factor. Figure 6: The results of the voting system by changing the threshold on the test set. To assess the impact of the voting system in saving specialists' time and effort during annotation verification, we calculated the number of entities needing changes (additions, deletions, or modifications) after applying the enhanced approach and compared it to the best single pre-trained model (DeepPavlov) with the highest accuracy on the test set. The impact of the voting system by reducing the total number of entities that needed to be fixed and verified by humans is shown in Figure 7 (where the \(x\)-axis shows the entity's type and the \(y\)-axis shows the amount of entities). This impact also appears clearly in the number of incorrect entities predicted by DeepPavlov compared to the enhancement approach, as shown in Figures 8 and 9. The voting system achieved better results, since it had significantly fewer incorrect annotations compared to the predictions of the single model. This reflects the importance of the voting system because incorrect entities require more time and effort to fix than missed entities. To fix an incorrect entity, the specialist has to either change the entity or delete it, where missed entities only need to be added. When the auto-annotation process guarantees high-quality prediction and fewer incorrect entities, it gives the chance to the specialists to focus more on missed annotations (rather than checking everything), which will reduce processing time and enhance the quality. The proposed system's generalized concept can be used to create similar datasets in different languages. It leverages the voting system to facilitate the annotation process, reducing time and cost while ensuring trusted results. We are considering the usage of an enhancement approach for annotating the rest of the volumes (Volumes 2 to 5), since it facilitates the process and saves time while providing high-quality and trusted output. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Entity** & **Precision** & **Recall** & **F1-Score** \\ \hline Overall & 0.9505 & 0.3812 & 0.5442 \\ PER & 0.9499 & 0.6957 & 0.8032 \\ DATE & 0.8157 & 0.7146 & 0.7618 \\ LOC & 0.9565 & 0.4177 & 0.5815 \\ ORG & 1.0000 & 0.0926 & 0.1695 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The validation results of the voting system on the test set.