# Automated Dataset-Creation and Evaluation Pipeline for NER in Russian Literary Heritage
## 4 Automated Pipeline Description for Dataset Creation and Evaluation  
This section introduces an enhanced approach to the NER annotation process. A pipeline illustrates the proposed procedure and highlights the evaluation and testing results that support and validate the approach. In the annotating process, a pre-trained NER model was used to automatically predict entities for types (PER, ORG, LOC, DATE). This is a common method for automating the annotating process and usually it is followed by human interaction to correct any mistakes. We propose to use multiple pre-trained models for auto-annotation instead of relying on only one model. A voting system that checks the predictions for each model and then elicits the common entities was used to choose which entities should be kept. The voting system could give weights separately for each model depending on the model's accuracy which can affect how much should be relying on the model's decision. Another approach for the voting system could be to use majority voting or a certain percentage of the votes, depending on the case study. For our research, we used a voting system that elicited an entity only when all chosen models voted for it. It provides higher precision, which means that the elicited entities have a high chance of accurately belonging to the predicted type. Figure 4: Visualizing sample from the dataset. Figure 3: The distribution of entity types within the dataset. \begin{table}
\begin{tabular}{c c c c} \hline
**Entity Type** & **Total Number of Enities** & **Total Tokens** & **Average Tokens per Entity** \\ \hline WOA & 3206 & 16,184 & 5.048 \\ PER & 7851 & 11,653 & 1.484 \\ DATE & 2539 & 6196 & 2.440 \\ LOC & 890 & 1007 & 1.131 \\ ORG & 221 & 443 & 2.004 \\ \hline \end{tabular}
\end{table} 
Table 3: Statistics about each entity type in the dataset. In other words, we can trust that the entities chosen by the system are correct and valid. The proposed pipeline for the voting system is shown in Figure 5.  
The proposed voting system depends on four models (Natasha [37], Stanza [38], Flair [39], DeepPavlov [35]). Not all of these models can recognize all the four types of entities that we want to annotate. Table 4 shows entity types recognized by each model. The voting system elicits a prediction only when all of the models vote for the entity taking into account the model's ability to predict the entity's type. The general mathematical modeling for the voting system is described in the following way. Let us first define the following items:  
* \(M_{i}\): Set of models used in the system where \(i\in\{1,2,\ldots,N\}\);
* \(C_{i}\): Set of coefficients referring to the importance of each model for the voting system, where \(i\in\{1,2,\ldots,N\}\);
* \(R_{k}\): Set of recognized entities of type \(k\), where \(k\in\{1,2,\ldots,K\}\);
* \(M_{i}^{k}\): Set of entities recognized by model \(M_{i}\), and from type \(k\);
* \(T\): The threshold for electing entities in the voting system. The selection function:  
\[S_{ijk}(x)= \begin{cases}0&x\
otin M_{i}^{k}\\ 1&x\in M_{i}^{k}\end{cases}  \tag{1}\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **Entity Types** \\ \hline Natasha & PER, ORG, LOC \\ Stanza & PER, ORG, LOC \\ Flair & PER, ORG, LOC, DATE \\ DeepPavlov & PER, ORG, LOC, DATE \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The recognized types of entities considered by each NER model. Figure 5: The pipeline for the enhancement approach with the voting system. The set of elected entities from type k is defined as follows:  
\[E_{k}=\{x\in R_{k}\mid\sum_{i=1}^{N}C_{i}S_{i,k}(x)\geq T\} \tag{2}\]  
The final set of entities recognized by the voting system is:  
\[E=\bigcup_{1}^{K}E_{k} \tag{3}\]  
In the proposed system, Equation (2) was simplified by giving the models the same importance \(C_{i}=C=1\). This is because all of the models were trained on general-purpose data and the task is focused on literature works. So, giving significant importance to any of them can lead to bias in the final results.