# Automated Dataset-Creation and Evaluation Pipeline for NER in Russian Literary Heritage
## 2 Related Work
### Automating NER Dataset Creation  
Through exploring the previous datasets, many of them used manual methods to annotate the dataset. This method ensures the highest level of accuracy since it depends on experts or volunteers to annotate the entities [12; 13; 14]. Using automation methods that utilize LLMs, pre-trained NER models, or synthetic data generation allows the creation of larger datasets but with variable accuracy [8; 27; 28]. Some datasets use semi-automating techniques to create links between the entities and auto-annotating for various lemmas based on larger dictionaries [29].  
Creating a dataset for solving NLP tasks is an extremely labor-intensive process, prompting researchers to seek automation methods. For instance, the study [30] describes a process for creating datasets for NER and NEL tasks in Greek. The authors used data from Wikipedia, annotating texts for entities of the following types: event, facility, geopolitical entity, location, organization, person, product, and work of art. Additionally, for the NEL task, the dataset was expanded through automatic translation and by supplementing missing links. The resulting dataset was employed to test multiple models. Another approach to dataset creation focuses on identifying artworks, as demonstrated in [31]. This study also relied on Wikipedia articles, with dataset generation based on analyzing the references provided within the articles.  
Textual noise, like spelling or typographical or OCR error could also affect the quality of named entity recognition, as shown in [32]. Existing NER models react differently to various noise sources, so it is important to combine results from several models to obtain better results on most types of texts.  
This study proposed a semi-automated pipeline with a voting system to create the NER dataset. The method tries to balance between accuracy and time efficiency while ensuring no biases in the annotated entities. The use of multiple models for dataset preparation has also been explored in [33]. This study focused on creating a medical dataset for annotating medical reports, which significantly accelerated the dataset-preparation process. A similar method was employed in [34], where large language models were used to classify texts. Interestingly, a comparison with a baseline model based on the bag-of-words approach and [PERSON] demonstrated that traditional methods achieved comparable classification quality. Furthermore, when considering cost, traditional methods significantly outperformed LLMs.  
By analyzing the available datasets for NER tasks in Russian, we can notice that they were built for general purposes and were collected from common resources on the web like Wikipedia and none of them were targeted for the literature domain. So in this paper, we produce our dataset for literature purposes by annotating works of the Russian poet  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Dataset** & **Language** & **Source** & \begin{tabular}{c} **Entity** \\ **Types** \\ \end{tabular} & \begin{tabular}{c} **Annotation** \\ **Method** \\ \end{tabular} & **Availability** & **Notable Features** \\ \hline WikiNEuRal\_RU [27] & \begin{tabular}{c} English, \\ French, \\ Spanish, \\ German, \\ Italian, \\ Russian, \\ Dutch, \\ Polish \\ \end{tabular} & \
\begin{tabular}{c} PER, \\ LOC, \\ ORG, \\ MPGuse, \\ MISC \\ \end{tabular} &
\begin{tabular}{c} \\ LOC, \\ ORG, \\ MPGuse, \\ MPGAlexander [PERSON]. This dataset plays a crucial part and will be a significant resource for many NLP tasks related to processing of literature heritage of [PERSON] and in particular NER.