# Advanced Techniques for Geospatial Referencing in Online Media Repositories
## 4 Analysis  
The methodology described was applied to ten videos from ARD Mediathek with a total of 24,050 extracted frames. Care was taken to evaluate primarily those videos with more frequent location references, i.e., contributions from documentary categories, particularly contemporary city and regional documentaries or history documentaries. Furthermore, only those videos with an available subtitle track were selected for comparison with the transcribed audio tracks. The video length, date of release, and other parameters were randomly selected. We manually determined location references from the videos as a reference value, separated according to the different methods that the automatic information extraction and the NER perform, i.e., the visible image, text in the image, and audio track or subtitles. An evaluation was conducted to find a suitable tool for NER. A small metadata sample (title and description) of 50 media library entries was compared with different NER tools. Since the focus is on extracting spatial information, the selection of suitable NER tools is based on their performance at extracting location information. [PERSON] et al. (2023)[PERSON] et al. (2023) describe a comprehensive comparison of NER tools in their study. Based on these results, NER tools were tested for the toponym extraction of media library entries. The requirements were a simple setup and usability with Python. * gpt-3.5-turbo: gpt-3.5-turbo is a large language model developed by OpenAI and optimized for chatbots ([PERSON] et al., 2018). The prompt was defined to output locations in the text in a standardized and parsable format to extract location references. * Flair NER: Flair is an NLP framework for facilitating the training and distribution of sequence labeling and text classification. Flair-NER is a standard four-class NER model trained on CoNLL-03. Locations were identified directly using the trained model by querying the LOC tag ([PERSON] et al., 2019; [PERSON], 2020). * GeoTxt: GeoTxt recognizes and extracts location references from text ([PERSON], 2020). * spaCy: It is a general NLP tool. The model de_core_news_md was used, and the LOC entities recognized by spaCy were kept as locations ([PERSON], 2020). * Stanford Core NLP: This Java implementation of a CRF-based NER was developed and maintained by the Stanford Natural Language Processing Group ([PERSON], 2020). It is used via the official Stanza package. The LOC (location) entities recognized by the Stanford NER were retained as locations. * Stanza: It is a general NLP toolkit with an NER tool built on BiLSTM and CRF, also developed by the Stanford NLP Group ([PERSON], 2020). The LOC-type entities were retained as locations. The NER model used is GermEval2014 ([PERSON] et al., 2019). As shown in Table 1 the most striking results are that Flair NER has the highest precision and a high F1 score, indicating the accurate recognition of place names. On the other hand, GeoTxt has the lowest precision, the lowest recall, and the lowest F1 score, meaning a poorer overall performance. Stanza shows an excellent overall balance between precision, recall, and F1 score. Based on this evaluation, Flair was used for its performance. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Tool** & **Precision** & **Recall** & **F1 Score** \\ \hline gpt-3.5-turbo & 0.61 & 0.79 & 0.67 \\ \hline Flair NER & 0.85 & 0.80 & 0.83 \\ \hline GeoText & 0.57 & 0.11 & 0.19 \\ \hline spaCy & 0.63 & 0.46 & 0.53 \\ \hline Stanford Core NLP & 0.67 & 0.55 & 0.60 \\ \hline Stanza & 0.78 & 0.78 & 0.78 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Evaluation of named entity recognition tools. Entity used: Location. Based on this, the German NER (large model) was used with Flair and trained on CoNLL-03. Locations were identified using this model and could be filtered out from all other recognized entities by querying the LOC tag. The results were each added to a separate column in the previous data structure.