# Crawling Karelian News: Creating a Dataset to Preserve Cultural Heritage

Lev Kharlashkin

► To cite this version:

Lev Kharlashkin. Crawling Karelian News: Creating a Dataset to Preserve Cultural Heritage. Lightning Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, Mika Hämäläinen; Flammie Pirinen, Nov 2024, Helsinki, Finland. 10.5281/zenodo.14212594. hal-04808896

HAL Id: hal-04808896

https://hal.science/hal-04808896v1

Submitted on 28 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Crawling Karelian News:
Creating a Dataset to
Preserve Cultural Heritage

Lev Kharlashkin

Metropolia University of Applied Sciences, Finland

## Introduction

Karelian, a Finno-Ugric language with deep cultural roots faces the threat of extinction, as fewer than 30,000 people speak it, mainly in Russia and Finland. Younger generations often switch to dominant languages such as Russian or Finnish, which puts Karelian at risk of marginalization (Väisänen et al., 2019). To cope with this decline, I created a dataset of Karelian news articles from the National Media Portal of Karelia to promote the language's use, support research and advance language technology development (Lindgren & Kinnunen, 2021).
Motivation

The lack of digital resources for Karelian limits revitalization efforts and restricts access to contemporary content for learners and researchers. A digital collection of Karelian news is valuable for documenting current events, cultural narratives and language use. It provides insights into modern language patterns, making it a useful resource for computational linguistic applications like machine translation and natural language processing (Lewis et al., 2020).

The aim is to make Karelian content more visible and accessible digitally, allowing for continuous updates and expansion as new content emerges.
# Methodology

The dataset was created by crawling and extracting articles from Karelia's National Media Portal, which supports five languages. The news was published by three main Karelian-language publishers: OMA MUA, KARJALAN SANOMAT and Kipinä. Using web scraping techniques as proposed by Marisova (2022), the title, subtitle, publication date and main text of each article were collected. The data was then cleaned up by removing unnecessary links and annotations and standardized for consistency in formatting and character handling.

The dataset contains 38 articles with a total volume of 6980 words, with an average of 184 words per article. Despite its small size, this dataset is an important step towards creating more significant digital resources for the Karelian language.
# Insights from the Dataset

The analysis revealed interesting patterns in Karelian news publications. Of the 38 articles, 34 were published by OMA MUA, making it the most active Karelian-language publisher. Since August 2024, OMA MUA has increased its publication activity, signaling a growing commitment to Karelian-language content. This trend suggests a promising potential for expanding the dataset as future publications continue.

The study of the most commonly used words in the texts gives an idea of the nature of writing Karelian news. Words such as "ta" (124 occurrences) and "da" (117 occurrences) function as common conjunctions in the Karelian language, similar to "and" in English, reflecting the connecting structure of the language. Frequent mentions of "Karjalan" (69 cases) indicate that the dataset focuses on regional news and cultural identity. Words such as "projektan" (project), "kielen" (language) and "Šuojun" (place name) reveal the central themes of these articles, such as local projects, language preservation and cultural events.
Word analysis also revealed the conversational style of some news reports, as evidenced by the frequent use of the word “sanu” (said). This suggests that news articles often contain direct quotes or elements of spoken language, which gives them a more conversational tone.

## Future Potential and Applications

Despite its current limitations, the dataset provides opportunities for linguistic research and the development of language technologies. The data can be used to study syntax, trends in vocabulary and language changes over time. It can serve as a training tool for machine translation models between Karelian and other languages such as Finnish or Russian and aid in developing text-to-speech systems to make Karelian content accessible to broader audiences.
The Karelian National Media Portal multilingual support enables cross-linguistic studies, comparing Karelian with other languages. Content from various language versions of the portal can help researchers explore the relationships between Karelian, Finnish, Russian and other languages. Given the current activities of OMA MUA, the prospect of expanding the dataset in the near future looks promising. As new Karelian news is published, the dataset may increase, which makes it increasingly valuable for language preservation, computational linguistics and cultural studies.
## Conclusion

The creation of this Karelian news dataset is a crucial step in supporting the preservation and revitalization of the endangered language. By digitizing modern news content from the National Media Portal of Karelia, current events, cultural manifestations and the daily use of the Karelian language are documented. This digital resource not only lays the foundation for future computing applications, but also serves as the basis for ongoing efforts to expand the digital presence of the Karelian language. Moving forward, I plan to further expand the dataset by including more publishers and annotating data for specific linguistic tasks. With OMA MUA's ongoing activity, the likelihood of expanding the dataset appears strong, enhancing its value for language preservation, computational linguistics and cultural studies.

## References

* Väisänen, R., Saarinen, T., & Huhtinen, A. (2019).

* Language vitality and revitalization efforts: The case of Karelian. *Journal of Endangered Languages*, 8(3), 213–230.
Lindgren, K., & Kinnunen, T. (2021). Challenges in Finno-Ugric language preservation: The digital gap in Karelian resources. *Finno-Ugric Linguistics Studies, 6*(2), 45–60.

Lewis, J., Schuster, S., & Ho, M. (2020). Leveraging NLP for endangered languages: A systematic study on Karelian. *Proceedings of Computational Linguistics*, 2020, 1124-1130.

Marisova, E. (2022). Ethical considerations in the digital collection of cultural heritage. *Digital Humanities Quarterly*, 16(1), 54-68.