Rosetta Balcanica: Deriving a “Gold Standard” Neural Machine Translation (NMT) Parallel Dataset for Western Balkan Languages (2024)

Abstract: The Rosetta Balcanica is an ongoing effort to expand the resources for low­resource western Balkan languages. This effort focuses on dis­ covering and using accurately translated, offi­ cially mapped, and curated parallel language resources and their preparation and use as neu­ ral machine translation (NMT) datasets. Some of the guiding principles, practices, and meth­ ods employed by Rosetta Balcanica are gener­ alizable and could apply to other low­resource language resource expansion efforts. With this goal in mind, we present our rationale and ap­ proach to discovering and using meticulously translated and officially curated low­resource language resources and our use of these re­ sources to develop a parallel “gold standard” translation training resource. Secondly, we describe our specific methodology for NMT dataset development from these resources and its publication to a widely­used and accessi­ ble repository for natural language processing (Hugging Face Hub). Finally, we discuss the trade­offs and limitations of our current ap­ proach and the roadmap for future develop­ ment and expansion of the current Rosetta Bal­ canica language resource.

heading: 1 Introduction
text: Many underresourced languages are spoken by ethnic groups residing in regions affected by eco nomic, social, or other crises. These circumstances frequently necessitate the involvement of interna tional institutions focused on economic assistance,
1The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non exclusive, paid up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Depart ment of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doepublic accessplan).
the promotion of human rights, the advancement of democratic structures and processes, humanitar ian aid, peacekeeping, and regional economic and political stabilization. As part of these activities, these institutions issue reports and studies that sup port their missions and goals. The documents produced by international orga nizations operating in regions affected by these factors require precise translation to: a) preserve the original meaning of the master communique, and b) accurately convey that meaning across the languages of the regional groups. Such documents constitute a “golden standard” for any downstream crosslingual computational linguistics tasks, par ticularly machine translation training. For a data set to fit such a standard, it must map across par allel data sets at the phrase and paragraph levels with high accuracy, be produced by professional translators, and be issued by an authoritative pub lication body. Neural machine translation (NMT) (Bahdanau et al., 2014) is a stateoftheart approach to ma chine translation that uses an endtoend encoder decoder architecture with attention mechanisms (Vaswani et al., 2017), to translate source lan guage sentences into target language sentences. Attentionbased models are particularly well suited for translation tasks as they support the training of models that can maintain attention on complex relationships and dependencies between two related sequences. For example, attention based models excel at learning relationships be tween words and phrases in two languages, even when the languages do not have a onetoone map ping. Accurate and complete sentencetosentence alignment between the source and target languages in a training dataset is particularly important for ef fectively training neural machine translation mod els because it helps the NMT model form accu rate patterns of attention. Therefore, a “golden”
186
training set in lowresource languages is of great value for effective lowresource NMT model de velopment, as the training content is meticulously translated and aligned at the phrase, sentence, and paragraph levels. This effort focuses on the Western Balkan lan guages, a region in South Central Europe compris ing countries formed by the breakup of former Yu goslavia and Albania. The languages spoken in this region include dialects of Southern Slavic lan guages (Serbian, Croatian, Bosnian, Macedonian, Slovenian, etc.), Albanian, and others (Friedman, 2011). These languages and resources were se lected due to our familiarity with the region’s cul tural, historical, and ethnic circumstances, and our fluency in several of these language groups (in cluding Croatian, Bosnian, Serbian, and to some degree, Macedonian). Furthermore, given the significant similarity be tween Croatian, Serbian, and Bosnian languages (Ljubesic et al., 2007), some of these parallel re sources, with translations from English, Shqip, and Macedonian to either Serbian, Croatian, or Bosnian, can potentially be used to crossmap the translations between all three languages. This sig nificantly boosts these three language resources and their use for other multilingual tasks (Pour damghani and Knight, 2017). The development of highquality, meticulously translated parallel datasets for lowresource lan guages such as those in the Western Balkans is crucial for advancing neural machine translation. In this paper, we outline the methodology, chal lenges, and future directions for creating these datasets, which serve as foundational resources for enhancing multilingual and crosslingual compu tational linguistics.

heading: 2 Background
text: Many of the languages of the Western Balkans are spoken by small national groups or ethnic minorities and are considered lowresource lan guages (Tyers andAlperen, 2010;Mati et al., 2021; Kunchukuttan and Bhattacharyya, 2021). Alba nian, or Shqip, is spoken by 7.5 million people worldwide, with approximately 4.5 million speak ers in the region. Although an IndoEuropean lan guage, it is unique and is not closely related to any other IndoEuropean language. Macedonian (Masson and Davies, 2016), an eastsouth Slavic language, is spoken by approximately two million
people. Furthermore, the region has been severely af fected by a series of regional wars, which, in addi tion to causing significant population losses, have been followed by economic depopulation (Lukic et al., 2012), a trend that continues today (Lutz and Gailey, 2020). It is estimated that 4.4 million peo ple have emigrated from the region (World Bank Group and WIIW, 2018, p.42).

heading: 2.1 Related Work
text: A number of parallel language resources for the languages of the Western Balkans. Many of these resources are the results of volunteer or officially sanctioned projects where news sites and individ ual works of wellknown literature or other re sources were collected into a corpus. METASHARE is the web site that curates and maintains most of these of parallel lan guage resources for Western Balkan languages. MacedonianCroatian Parallel Corpus (mk hr_pcorp) (Cebović and Tadić, 2016) is a parallel corpus consisting of fictional synchronic prose texts with over 500,000 tokens in each language and 39,735 aligned sentences. SouthEast Euro pean Parallel Corpus (Tyers and Alperen, 2010) is a corpus of SouthEast European languages derived from The SouthEast European Times website. The website is a collection of regional news that was sponsored by the United States European Command, dedicated to coverage of Southeast Europe (it ended publication in March 2015). The SouthEast European Parallel Corpus includes, among others, resources in Albanian (41,741,782 tokens), Macedonian (37,623,521 tokens) and Croatian (34,968,453 tokens). A parallel corpus for the tourism domain focus ing on English and Croatian languages was cre ated by leveraging automatic data crawlers to col lect parallel data from the web (Toral et al., 2017). The dataset was used to train machine translation (MT) systems for the tourism domain, aiming to optimize translation tasks relevant to this specific field. Parallel Global Voices (PGV) (Prokopidis et al., 2016) is a parallel language dataset, derived from the Global Voices multilingual group of websites, where volunteers publish and translate news sto ries in more than 40 languages. Parallel Data, Tools and Interfaces in OPUS is a growing resource of freely accessible parallel cor pora. It also provides tools for processing paral
lel and monolingual data, as well as several in terfaces to search the data, making it a unique re source for various research activities (Tiedemann, 2012). Albanian from Taoteba, a collection of vol unteer contributed sentences and translations that has 2,532 Albanian, 77,988 Macedonian, 5,301 Croatian, 45,786 Serbian and 619 Bosnian sen tences. A recent survey of resources and methods for Serbian Language was presented in (Marovac et al., 2023). However, most of the available resources such as Common Crawl2 or Internet Archive3 that are commonly used by the research community to build parallel datasets (Banón et al., 2022; El Kishky et al., 2019) suffer from limitations. They often comprise language structures that are either inadequately translated or lack meticulous align ment across languages. In contrast, the resources utilized inRosetta Balcanica stand out for their rig orous curation process. They have been curated from texts translated by official sources and metic ulously mapped to multiple languages, ensuring highquality and accurate representations to serve as resources for a wide range of applications.

heading: 2.2 Data Source
text: The Organization for Security and Cooperation in Europe (OSCE) (Galbreath, 2007) is the world’s largest intergovernmental securityoriented orga nization. Its mission includes conflict prevention, arms control, the promotion of democratic val ues and processes, and the protection and pro motion of human rights, among other objectives. The OSCE has several missions in the West ern Balkans, specifically in Albania, Bosnia and Herzegovina, Kosovo, and Serbia. As part of its mission, the OSCE publishes and curates multi lingual official documents relevant to its activi ties in the region. Their website 4 hosts an exten sive resource library of reports and white papers, with a significant section dedicated to the Western Balkans. For its translators, the OSCE requires a uni versity degree in interpretation or a related field, with a requisite perfect command of the languages of interest and professional proficiency in English (preferably as a mother tongue). Pertinent to the region and languages of interest, the OSCE pub lishes reports, studies, and white papers on human
2https://commoncrawl.org/ 3https://archive.org/ 4https://www.osce.org/resources/documents
rights, democratic elections, hate crime statistics, and other topics within the OSCE’s scope and mis sion. Given the scarcity of language resources and the high quality of multilingual translations, we have included all these resources.

heading: 3 Approach
text: Our overall approach to the development of Rosetta Balcanica was to identify the spe cific, officially translated multilanguage resources amenable to raw language processing and then convert them into a general and neutrally formatted parallel language resource. From there, we chose to postformat this neutral resource into a specific format (e.g. Hugging Face Hub format), with the intention of making it accessible to a large group of NLP practitioners. We started the collection process by attempt ing to automate the search, retrieval, and prepro cessing of downloaded resources. We encountered a number of errors related to standard document formatting (page numbers, header/footer dividers) that made automation complex, uncertain and error prone, so we delayed the entire automation process in favor of manual collection and preprocessing to make initial progress in corpus development (first stage/release in a roadmap (see Figure 4). We in tend to develop and proceed with a more robust automation starting with the second stage.

heading: 3.1 Selection and Preparation
text: We selected OSCE resources by language, starting with Shqip, assuming that there will be at least one translation (English) for the Shqip resources. In most cases, we found parallel translations between English, Shqip and the combination of Macedo nian and Serbian, and in some instances, such as regional publications or annual OSCE reports, the documents were translated into multiple lan guages.

heading: 3.2 Parallel Dataset Creation
text: Following best practices for the development of parallel datasets, we create a folder for each docu ment and then convert each PDF into a plain text representation. In particular, all document folders reside in a language combination directory named corresponding to the language of the documents it contains. For example, if a directory contains fold ers for documents that involve the three languages English, Macedonian, and Shqip, our language
combination directory would be named english macedonianshqip. This folder structure allows us to add more documents to an already existing di rectory or to add new language combinations with new documents in the future. We applied minimal formatting to remove page numbers and other nonlinguistic features such as header artifacts. Then, wemade sure that language files are aligned with each other at the paragraph and sentence level. The resulting structure is a folder named after each document, with each file entry representing a translation of the same content as seen in figure 1.
This shows the structure for the content ex tracted from the document titled “Conclusions and Recommendations” from the 5th Annual South East Media Conference5. Table 1 shows some ran dom samples extracted from this document for the three languages.

heading: 3.3 Validation
text: Given that the sources covered under Rosetta Bal canica are officially translated materials, we only performed spot validation to ensure alignment be tween the texts. Specifically, we validated the translations by randomly sampling parallel sen tences and verifying their translations in Google Translate.
3.4 Hugging Face Hub Hugging Face is the largest online NLP commu nity engaged around the use and sharing of state oftheart models. A part of this community is a Hugging Face Hub where NLP researchers can up load, share, and access data sets and models. At the time of the writing of this manuscript, there are
5https://www.osce.org/ representative-on-freedom-of-media/384549
over 900 datasets and more than 25 metrics avail able. Data on Hugging Face hub follows aDataset convention. Datasets is a library to easily access and share data sets and evaluation metrics for Nat ural Language Processing (NLP), computer vision, and audio tasks. Once part of the hub, the data sets can be loaded into a Hugging Face pipeline with a single line of code and used to train neural machine translation models (NMT).
We use a script that automatically scrapes all the document folders within a language combi nation directory and then extracts and aggregates all the sentences in each language text file to cre ate a single text file corresponding to each lan guage. This script then parses these language files to create a Hugging Face compliant dataset. Specifically, we create a temporary folder, which contains languagepair folders for each language. For example, the language combination directory englishmacedonianshqip containing documents from the three languages would result in two lan guage pair folders enma and ensh. We followed the convention of using the first two letters of the language to name the folder. Each languagepair directory contains 4 files corresponding to training and testing files for each language in the language pair followingHugging Face’s machine translation dataset convention. These are then zipped up into
rosetta_balcanica.tar.gz which is saved at the root of the repository. All these files are currently hosted on Github at our repository6. The data sets are made accessible via the compressed zip file through the Hugging Face Hub by directly accessing the Github repos itory. Finally, we upload the dataset to Hugging Face Hub 7 for easy access.

heading: 4 Dataset Statistics
text: The first release of the Rosetta Balcanica focuses on the parallel documents in Shqip and Macedo nian. For this release, we used only Shqip re sources available through OSCE.

heading: 4.1 Corpus Statistics

heading: 4.2 Topics Represented
text: As expected, there is an obvious and deliberate bias in the topics presented in theOSCE corpus fig ure 3. These topics reflect the subjects of the pub lications and reports that are within the scope of OSCE’s mission in the Western Balkan countries 
6https://github.com/ebegoli/ rosetta-balcanica
7https://huggingface.co/datasets/ sudarshan85/rosetta_balcanica
human rights watch (and related violations), elec tions, conflicts and incidents, and overall socio cultural and economic development. These top ics are expected to dominate the corpus for the first two to three releases of the dataset while we are focusing on OSCE and Hague Tribunal origi nal sources. We plan to diversify the topics over time, but, overall, we are deliberately choosing a narrowly focused dataset that is of a highest qual ity, and the tradeoff of that approach will be lower topic diversity.

heading: 5 Challenges and Limitations
text: Maceodonian and Serbian entries are written in Cyrillic script. To compare the entries between Serbian, and, for example, Croatian and Bosnian, which are otherwise very similar languages, an ad ditional transliteration step is required. OSCE is a trusted dataset so we did not perform translitera tion of all the data. We checked 1% of randomly selected samples and found those accurately trans lated.

heading: 6 Future Work
text: The scope of the work presented in this encom passes the work on the OSCE dataset, which is topically narrow, and primarily performing man ual preprocessing to minimize “debugging” time. Future work will encompass a) inclusion of other, similarly structured parallel language datasets, and b) automation of dataset retrieval and pre processing. Once we process OSCE resources for all West ern languages, we will work to process Hague tri bunal documents which are, like OSCE, profes
sionally translated and available in most Western Balkan languages. After that, we will focus on the general multilingual resources from official me dia sources. We will focus on the latter mostly to diversify the topics and balance out representation.
Rosetta Balcanica is on an ongoing project, and the development of new resources continues. Al though it is an ongoing process, we have identified releases and milestones (Figure 4) in a roadmap that maps to the inclusion of specific language re sources and the dataset development tools (e.g. resource retrieval and preprocessing automation, quality assurance, dataset registration, etc.).
We mentioned earlier that we attempted to au tomate the dataset collection/retrieval and pre processing step. We found that the complexi ties and errorproneness of automation attempts, at that time, were slowing us down in the process of dataset development more than they were helping us. It is evident, though, that this process needs to be automated in order to be scalable and easily reusable, and it is on our roadmap to automate the process. In fact, we are in the process of improving the automation process, and we expect that the au tomation scripts will be available by the time this paper is published (mid2022).

heading: 7 Conclusion
text: There are three takeaways from the Rosetta Bal canica effort of that are, we presume, of interest to the natural language processing community:
1. Similar organizations, with a perhaps differ ent mission, are likely sources of the similar “golden set” materials that can be used for the development of similar parallel datasets for lowresource languages. We recommend exploring similar resources for other low resource languages. These could be UN ESCO, UNICEF, United Nations, and other international organizations.
2. The workflow presented in this paper is a practice that we recommend as well. The ap proach we have taken, where we curate a sin gle, raw corpus in parallel languages and then use it to create a training library or modality specific dataset (Hugging Face Hub) is an ap proach that makes the dataset readily avail able to a broad community that uses stateof theart NLP methods (neural machine trans lation, etc.). This approach also scales well because the original, raw source can be used for the development of other library and modalityspecific datasets.
3. While we found automated retrieval and preparation of data sources to be challenging and errorprone, we still intend to pursue this route in the future, and we encourage other similar efforts to attempt the same.

heading: 8 Acknowledgements
text: This manuscript has been in part coauthored by UTBattelle, LLC under Contract No. DEAC05
00OR22725 with the U.S. Department of Energy.

heading: Future Work
text: While we found automated retrieval and preparation of data sources to be challenging and errorprone, we still intend to pursue this route in the future, and we encourage other similar efforts to attempt the same