Developing a Fine-Grained Corpus for a Less-resourced Language: the case of Kurdish

Abstract: Kurdish is a less-resourced language consisting of different dialects written in various scripts. Approximately 30 million people in different countries speak the language. The lack of corpora is one of the main obstacles in Kurdish language processing. In this paper, we present KTC-the Kurdish Textbooks Corpus, which is composed of 31 K-12 textbooks in Sorani dialect. The corpus is normalized and categorized into 12 educational subjects containing 693,800 tokens (110,297 types). Our resource is publicly available for non-commercial use under the CC BY-NC-SA 4.0 license.

Kurdish is an Indo-European language mainly spoken in central and eastern Turkey, northern Iraq and Syria, and western Iran. It is a less-resourced language BIBREF0, in other words, a language for which general-purpose grammars and raw internet-based corpora are the main existing resources. The language is spoken in five main dialects, namely, Kurmanji (aka Northern Kurdish), Sorani (aka Central Kurdish), Southern Kurdish, Zazaki and Gorani BIBREF1.
Creating lexical databases and text corpora are essential tasks in natural language processing (NLP) development. Text corpora are knowledge repositories which provide semantic descriptions of words. The Kurdish language lacks diverse corpora in both raw and annotated forms BIBREF2, BIBREF3. According to the literature, there is no domain-specific corpus for Kurdish.
In this paper, we present KTC, a domain-specific corpus containing K-12 textbooks in Sorani. We consider a domain as a set of related concepts, and a domain-specific corpus as a collection of documents relevant to those concepts BIBREF4. Accordingly, we introduce KTC as a domain-specific corpus because it is based on the textbooks which have been written and compiled by a group of experts, appointed by the Ministry of Education (MoE) of the Kurdistan Region of Iraq, for educational purposes at the K-12 level. The textbooks are selected, written, compiled, and edited by experts in each subject and also by language editors based on a unified grammar and orthography. This corpus was initially collected as an accurate source for developing a Sorani Kurdish spellchecker for scientific writing. KTC contains a range of subjects, and its content is categorized according to those subjects. Given the accuracy of the text from scientific, grammatical, and orthographic points of view, we believe that it is also a fine-grained resource. The corpus will contribute to various NLP tasks in Kurdish, particularly in language modeling and grammatical error correction.
In the rest of this paper, Section SECREF2 reviews the related work, Section SECREF3 presents the corpus, Section SECREF4 addresses the challenges in the project and, Section SECREF5 concludes the paper.
Although the initiative to create a corpus for Kurdish dates back to 1998 BIBREF5, efforts in creating machine-readable corpora for Kurdish are recent. The first machine-readable corpus for Kurdish is the Leipzig Corpora Collection which is constructed using different sources on the Web BIBREF6. Later, Pewan BIBREF2 and Bianet BIBREF7 were developed as general-purpose corpora based on news articles. Kurdish corpora are also constructed for specific tasks such as dialectology BIBREF8, BIBREF3, machine transliteration BIBREF9, and part-of-speech (POS) annotation BIBREF10, BIBREF11. However, to the best of our knowledge, currently, there is no domain-specific corpus for Kurdish dialects.
KTC is composed of 31 educational textbooks published from 2011 to 2018 in various topics by the MoE. We received the material from the MoE partly in different versions of Microsoft Word and partly in Adobe InDesign formats. In the first step, we categorized each textbook based on the topics and chapters. As the original texts were not in Unicode, we converted the content to Unicode. This step was followed by a pre-processing stage where the texts were normalized by replacing zero-width-non-joiner (ZWNJ) BIBREF2 and manually verifying the orthography based on the reference orthography of the Kurdistan Region of Iraq. In the normalization process, we did not remove punctuation and special characters so that the corpus can be easily adapted our current task and also to future tasks where the integrity of the text may be required.
The students could choose to go to the Institutes instead of High Schools after the Secondary School. The Institutes focus on professional and technical education aiming at training technicians.
As an experiment, we present the top 15 most used tokens of the textbooks in KTC, which are illustrated in Figure FIGREF4. We observe that the most frequent tokens such as (ئابوورى > (economics), بازەرگانى > (business)) in economics, (=,$\times $ and وزەى > (energy)) in physics, and (خوداى > (god), گەورە > (great) and واتە > (meaning)) in theology are conjunctions, prepositions, pronouns or punctuation. These are not descriptive of any one subject, while each subject's top tokens are descriptive of its content. The plot in Figure FIGREF4 follows Zipf's law to some extent, wherein the frequency of a word is proportional to its rank BIBREF12. Here, not only the words but also the punctuation and special characters are also considered tokens (see Section SECREF1).
The corpus is available at https://github.com/KurdishBLARK/KTC.
width=7cm,compat=1.9
Previously, researchers have addressed the challenges in Kurdish corpora development BIBREF2, BIBREF13, BIBREF3. We highlight two main challenges we faced during the KTC development. First, most of the written Kurdish resources have not been digitized BIBREF14, or they are either not publicly available or are not fully convertible. Second, Kurdish text processing suffers from different orthographic issues BIBREF9 mainly due to the lack of standard orthography and the usage of non-Unicode keyboards. Therefore, we carried out a semi-automatic conversion, which made the process costly in terms of time and human assistance.
We presented KTC–the Kurdish Textbook Corpus, as the first domain-specific corpus for Sorani Kurdish. This corpus will pave the way for further developments in Kurdish language processing. We have mad the corpus available at https://github.com/KurdishBLARK/KTC for non-commercial use. We are currently working on a project on the Sorani spelling error detection and correction. As future work, we are aiming to develop a similar corpus for all Kurdish dialects, particularly Kurmanji.
We would like to appreciate the generous assistance of the Ministry of Education of the Kurdistan Region of Iraq, particularly the General Directorate of Curriculum and Printing, for providing us with the data for the KTC corpus. Our special gratitude goes to Ms. Namam Jalal Rasheed and Mr. Kawa Omer Muhammad for their assistance in making the required data available and resolving of the copyright issues.

Figure and table captions:
Table 1: Statistics of the corpus - In the Course Level column, (i) represents Institute2 .
Figure 1: Common tokens among textbook subjects.

Questions about this paper:
What are the 12 categories devised?

Answers:
Economics, Genocide, Geography, History, Human Rights, Kurdish, Kurdology, Philosophy, Physics, Theology, Sociology, Social Study