# On Hierarchical Text Language-Identification Algorithms
## 3 Proposed Method  
### LI Corpora  
We used the Leipzig Corpora Collection, Project Gutenberg, the European legislature datasets and discriminative similar language (DSL) Corpus Collections to create the LI corpora [PERSON] et al. (2014); [PERSON] et al. (2016); [PERSON] et al. (2016); [PERSON] et al. (2017); [PERSON] et al. (2017). Among them, the Leipzig Corpora Collection has the most languages. To evaluate our proposed LI algorithm, we compared it to other LI open-source projects, such as Polyglot Polyglot (1999) and langid.py Polyglot (1999). During evaluation, we used the same language corpora for both our algorithm and the open-source LI project. Polyglot can identify more than 196 languages, but we were unable to access all the language sources in Polyglot from currently available corpora sets. langid.py was pre-trained on 97 languages, all of which can be accessed in the Leipzig Corpora Collection. langid.py is superior to most other open-source LI tools in terms of short-text LI accuracy [36]. For these reasons, we selected the Leipzig Corpora Collection to create our LI corpora and selected langid.py as the baseline approach. Another reason for choosing the Leipzig Corpora Collection was that our research objective was to identify short text language. The Leipzig Corpora Collection contains randomly selected sentences in the language of the corpus. The sources were newspapers or text randomly collected from the web and split into sentences [37; 38]. In our experiment, we used sentences from the Leipzig Corpora Collection to train, test and evaluate the LI algorithm. The minimum length for most languages' sentences in our test was approximately 20 characters. We used ISO 639-2/T codes to annotate language [39]. In this coding scheme, each language is represented by a three-letter code. Node that \"Code\" in the tables is the language ISO code. The 97 languages initially supported by langid.py belong to 25 writing scripts. Among them, 20 scripts have only one member (Table 1). The other five writing scripts--Arabic, Cyrillic, Devanagari, Latin and Eastern Nagari--have many languages (Tables 2-5). According to language family knowledge, the Arabic-, Cyrillic- and Latin-script members in the corpora belong to different language groups (Tables 2-4). Some similar language groups also exist in the Latin-script datasets. In Table 2, the languages in parentheses represent the similar language group within the relevant language group. In our test, we selected the 10K-format dataset from the Leipzig Corpora Collection, in which each language has 10,000 sentences. Of the 97 languages used in langid.py, 93 can be directly downloaded from the Leipzig Corpora Collection's website. For the other four languages, we contacted the paper's [37] authors, who provided the remaining languages. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Script** & **Code** & **Script** & **Code** & **Script** & **Code** & **Script** & **Code** \\ \hline Armenian & hye & Japanese Braille & jpn & Ge\({}^{\circ}\)ez & amh & Georgia & kat \\ Greek & ell & Gujarati & guj & Gurmukh & pan & Hebrew & heb \\ Kannada & kan & Khmer & khm & Korean & kor & Tamil & tam \\ Lao & lao & Odia & ori & Sinhala & sin & Telugu & tel \\ Thai & tha & Simplified Chinese & zho & Tibetan & dzo & Tirhuta & mal \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Script datasets with only one member.