Automatic Language Identification in Texts: A Survey

Abstract: Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

Language identification (“”) is the task of determining the natural language that a document or part thereof is written in. Recognizing text in a specific language comes naturally to a human reader familiar with the language. intro:langid presents excerpts from Wikipedia articles in different languages on the topic of Natural Language Processing (“NLP”), labeled according to the language they are written in. Without referring to the labels, readers of this article will certainly have recognized at least one language in intro:langid, and many are likely to be able to identify all the languages therein.
Research into aims to mimic this human ability to recognize specific languages. Over the years, a number of computational approaches have been developed that, through the use of specially-designed algorithms and indexing structures, are able to infer the language being used without the need for human intervention. The capability of such systems could be described as super-human: an average person may be able to identify a handful of languages, and a trained linguist or translator may be familiar with many dozens, but most of us will have, at some point, encountered written texts in languages they cannot place. However, research aims to develop systems that are able to identify any human language, a set which numbers in the thousands BIBREF0 .
In a broad sense, applies to any modality of language, including speech, sign language, and handwritten text, and is relevant for all means of information storage that involve language, digital or otherwise. However, in this survey we limit the scope of our discussion to of written text stored in a digitally-encoded form.
Research to date on has traditionally focused on monolingual documents BIBREF1 (we discuss for multilingual documents in openissues:multilingual). In monolingual , the task is to assign each document a unique language label. Some work has reported near-perfect accuracy for of large documents in a small number of languages, prompting some researchers to label it a “solved task” BIBREF2 . However, in order to attain such accuracy, simplifying assumptions have to be made, such as the aforementioned monolinguality of each document, as well as assumptions about the type and quantity of data, and the number of languages considered.
The ability to accurately detect the language that a document is written in is an enabling technology that increases accessibility of data and has a wide variety of applications. For example, presenting information in a user's native language has been found to be a critical factor in attracting website visitors BIBREF3 . Text processing techniques developed in natural language processing and Information Retrieval (“IR”) generally presuppose that the language of the input text is known, and many techniques assume that all documents are in the same language. In order to apply text processing techniques to real-world data, automatic is used to ensure that only documents in relevant languages are subjected to further processing. In information storage and retrieval, it is common to index documents in a multilingual collection by the language that they are written in, and is necessary for document collections where the languages of documents are not known a-priori, such as for data crawled from the World Wide Web. Another application of that predates computational methods is the detection of the language of a document for routing to a suitable translator. This application has become even more prominent due to the advent of Machine Translation (“MT”) methods: in order for MT to be applied to translate a document to a target language, it is generally necessary to determine the source language of the document, and this is the task of . also plays a part in providing support for the documentation and use of low-resource languages. One area where is frequently used in this regard is in linguistic corpus creation, where is used to process targeted web crawls to collect text resources for low-resource languages.
A large part of the motivation for this article is the observation that lacks a “home discipline”, and as such, the literature is fragmented across a number of fields, including NLP, IR, machine learning, data mining, social medial analysis, computer science education, and systems science. This has hampered the field, in that there have been many instances of research being carried out with only partial knowledge of other work on the topic, and the myriad of published systems and datasets.
Finally, it should be noted that this survey does not make a distinction between languages, language varieties, and dialects. Whatever demarcation is made between languages, varieties and dialects, a system is trained to identify the associated document classes. Of course, the more similar two classes are, the more challenging it is for a system to discriminate between them. Training a system to discriminate between similar languages such as Croatian and Serbian BIBREF4 , language varieties like Brazilian and European Portuguese BIBREF5 , or a set of Arabic dialects BIBREF6 is more challenging than training systems to discriminate between, for example, Japanese and Finnish. Even so, as evidenced in this article, from a computational perspective, the algorithms and features used to discriminate between languages, language varieties, and dialects are identical.
is in some ways a special case of text categorization, and previous research has examined applying standard text categorization methods to BIBREF7 , BIBREF8 .
 BIBREF9 provides a definition of text categorization, which can be summarized as the task of mapping a document onto a pre-determined set of classes. This is a very broad definition, and indeed one that is applicable to a wide variety of tasks, amongst which falls modern-day . The archetypal text categorization task is perhaps the classification of newswire articles according to the topics that they discuss, exemplified by the Reuters-21578 dataset BIBREF10 . However, has particular characteristics that make it different from typical text categorization tasks:
These distinguishing characteristics present unique challenges and offer particular opportunities, so much so that research in has generally proceeded independently of text categorization research. In this survey, we will examine the common themes and ideas that underpin research in . We begin with a brief history of research that has led to modern (history), and then proceed to review the literature, first introducing the mathematical notation used in the article (notation), and then providing synthesis and analysis of existing research, focusing specifically on the representation of text (features) and the learning algorithms used (methods). We examine the methods for evaluating the quality of the systems (evaluation) as well as the areas where has been applied (applications), and then provide an overview of “off-the-shelf” systems (ots). We conclude the survey with a discussion of the open issues in (openissues), enumerating issues and existing efforts to address them, as well as charting the main directions where further research in is required.
Although there are some dedicated survey articles, these tend to be relatively short; there have not been any comprehensive surveys of research in automated LI of text to date. The largest survey so far can be found in the literature review of Marco Lui's PhD thesis BIBREF11 , which served as an early draft and starting point for the current article. BIBREF12 provides a historical overview of language identification focusing on the use of language models. BIBREF13 gives a brief overview of some of the methods used for , and BIBREF14 provide a review of some of the techniques and applications used previously. BIBREF15 gives a short overview of some of the challenges, algorithms and available tools for . BIBREF16 provides a brief summary of , how it relates to other research areas, and some outstanding challenges, but only does so in general terms and does not go into any detail about existing work in the area. Another brief article about is BIBREF17 , which covers both of spoken language as well as of written documents, and also discusses of documents stored as images rather than digitally-encoded text.
as a task predates computational methods – the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional program for text is by BIBREF18 , a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.
In the early 1970s, BIBREF19 considered the problem of automatic . According to BIBREF20 and the available abstract of Nakamura's article, his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual .
 BIBREF20 wrote his master's thesis “Language Identification by Statistical Analysis” for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use of text in military intelligence settings is evidenced by the recent articles of, for example, BIBREF21 , BIBREF22 , BIBREF23 , and BIBREF24 . As features for , BIBREF20 used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic ( INLINEFORM0 ), he managed to achieve 89% accuracy over 53 characters when distinguishing between English and Spanish. His thesis actually includes the identifier program code (for the IBM System/360 Model 67 mainframe), and even the language models in printed form.
Much of the earliest work on automatic was focused on identification of spoken language, or did not make a distinction between written and spoken language. For example, the work of BIBREF25 is primarily focused on of spoken utterances, but makes a broader contribution in demonstrating the feasibility of on the basis of a statistical model of broad phonetic information. However, their experiments do not use actual speech data, but rather “synthetic” data in the form of phonetic transcriptions derived from written text.
Another subfield of speech technology, speech synthesis, has also generated a considerable amount of research in the of text, starting from the 1980s. In speech synthesis, the need to know the source language of individual words is crucial in determining how they should be pronounced. BIBREF26 uses the relative frequencies of character trigrams as probabilities and determines the language of words using a Bayesian model. Church explains the method – that has since been widely used in LI – as a small part of an article concentrating on many aspects of letter stress assignment in speech synthesis, which is probably why BIBREF27 is usually attributed to being the one to have introduced the aforementioned method to of text. As Beesley's article concentrated solely on the problem of LI, this single focus probably enabled his research to have greater visibility. The role of the program implementing his method was to route documents to MT systems, and Beesley's paper more clearly describes what has later come to be known as a character model. The fact that the distribution of characters is relatively consistent for a given language was already well known.
The highest-cited early work on automatic is BIBREF7 . Cavnar and Trenkle's method (which we describe in detail in outofplace) builds up per-document and per-language profiles, and classifies a document according to which language profile it is most similar to, using a rank-order similarity metric. They evaluate their system on 3478 documents in eight languages obtained from USENET newsgroups, reporting a best overall accuracy of 99.8%. Gertjan van Noord produced an implementation of the method of Cavnar and Trenkle named , which has become eponymous with the method itself. is packaged with pre-trained models for a number of languages, and so it is likely that the strong results reported by Cavnar and Trenkle, combined with the ready availability of an “off-the-shelf” implementation, has resulted in the exceptional popularity of this particular method. BIBREF7 can be considered a milestone in automatic , as it popularized the use of automatic methods on character models for , and to date the method is still considered a benchmark for automatic .
This section introduces the notation used throughout this article to describe methods. We have translated the notation in the original papers to our notation, to make it easier to see the similarities and differences between the methods presented in the literature. The formulas presented could be used to implement language identifiers and re-evaluate the studies they were originally presented in.
A corpus INLINEFORM0 consists of individual tokens INLINEFORM1 which may be bytes, characters or words. INLINEFORM2 is comprised of a finite sequence of individual tokens, INLINEFORM3 . The total count of individual tokens INLINEFORM4 in INLINEFORM5 is denoted by INLINEFORM6 . In a corpus INLINEFORM7 with non-overlapping segments INLINEFORM8 , each segment is referred to as INLINEFORM9 , which may be a short document or a word or some other way of segmenting the corpus. The number of segments is denoted as INLINEFORM10 .
A feature INLINEFORM0 is some countable characteristic of the corpus INLINEFORM1 . When referring to the set of all features INLINEFORM2 in a corpus INLINEFORM3 , we use INLINEFORM4 , and the number of features is denoted by INLINEFORM5 . A set of unique features in a corpus INLINEFORM6 is denoted by INLINEFORM7 . The number of unique features is referred to as INLINEFORM8 . The count of a feature INLINEFORM9 in the corpus INLINEFORM10 is referred to as INLINEFORM11 . If a corpus is divided into segments INLINEFORM12 , the count of a feature INLINEFORM13 in INLINEFORM14 is defined as the sum of counts over the segments of the corpus, i.e. INLINEFORM15 . Note that the segmentation may affect the count of a feature in INLINEFORM16 as features do not cross segment borders.
A frequently-used feature is an , which consists of a contiguous sequence of INLINEFORM0 individual tokens. An starting at position INLINEFORM1 in a corpus segment is denoted INLINEFORM2 , where positions INLINEFORM3 remain within the same segment of the corpus as INLINEFORM4 . If INLINEFORM5 , INLINEFORM6 is an individual token. When referring to all of length INLINEFORM7 in a corpus INLINEFORM8 , we use INLINEFORM9 and the count of all such is denoted by INLINEFORM10 . The count of an INLINEFORM11 in a corpus segment INLINEFORM12 is referred to as INLINEFORM13 and is defined by count: DISPLAYFORM0 
The set of languages is INLINEFORM0 , and INLINEFORM1 denotes the number of languages. A corpus INLINEFORM2 in language INLINEFORM3 is denoted by INLINEFORM4 . A language model INLINEFORM5 based on INLINEFORM6 is denoted by INLINEFORM7 . The features given values by the model INLINEFORM8 are the domain INLINEFORM9 of the model. In a language model, a value INLINEFORM10 for the feature INLINEFORM11 is denoted by INLINEFORM12 . For each potential language INLINEFORM13 of a corpus INLINEFORM14 in an unknown language, a resulting score INLINEFORM15 is calculated. A corpus in an unknown language is also referred to as a test document.
The design of a supervised language identifier can generally be deconstructed into four key steps:
A representation of text is selected
A model for each language is derived from a training corpus of labelled documents
A function is defined that determines the similarity between a document and each language
The language of a document is predicted based on the highest-scoring model
The theoretical description of some of the methods leaves room for interpretation on how to implement them. BIBREF28 define an algorithm to be any well-defined computational procedure. BIBREF29 introduces a three-tiered classification where programs implement algorithms and algorithms implement functions. The examples of functions given by BIBREF29 , sort and find max differ from our identify language as they are always solvable and produce the same results. In this survey, we have considered two methods to be the same if they always produce exactly the same results from exactly the same inputs. This would not be in line with the definition of an algorithm by BIBREF29 , as in his example there are two different algorithms mergesort and quicksort that implement the function sort, always producing identical results with the same input. What we in this survey call a method, is actually a function in the tiers presented by BIBREF29 .
In this section, we present an extensive list of features used in , some of which are not self-evident. The equations written in the unified notation defined earlier show how the values INLINEFORM0 used in the language models are calculated from the tokens INLINEFORM1 . For each feature type, we generally introduce the first published article that used that feature type, as well as more recent articles where the feature type has been considered.
In , text is typically modeled as a stream of characters. However, there is a slight mismatch between this view and how text is actually stored: documents are digitized using a particular encoding, which is a mapping from characters (e.g. a character in an alphabet), onto the actual sequence of bytes that is stored and transmitted by computers. Encodings vary in how many bytes they use to represent each character. Some encodings use a fixed number of bytes for each character (e.g. ASCII), whereas others use a variable-length encoding (e.g. UTF-8). Some encodings are specific to a given language (e.g. GuoBiao 18030 or Big5 for Chinese), whereas others are specifically designed to represent as many languages as possible (e.g. the Unicode family of encodings). Languages can often be represented in a number of different encodings (e.g. UTF-8 and Shift-JIS for Japanese), and sometimes encodings are specifically designed to share certain codepoints (e.g. all single-byte UTF-8 codepoints are exactly the same as ASCII). Most troubling for , isomorphic encodings can be used to encode different languages, meaning that the determination of the encoding often doesn't help in honing in on the language. Infamous examples of this are the ISO-8859 and EUC encoding families. Encodings pose unique challenges for practical applications: a given language can often be encoded in different forms, and a given encoding can often map onto multiple languages.
Some research has included an explicit encoding detection step to resolve bytes to the characters they represent BIBREF30 , effectively transcoding the document into a standardized encoding before attempting to identify the language. However, transcoding is computationally expensive, and other research suggests that it may be possible to ignore encoding and build a single per-language model covering multiple encodings simultaneously BIBREF31 , BIBREF32 . Another solution is to treat each language-encoding pair as a separate category BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 . The disadvantage of this is that it increases the computational cost by modeling a larger number of classes. Most of the research has avoided issues of encoding entirely by assuming that all documents use the same encoding BIBREF37 . This may be a reasonable assumption in some settings, such as when processing data from a single source (e.g. all data from Twitter and Wikipedia is UTF-8 encoded). In practice, a disadvantage of this approach may be that some encodings are only applicable to certain languages (e.g. S-JIS for Japanese and Big5 for Chinese), so knowing that a document is in a particular encoding can provide information that would be lost if the document is transcoded to a universal encoding such as UTF-8. BIBREF38 used a parallel state machine to detect which encoding scheme a file could potentially have been encoded with. The knowledge of the encoding, if detected, is then used to narrow down the possible languages.
Most features and methods do not make a distinction between bytes or characters, and because of this we will present feature and method descriptions in terms of characters, even if byte tokenization was actually used in the original research.
In this section, we review how individual character tokens have been used as features in .
 BIBREF39 used the formatting of numbers when distinguishing between Malay and Indonesian. BIBREF40 used the presence of non-alphabetic characters between the current word and the words before and after as features. BIBREF41 used emoticons (or emojis) in Arabic dialect identification with Naive Bayes (“NB”; see product). Non-alphabetic characters have also been used by BIBREF42 , BIBREF43 , BIBREF44 , and BIBREF45 .
 BIBREF46 used knowledge of alphabets to exclude languages where a language-unique character in a test document did not appear. BIBREF47 used alphabets collected from dictionaries to check if a word might belong to a language. BIBREF48 used the Unicode database to get the possible languages of individual Unicode characters. Lately, the knowledge of relevant alphabets has been used for also by BIBREF49 and BIBREF44 .
Capitalization is mostly preserved when calculating character frequencies, but in contexts where it is possible to identify the orthography of a given document and where capitalization exists in the orthography, lowercasing can be used to reduce sparseness. In recent work, capitalization was used as a special feature by BIBREF42 , BIBREF43 , and BIBREF45 .
 BIBREF50 was the first to use the length of words in . BIBREF51 used the length of full person names comprising several words. Lately, the number of characters in words has been used for by BIBREF52 , BIBREF53 , BIBREF44 , and BIBREF45 . BIBREF52 also used the length of the two preceding words.
 BIBREF54 used character frequencies as feature vectors. In a feature vector, each feature INLINEFORM0 has its own integer value. The raw frequency – also called term frequency (TF) – is calculated for each language INLINEFORM1 as: DISPLAYFORM0 
 BIBREF20 was the first to use the probability of characters. He calculated the probabilities as relative frequencies, by dividing the frequency of a feature found in the corpus by the total count of features of the same type in the corpus. When the relative frequency of a feature INLINEFORM0 is used as a value, it is calculated for each language INLINEFORM1 as: DISPLAYFORM0 
 BIBREF55 calculated the relative frequencies of one character prefixes, and BIBREF56 did the same for one character suffixes.
 BIBREF57 calculated character frequency document frequency (“LFDF”) values. BIBREF58 compared their own Inverse Class Frequency (“ICF”) method with the Arithmetic Average Centroid (“AAC”) and the Class Feature Centroid (“CFC”) feature vector updating methods. In ICF a character appearing frequently only in some language gets more positive weight for that language. The values differ from Inverse Document Frequency (“IDF”, artemenko1), as they are calculated using also the frequencies of characters in other languages. Their ICF-based vectors generally performed better than those based on AAC or CFC. BIBREF59 explored using the relative frequencies of characters with similar discriminating weights. BIBREF58 also used Mutual Information (“MI”) and chi-square weighting schemes with characters.
 BIBREF32 compared the identification results of single characters with the use of character bigrams and trigrams when classifying over 67 languages. Both bigrams and trigrams generally performed better than unigrams. BIBREF60 also found that the identification results from identifiers using just characters are generally worse than those using character sequences.
In this section we consider the different combinations of characters used in the literature. Character mostly consist of all possible characters in a given encoding, but can also consist of only alphabetic or ideographic characters.
 BIBREF56 calculated the co-occurrence ratios of any two characters, as well as the ratio of consonant clusters of different sizes to the total number of consonants. BIBREF61 used the combination of every bigram and their counts in words. BIBREF53 used the proportions of question and exclamation marks to the total number of the end of sentence punctuation as features with several machine learning algorithms.
 BIBREF62 used FastText to generate character n-gram embeddings BIBREF63 . Neural network generated embeddings are explained in cooccurrencesofwords.
 BIBREF20 used the relative frequencies of vowels following vowels, consonants following vowels, vowels following consonants and consonants following consonants. BIBREF52 used vowel-consonant ratios as one of the features with Support Vector Machines (“SVMs”, supportvectormachines), Decision Trees (“DTs”, decisiontrees), and Conditional Random Fields (“CRFs”, openissues:short).
 BIBREF41 used the existence of word lengthening effects and repeated punctuation as features. BIBREF64 used the presence of characters repeating more than twice in a row as a feature with simple scoring (simple1). BIBREF65 used more complicated repetitions identified by regular expressions. BIBREF66 used letter and character bigram repetition with a CRF. BIBREF67 used the count of character sequences with three or more identical characters, using several machine learning algorithms.
Character are continuous sequences of characters of length INLINEFORM0 . They can be either consecutive or overlapping. Consecutive character bigrams created from the four character sequence door are do and or, whereas the overlapping bigrams are do, oo, and or. Overlapping are most often used in the literature. Overlapping produces a greater number and variety of from the same amount of text.
 BIBREF20 was the first to use combinations of any two characters. He calculated the relative frequency of each bigram. RFTable2 lists more recent articles where relative frequencies of of characters have been used. BIBREF20 also used the relative frequencies of two character combinations which had one unknown character between them, also known as gapped bigrams. BIBREF68 used a modified relative frequency of character unigrams and bigrams.
Character trigram frequencies relative to the word count were used by BIBREF92 , who calculated the values INLINEFORM0 as in vega1. Let INLINEFORM1 be the word-tokenized segmentation of the corpus INLINEFORM2 of character tokens, then: DISPLAYFORM0 
where INLINEFORM0 is the count of character trigrams INLINEFORM1 in INLINEFORM2 , and INLINEFORM3 is the total word count in the corpus. Later frequencies relative to the word count were used by BIBREF93 for character bigrams and trigrams.
 BIBREF25 divided characters into five phonetic groups and used a Markovian method to calculate the probability of each bigram consisting of these phonetic groups. In Markovian methods, the probability of a given character INLINEFORM0 is calculated relative to a fixed-size character context INLINEFORM1 in corpus INLINEFORM2 , as follows: DISPLAYFORM0 
where INLINEFORM0 is an prefix of INLINEFORM1 of length INLINEFORM2 . In this case, the probability INLINEFORM3 is the value INLINEFORM4 , where INLINEFORM5 , in the model INLINEFORM6 . BIBREF94 used 4-grams with recognition weights which were derived from Markovian probabilities. MarkovianTable lists some of the more recent articles where Markovian character have been used.
 BIBREF110 was the first author to propose a full-fledged probabilistic language identifier. He defines the probability of a trigram INLINEFORM0 being written in the language INLINEFORM1 to be: DISPLAYFORM0 
He considers the prior probabilities of each language INLINEFORM0 to be equal, which leads to: DISPLAYFORM0 
 BIBREF110 used the probabilities INLINEFORM0 as the values INLINEFORM1 in the language models.
 BIBREF111 used a list of the most frequent bigrams and trigrams with logarithmic weighting. BIBREF112 was the first to use direct frequencies of character as feature vectors. BIBREF113 used Principal Component Analysis (“PCA”) to select only the most discriminating bigrams in the feature vectors representing languages. BIBREF114 used the most frequent and discriminating byte unigrams, bigrams, and trigrams among their feature functions. They define the most discriminating features as those which have the most differing relative frequencies between the models of the different languages. BIBREF115 tested from two to five using frequencies as feature vectors, frequency ordered lists, relative frequencies, and Markovian probabilities. FrequencyVectorTable lists the more recent articles where the frequency of character have been used as features. In the method column, “RF” refers to Random Forest (cf. decisiontrees), “LR” to Logistic Regression (discriminantfunctions), “KRR” to Kernel Ridge Regression (vectors), “KDA” to Kernel Discriminant Analysis (vectors), and “NN” to Neural Networks (neuralnetworks).
 BIBREF47 used the last two and three characters of open class words. BIBREF34 used an unordered list of distinct trigrams with the simple scoring method (Simplescoring). BIBREF132 used Fisher's discriminant function to choose the 1000 most discriminating trigrams. BIBREF133 used unique 4-grams of characters with positive Decision Rules (Decisionrule). BIBREF134 used the frequencies of bi- and trigrams in words unique to a language. BIBREF135 used lists of the most frequent trigrams.
 BIBREF38 divided possible character bigrams into those that are commonly used in a language and to those that are not. They used the ratio of the commonly used bigrams to all observed bigrams to give a confidence score for each language. BIBREF136 used the difference between the ISO Latin-1 code values of two consecutive characters as well as two characters separated by another character, also known as gapped character bigrams.
 BIBREF137 used the IDF and the transition probability of trigrams. They calculated the IDF values INLINEFORM0 of trigrams INLINEFORM1 for each language INLINEFORM2 , as in artemenko1, where INLINEFORM3 is the number of trigrams INLINEFORM4 in the corpus of the language INLINEFORM5 and INLINEFORM6 is the number of languages in which the trigram INLINEFORM7 is found, where INLINEFORM8 is the language-segmented training corpus with each language as a single segment. DISPLAYFORM0 
 INLINEFORM0 is defined as: DISPLAYFORM0 
 BIBREF138 used from one to four, which were weighted with “TF-IDF” (Term Frequency–Inverse Document Frequency). TF-IDF was calculated as: DISPLAYFORM0 
TF-IDF weighting or close variants have been widely used for . BIBREF139 used “CF-IOF” (Class Frequency-Inverse Overall Frequency) weighted 3- and 4-grams.
 BIBREF140 used the logarithm of the ratio of the counts of character bigrams and trigrams in the English and Hindi dictionaries. BIBREF141 used a feature weighting scheme based on mutual information (“MI”). They also tried weighting schemes based on the “GSS” (Galavotti, Sebastiani, and Simi) and “NGL” (Ng, Goh, and Low) coefficients, but using the MI-based weighting scheme proved the best in their evaluations when they used the sum of values method (sumvalues1). BIBREF67 used punctuation trigrams, where the first character has to be a punctuation mark (but not the other two characters). BIBREF142 used consonant bi- and trigrams which were generated from words after the vowels had been removed.
The language models mentioned earlier consisted only of of the same size INLINEFORM0 . If from one to four were used, then there were four separate language models. BIBREF7 created ordered lists of the most frequent for each language. BIBREF143 used similar lists with symmetric cross-entropy. BIBREF144 used a Markovian method to calculate the probability of byte trigrams interpolated with byte unigrams. BIBREF145 created a language identifier based on character of different sizes over 281 languages, and obtained an identification accuracy of 62.8% for extremely short samples (5–9 characters). Their language identifier was used or evaluated by BIBREF146 , BIBREF147 , and BIBREF148 . BIBREF146 managed to improve the identification results by feeding the raw language distance calculations into an SVM.
DifferingNgramTable3 lists recent articles where character of differing sizes have been used. “LR” in the methods column refer to Logistic Regression (maxent), “LSTM RNN” to Long Short-Term Memory Recurrent Neural Networks (neuralnetworks), and “DAN” to Deep Averaging Networks (neuralnetworks). BIBREF30 used up to the four last characters of words and calculated their relative frequencies. BIBREF149 used frequencies of 2–7-grams, normalized relative to the total number of in all the language models as well as the current language model. BIBREF60 compared the use of different sizes of in differing combinations, and found that combining of differing sizes resulted in better identification scores. BIBREF150 , BIBREF151 , BIBREF152 used mixed length domain-independent language models of byte from one to three or four.
Mixed length language models were also generated by BIBREF36 and later by BIBREF153 , BIBREF101 , who used the most frequent and discriminating longer than two bytes, up to a maximum of 12 bytes, based on their weighted relative frequencies. INLINEFORM0 of the most frequent were extracted from training corpora for each language, and their relative frequencies were calculated. In the tests reported in BIBREF153 , INLINEFORM1 varied from 200 to 3,500 . Later BIBREF154 also evaluated different combinations of character as well as their combinations with words.
 BIBREF155 used mixed-order frequencies relative to the total number of in the language model. BIBREF61 used frequencies of from one to five and gapped 3- and 4-grams as features with an SVM. As an example, some gapped 4-grams from the word Sterneberg would be Senb, tree, enbr, and reeg. BIBREF156 used character as a backoff from Markovian word . BIBREF157 used the frequencies of word initial ranging from 3 to the length of the word minus 1. BIBREF158 used the most relevant selected using the absolute value of the Pearson correlation. BIBREF159 used only the first 10 characters from a longer word to generate the , while the rest were ignored. BIBREF160 used only those which had the highest TF-IDF scores. BIBREF43 used character weighted by means of the “BM25” (Best Match 25) weighting scheme. BIBREF161 used byte up to length 25.
 BIBREF61 used consonant sequences generated from words. BIBREF189 used the presence of vowel sequences as a feature with a NB classifier (see naivebayes) when distinguishing between English and transliterated Indian languages.
 BIBREF190 used a basic dictionary (basicdictionary) composed of the 400 most common character 4-grams.
 BIBREF46 and BIBREF110 used character combinations (of different sizes) that either existed in only one language or did not exist in one or more languages.
 BIBREF191 used the suffixes of lexical words derived from untagged corpora. BIBREF192 used prefixes and suffixes determined using linguistic knowledge of the Arabic language. BIBREF193 used suffixes and prefixes in rule-based . BIBREF134 used morphemes and morpheme trigrams (morphotactics) constructed by Creutz's algorithm BIBREF194 . BIBREF195 used prefixes and suffixes constructed by his own algorithm, which was later also used by BIBREF196 . BIBREF197 used morpheme lexicons in . BIBREF196 compared the use of morphological features with the use of variable sized character . When choosing between ten European languages, the morphological features obtained only 26.0% accuracy while the reached 82.7%. BIBREF198 lemmatized Malay words in order to get the base forms. BIBREF199 used a morphological analyzer of Arabic. BIBREF70 used morphological information from a part-of-speech (POS) tagger. BIBREF189 and BIBREF64 used manually selected suffixes as features. BIBREF200 created morphological grammars to distinguish between Croatian and Serbian. BIBREF201 used morphemes created by Morfessor, but they also used manually created morphological rules. BIBREF102 used a suffix module containing the most frequent suffixes. BIBREF202 and BIBREF159 used word suffixes as features with CRFs. BIBREF119 used an unsupervised method to learn morphological features from training data. The method collects candidate affixes from a dictionary built using the training data. If the remaining part of a word is found from the dictionary after removing a candidate affix, the candidate affix is considered to be a morpheme. BIBREF119 used 5% of the most frequent affixes in language identification. BIBREF183 used character classified into different types, which included prefixes and suffixes. PrefixSuffixTable lists some of the more recent articles where prefixes and suffixes collected from a training corpus has been used for .
 BIBREF206 used trigrams composed of syllables. BIBREF198 used Markovian syllable bigrams for between Malay and English. Later BIBREF207 also experimented with syllable uni- and trigrams. BIBREF114 used the most frequent as well as the most discriminating Indian script syllables, called aksharas. They used single aksharas, akshara bigrams, and akshara trigrams. Syllables would seem to be especially apt in situations where distinction needs to be made between two closely-related languages.
 BIBREF96 used the trigrams of non-syllable chunks that were based on MI. BIBREF198 experimented also with Markovian bigrams using both character and grapheme bigrams, but the syllable bigrams proved to work better. Graphemes in this case are the minimal units of the writing system, where a single character may be composed of several graphemes (e.g. in the case of the Hangul or Thai writing systems). Later, BIBREF207 also used grapheme uni- and trigrams. BIBREF207 achieved their best results combining word unigrams and syllable bigrams with a grapheme back-off. BIBREF208 used the MADAMIRA toolkit for D3 decliticization and then used D3-token 5-grams. D3 decliticization is a way to preprocess Arabic words presented by BIBREF209 .
Graphones are sequences of characters linked to sequences of corresponding phonemes. They are automatically deduced from a bilingual corpus which consists of words and their correct pronunciations using Joint Sequence Models (“JSM”). BIBREF210 used language tags instead of phonemes when generating the graphones and then used Markovian graphone from 1 to 8 in .
 BIBREF211 used the position of the current word in word-level . The position of words in sentences has also been used as a feature in code-switching detection by BIBREF52 . It had predictive power greater than the language label or length of the previous word.
 BIBREF18 used the characteristics of words as parts of discriminating functions. BIBREF212 used the string edit distance and overlap between the word to be identified and words in dictionaries. Similarly BIBREF140 used a modified edit distance, which considers the common spelling substitutions when Hindi is written using latin characters. BIBREF213 used the Minimum Edit Distance (“MED”).
Basic dictionaries are unordered lists of words belonging to a language. Basic dictionaries do not include information about word frequency, and are independent of the dictionaries of other languages. BIBREF110 used a dictionary for as a part of his speech synthesizer. Each word in a dictionary had only one possible “language”, or pronunciation category. More recently, a basic dictionary has been used for by BIBREF214 , BIBREF52 , and BIBREF90 .
Unique word dictionaries include only those words of the language, that do not belong to the other languages targeted by the language identifier. BIBREF215 used unique short words (from one to three characters) to differentiate between languages. Recently, a dictionary of unique words was used for by BIBREF116 , BIBREF216 , and BIBREF67 .
 BIBREF47 used exhaustive lists of function words collected from dictionaries. BIBREF217 used stop words – that is non-content or closed-class words – as a training corpus. Similarly, BIBREF218 used words from closed word classes, and BIBREF97 used lists of function words. BIBREF219 used a lexicon of Arabic words and phrases that convey modality. Common to these features is that they are determined based on linguistic knowledge.
 BIBREF220 used the most relevant words for each language. BIBREF221 used unique or nearly unique words. BIBREF80 used Information Gain Word-Patterns (“IG-WP”) to select the words with the highest information gain.
 BIBREF222 made an (unordered) list of the most common words for each language, as, more recently, did BIBREF223 , BIBREF83 , and BIBREF85 . BIBREF224 encoded the most common words to root forms with the Soundex algorithm.
 BIBREF225 collected the frequencies of words into feature vectors. BIBREF112 compared the use of character from 2 to 5 with the use of words. Using words resulted in better identification results than using character bigrams (test document sizes of 20, 50, 100 or 200 characters), but always worse than character 3-, 4- or 5-grams. However, the combined use of words and character 4-grams gave the best results of all tested combinations, obtaining 95.6% accuracy for 50 character sequences when choosing between 13 languages. BIBREF158 used TF-IDF scores of words to distinguish between language groups. Recently, the frequency of words has also been used for by BIBREF180 , BIBREF183 , BIBREF129 , and BIBREF142 .
 BIBREF226 and BIBREF227 were the first to use relative frequencies of words in . As did BIBREF112 for word frequencies, also BIBREF60 found that combining the use of character with the use of words provided the best results. His language identifier obtained 99.8% average recall for 50 character sequences for the 10 evaluated languages (choosing between the 13 languages known by the language identifier) when using character from 1 to 6 combined with words. BIBREF98 calculated the relative frequency of words over all the languages. BIBREF137 calculated the IDF of words, following the approach outlined in artemenko1. BIBREF177 calculated the Pointwise Mutual Information (“PMI”) for words and used it to group words to Chinese dialects or dialect groups. Recently, the relative frequency of words has also been used for by BIBREF184 , BIBREF148 and BIBREF91 
 BIBREF228 used the relative frequency of words with less than six characters. Recently, BIBREF83 also used short words, as did BIBREF45 .
 BIBREF229 used the relative frequency calculated from Google searches. Google was later also used by BIBREF96 and BIBREF230 .
 BIBREF231 created probability maps for words for German dialect identification between six dialects. In a word probability map, each predetermined geographic point has a probability for each word form. Probabilities were derived using a linguistic atlas and automatically-induced dialect lexicons.
 BIBREF232 used commercial spelling checkers, which utilized lexicons and morphological analyzers. The language identifier of BIBREF232 obtained 97.9% accuracy when classifying one-line texts between 11 official South African languages. BIBREF233 used the ALMORGEANA analyzer to check if the word had an analysis in modern standard Arabic. They also used sound change rules to use possible phonological variants with the analyzer. BIBREF234 used spellchecking and morphological analyzers to detect English words from Hindi–English mixed search queries. BIBREF235 used spelling checkers to distinguish between 15 languages, extending the work of BIBREF232 with dynamic model selection in order to gain better performance. BIBREF157 used a similarity count to find if mystery words were misspelled versions of words in a dictionary.
 BIBREF236 used an “LBG-VQ” (Linde, Buzo & Gray algorithm for Vector Quantization) approach to design a codebook for each language BIBREF237 . The codebook contained a predetermined number of codevectors. Each codeword represented the word it was generated from as well as zero or more words close to it in the vector space.
 BIBREF41 used the number of words in a sentence with NB. BIBREF53 and BIBREF45 used the sentence length calculated in both words and characters with several machine learning algorithms.
 BIBREF53 used the ratio to the total number of words of: once-occurring words, twice-occurring words, short words, long words, function words, adjectives and adverbs, personal pronouns, and question words. They also used the word-length distribution for words of 1–20 characters.
 BIBREF193 used at least the preceding and proceeding words with manual rules in word-level for text-to-speech synthesis. BIBREF238 used Markovian word with a Hidden Markov Model (“HMM”) tagger (othermethods). WordNgramTable lists more recent articles where word or similar constructs have been used. “PPM” in the methods column refers to Prediction by Partial Matching (smoothing), and “kNN” to INLINEFORM0 Nearest Neighbor classification (ensemble).
 BIBREF239 used word trigrams simultaneously with character 4-grams. He concluded that word-based models can be used to augment the results from character when they are not providing reliable identification results. WordCharacterNgramTable lists articles where both character and word have been used together. “CBOW” in the methods column refer to Continuous Bag of Words neural network (neuralnetworks), and “MIRA” to Margin Infused Relaxed Algorithm (supportvectormachines). BIBREF154 evaluated different combinations of word and character with SVMs. The best combination for language variety identification was using all the features simultaneously. BIBREF187 used normal and gapped word and character simultaneously.
 BIBREF240 uses word embeddings consisting of Positive Pointwise Mutual Information (“PPMI”) counts to represent each word type. Then they use Truncated Singular Value Decomposition (“TSVD”) to reduce the dimension of the word vectors to 100. BIBREF241 used INLINEFORM0 -means clustering when building dialectal Arabic corpora. BIBREF242 used features provided by Latent Semantic Analysis (“LSA”) with SVMs and NB.
 BIBREF243 present two models, the CBOW model and the continuous skip-gram model. The CBOW model can be used to generate a word given it's context and the skip-gram model can generate the context given a word. The projection matrix, which is the weight matrix between the input layer and the hidden layer, can be divided into vectors, one vector for each word in the vocabulary. These word-vectors are also referred to as word embeddings. The embeddings can be used as features in other tasks after the neural network has been trained. BIBREF244 , BIBREF245 , BIBREF80 , BIBREF246 , BIBREF247 , BIBREF248 , BIBREF62 , and BIBREF130 used word embeddings generated by the word2vec skip-gram model BIBREF243 as features in . BIBREF249 used word2vec word embeddings and INLINEFORM0 -means clustering. BIBREF250 , BIBREF251 , and BIBREF44 also used word embeddings created with word2vec.
 BIBREF167 trained both character and word embeddings using FastText text classification method BIBREF63 on the Discriminating between Similar Languages (“DSL”) 2016 shared task, where it reached low accuracy when compared with the other methods. BIBREF205 used FastText to train word vectors including subword information. Then he used these word vectors together with some additional word features to train a CRF-model which was used for codeswitching detection.
 BIBREF212 extracted features from the hidden layer of a Recurrent Neural Network (“RNN”) that had been trained to predict the next character in a string. They used the features with a SVM classifier.
 BIBREF229 evaluated methods for detecting foreign language inclusions and experimented with a Conditional Markov Model (“CMM”) tagger, which had performed well on Named Entity Recognition (“NER”). BIBREF229 was able to produce the best results by incorporating her own English inclusion classifier's decision as a feature for the tagger, and not using the taggers POS tags. BIBREF197 used syntactic parsers together with dictionaries and morpheme lexicons. BIBREF278 used composed of POS tags and function words. BIBREF173 used labels from a NER system, cluster prefixes, and Brown clusters BIBREF279 . BIBREF214 used POS tag from one to three and BIBREF43 from one to five, and BIBREF67 used POS tag trigrams with TF-IDF weighting. BIBREF203 , BIBREF42 , BIBREF53 , and BIBREF45 have also recently used POS tags. BIBREF80 used POS tags with emotion-labeled graphs in Spanish variety identification. In emotion-labeled graphs, each POS-tag was connected to one or more emotion nodes if a relationship between the original word and the emotion was found from the Spanish Emotion Lexicon. They also used POS-tags with IG-WP. BIBREF208 used the MADAMIRA tool for morphological analysis disambiguation. The polySVOX text analysis module described by BIBREF197 uses two-level rules and morpheme lexicons on sub-word level and separate definite clause grammars (DCGs) on word, sentence, and paragraph levels. The language of sub-word units, words, sentences, and paragraphs in multilingual documents is identified at the same time as performing syntactic analysis for the document. BIBREF280 converted sentences into POS-tag patterns using a word-POS dictionary for Malay. The POS-tag patterns were then used by a neural network to indicate whether the sentences were written in Malay or not. BIBREF281 used Jspell to detect differences in the grammar of Portuguese variants. BIBREF200 used a syntactic grammar to recognize verb-da-verb constructions, which are characteristic of the Serbian language. The syntactic grammar was used together with several morphological grammars to distinguish between Croatian and Serbian.
 BIBREF193 used the weighted scores of the words to the left and right of the word to be classified. BIBREF238 used language labels within an HMM. BIBREF282 used the language labels of other words in the same sentence to determine the language of the ambiguous word. The languages of the other words had been determined by the positive Decision Rules (Decisionrule), using dictionaries of unique words when possible. BIBREF213 , BIBREF71 used the language tags of the previous three words with an SVM. BIBREF283 used language labels of surrounding words with NB. BIBREF82 used the language probabilities of the previous word to determining weights for languages. BIBREF156 used unigram, bigram and trigram language label transition probabilities. BIBREF284 used the language labels for the two previous words as well as knowledge of whether code-switching had already been detected or not. BIBREF285 used the language label of the previous word to determine the language of an ambiguous word. BIBREF286 also used the language label of the previous word. BIBREF287 used the language identifications of 2–4 surrounding words for post-identification correction in word-level . BIBREF109 used language labels with a CRF. BIBREF52 used language labels of the current and two previous words in code-switching point prediction. Their predictive strength was lower than the count of code-switches, but better than the length or position of the word. All of the features were used together with NB, DT and SVM. BIBREF288 used language label bigrams with an HMM. BIBREF41 used the word-level language labels obtained with the approach of BIBREF289 on sentence-level dialect identification.
Feature smoothing is required in order to handle the cases where not all features INLINEFORM0 in a test document have been attested in the training corpora. Thus, it is used especially when the count of features is high, or when the amount of training data is low. Smoothing is usually handled as part of the method, and not pre-calculated into the language models. Most of the smoothing methods evaluated by BIBREF290 have been used in , and we follow the order of methods in that article.
In Laplace smoothing, an extra number of occurrences is added to every possible feature in the language model. BIBREF291 used Laplace's sample size correction (add-one smoothing) with the product of Markovian probabilities. BIBREF292 experimented with additive smoothing of 0.5, and noted that it was almost as good as Good-Turing smoothing. BIBREF290 calculate the values for each as: DISPLAYFORM0 
where INLINEFORM0 is the probability estimate of INLINEFORM1 in the model and INLINEFORM2 its frequency in the training corpus. INLINEFORM3 is the total number of of length INLINEFORM4 and INLINEFORM5 the number of distinct in the training corpus. INLINEFORM6 is the Lidstone smoothing parameter. When using Laplace smoothing, INLINEFORM7 is equal to 1 and with Lidstone smoothing, the INLINEFORM8 is usually set to a value between 0 and 1.
The penalty values used by BIBREF170 with the HeLI method function as a form of additive smoothing. BIBREF145 evaluated additive, Katz, absolute discounting, and Kneser-Ney smoothing methods. Additive smoothing produced the least accurate results of the four methods. BIBREF293 and BIBREF258 evaluated NB with several different Lidstone smoothing values. BIBREF107 used additive smoothing with character as a baseline classifier, which they were unable to beat with Convolutional Neural Networks (“CNNs”).
 BIBREF292 used Good-Turing smoothing with the product of Markovian probabilities. BIBREF290 define the Good-Turing smoothed count INLINEFORM0 as: DISPLAYFORM0 
where INLINEFORM0 is the number of features occurring exactly INLINEFORM1 times in the corpus INLINEFORM2 . Lately Good-Turing smoothing has been used by BIBREF294 and BIBREF88 .
 BIBREF220 used Jelinek-Mercer smoothing correction over the relative frequencies of words, calculated as follows: DISPLAYFORM0 
where INLINEFORM0 is a smoothing parameter, which is usually some small value like 0.1. BIBREF105 used character 1–8 grams with Jelinek-Mercer smoothing. Their language identifier using character 5-grams achieved 3rd place (out of 12) in the TweetLID shared task constrained track.
 BIBREF95 and BIBREF145 used the Katz back-off smoothing BIBREF295 from the SRILM toolkit, with perplexity. Katz smoothing is an extension of Good-Turing discounting. The probability mass left over from the discounted is then distributed over unseen via a smoothing factor. In the smoothing evaluations by BIBREF145 , Katz smoothing performed almost as well as absolute discounting, which produced the best results. BIBREF296 evaluated Witten-Bell, Katz, and absolute discounting smoothing methods. Witten-Bell got 87.7%, Katz 87.5%, and absolute discounting 87.4% accuracy with character 4-grams.
 BIBREF297 used the PPM-C algorithm for . PPM-C is basically a product of Markovian probabilities with an escape scheme. If an unseen context is encountered for the character being processed, the escape probability is used together with a lower-order model probability. In PPM-C, the escape probability is the sum of the seen contexts in the language model. PPM-C was lately used by BIBREF165 . The PPM-D+ algorithm was used by BIBREF298 . BIBREF299 and BIBREF300 used a PPM-A variant. BIBREF301 also used PPM. The language identifier of BIBREF301 obtained 91.4% accuracy when classifying 100 character texts between 277 languages. BIBREF302 used Witten-Bell smoothing with perplexity.
 BIBREF303 used a Chunk-Based Language Model (“CBLM”), which is similar to PPM models.
 BIBREF145 used several smoothing techniques with Markovian probabilities. Absolute discounting from the VariKN toolkit performed the best. BIBREF145 define the smoothing as follows: a constant INLINEFORM0 is subtracted from the counts INLINEFORM1 of all observed INLINEFORM2 and the held-out probability mass is distributed between the unseen in relation to the probabilities of lower order INLINEFORM3 , as follows: DISPLAYFORM0 
where INLINEFORM0 is a scaling factor that makes the conditional distribution sum to one. Absolute discounting with Markovian probabilities from the VariKN toolkit was later also used by BIBREF146 , BIBREF147 , and BIBREF148 .
The original Kneser-Ney smoothing is based on absolute discounting with an added back-off function to lower-order models BIBREF145 . BIBREF290 introduced a modified version of the Kneser-Ney smoothing using interpolation instead of back-off. BIBREF304 used the Markovian probabilities with Witten-Bell and modified Kneser-Ney smoothing. BIBREF88 , BIBREF166 , and BIBREF261 also recently used modified Kneser-Ney discounting. BIBREF119 used both original and modified Kneser-Ney smoothings. In the evaluations of BIBREF145 , Kneser-Ney smoothing fared better than additive, but somewhat worse than the Katz and absolute discounting smoothing. Lately BIBREF109 also used Kneser-Ney smoothing.
 BIBREF86 , BIBREF87 evaluated several smoothing techniques with character and word : Laplace/Lidstone, Witten-Bell, Good-Turing, and Kneser-Ney. In their evaluations, additive smoothing with 0.1 provided the best results. Good-Turing was not as good as additive smoothing, but better than Witten-Bell and Kneser-Ney smoothing. Witten-Bell proved to be clearly better than Kneser-Ney.
In recent years there has been a tendency towards attempting to combine several different types of features into one classifier or classifier ensemble. Many recent studies use readily available classifier implementations and simply report how well they worked with the feature set used in the context of their study. There are many methods presented in this article that are still not available as out of the box implementations, however. There are many studies which have not been re-evaluated at all, going as far back as BIBREF18 . Our hope is that this article will inspire new studies and many previously unseen ways of combining features and methods. In the following sections, the reviewed articles are grouped by the methods used for .
 BIBREF46 used a positive Decision Rules with unique characters and character , that is, if a unique character or character was found, the language was identified. The positive Decision Rule (unique features) for the test document INLINEFORM0 and the training corpus INLINEFORM1 can be formulated as follows: DISPLAYFORM0 
where INLINEFORM0 is the set of unique features in INLINEFORM1 , INLINEFORM2 is the corpus for language INLINEFORM3 , and INLINEFORM4 is a corpus of any other language INLINEFORM5 . Positive decision rules can also be used with non-unique features when the decisions are made in a certain order. For example, BIBREF52 presents the pseudo code for her dictionary lookup tool, where these kind of decisions are part of an if-then-else statement block. Her (manual) rule-based dictionary lookup tool works better for Dutch–English code-switching detection than the SVM, DT, or CRF methods she experiments with. The positive Decision Rule has also been used recently by BIBREF85 , BIBREF190 , BIBREF287 , BIBREF216 , BIBREF305 , BIBREF169 , and BIBREF214 .
In the negative Decision Rule, if a character or character combination that was found in INLINEFORM0 does not exist in a particular language, that language is omitted from further identification. The negative Decision Rule can be expressed as: DISPLAYFORM0 
where INLINEFORM0 is the corpus for language INLINEFORM1 . The negative Decision Rule was first used by BIBREF47 in .
 BIBREF118 evaluated the JRIP classifier from the Waikato Environment for Knowledge Analysis (“WEKA”). JRIP is an implementation of the propositional rule learner. It was found to be inferior to the SVM, NB and DT algorithms.
In isolation the desicion rules tend not to scale well to larger numbers of languages (or very short test documents), and are thus mostly used in combination with other methods or as a Decision Tree.
 BIBREF306 were the earliest users of Decision Trees (“DT”) in . They used DT based on characters and their context without any frequency information. In training the DT, each node is split into child nodes according to an information theoretic optimization criterion. For each node a feature is chosen, which maximizes the information gain at that node. The information gain is calculated for each feature and the feature with the highest gain is selected for the node. In the identification phase, the nodes are traversed until only one language is left (leaf node). Later, BIBREF196 , BIBREF307 , and BIBREF308 have been especially successful in using DTs.
Random Forest (RF) is an ensemble classifier generating many DTs. It has been succesfully used in by BIBREF140 , BIBREF201 , BIBREF309 , and BIBREF185 , BIBREF172 .
In simple scoring, each feature in the test document is checked against the language model for each language, and languages which contain that feature are given a point, as follows: DISPLAYFORM0 
where INLINEFORM0 is the INLINEFORM1 th feature found in the test document INLINEFORM2 . The language scoring the most points is the winner. Simple scoring is still a good alternative when facing an easy problem such as preliminary language group identification. It was recently used for this purpose by BIBREF246 with a basic dictionary. They achieved 99.8% accuracy when identifying between 6 language groups. BIBREF310 use a version of simple scoring as a distance measure, assigning a penalty value to features not found in a model. In this version, the language scoring the least amount of points is the winner. Their language identifier obtained 100% success rate with character 4-grams when classifying relatively large documents (from 1 to 3 kilobytes), between 10 languages. Simple scoring was also used lately by BIBREF166 , BIBREF311 , and BIBREF90 .
The sum of values can be expressed as: DISPLAYFORM0 
where INLINEFORM0 is the INLINEFORM1 th feature found in the test document INLINEFORM2 , and INLINEFORM3 is the value for the feature in the language model of the language INLINEFORM4 . The language with the highest score is the winner.
The simplest case of sumvalues1 is when the text to be identified contains only one feature. An example of this is BIBREF157 who used the frequencies of short words as values in word-level identification. For longer words, he summed up the frequencies of different-sized found in the word to be identified. BIBREF210 first calculated the language corresponding to each graphone. They then summed up the predicted languages, and the language scoring the highest was the winner. When a tie occurred, they used the product of the Markovian graphone . Their method managed to outperform SVMs in their tests.
 BIBREF46 used the average of all the relative frequencies of the in the text to be identified. BIBREF312 evaluated several variations of the LIGA algorithm introduced by BIBREF313 . BIBREF308 and BIBREF148 also used LIGA and logLIGA methods. The average or sum of relative frequencies was also used recently by BIBREF85 and BIBREF108 .
 BIBREF57 summed up LFDF values (see characters), obtaining 99.75% accuracy when classifying document sized texts between four languages using Arabic script. BIBREF110 calculates the score of the language for the test document INLINEFORM0 as the average of the probability estimates of the features, as follows: DISPLAYFORM0 
where INLINEFORM0 is the number of features in the test document INLINEFORM1 . BIBREF153 summed weighted relative frequencies of character , and normalized the score by dividing by the length (in characters) of the test document. Taking the average of the terms in the sums does not change the order of the scored languages, but it gives comparable results between different lengths of test documents.
 BIBREF92 , BIBREF314 summed up the feature weights and divided them by the number of words in the test document in order to set a threshold to detect unknown languages. Their language identifier obtained 89% precision and 94% recall when classifying documents between five languages. BIBREF192 used a weighting method combining alphabets, prefixes, suffixes and words. BIBREF233 summed up values from a word trigram ranking, basic dictionary and morphological analyzer lookup. BIBREF282 summed up language labels of the surrounding words to identify the language of the current word. BIBREF200 summed up points awarded by the presence of morphological and syntactic features. BIBREF102 used inverse rank positions as values. BIBREF158 computed the sum of keywords weighted with TF-IDF. BIBREF315 summed up the TF-IDF derived probabilities of words.
The product of values can be expressed as follows: DISPLAYFORM0 
where INLINEFORM0 is the INLINEFORM1 th feature found in test document INLINEFORM2 , and INLINEFORM3 is the value for the feature in the language model of language INLINEFORM4 . The language with the highest score is the winner. Some form of feature smoothing is usually required with the product of values method to avoid multiplying by zero.
 BIBREF26 was the first to use the product of relative frequencies and it has been widely used ever since; recent examples include BIBREF86 , BIBREF87 , BIBREF161 , and BIBREF148 . Some of the authors use a sum of log frequencies rather than a product of frequencies to avoid underflow issues over large numbers of features, but the two methods yield the same relative ordering, with the proviso that the maximum of multiplying numbers between 0 and 1 becomes the minimum of summing their negative logarithms, as can be inferred from: DISPLAYFORM0 
When (multinomial) NB is used in , each feature used has a probability to indicate each language. The probabilities of all features found in the test document are multiplied for each language, and the language with the highest probability is selected, as in productvalues1. Theoretically the features are assumed to be independent of each other, but in practice using features that are functionally dependent can improve classification accuracy BIBREF316 .
NB implementations have been widely used for , usually with a more varied set of features than simple character or word of the same type and length. The features are typically represented as feature vectors given to a NB classifier. BIBREF283 trained a NB classifier with language labels of surrounding words to help predict the language of ambiguous words first identified using an SVM. The language identifier used by BIBREF77 obtained 99.97% accuracy with 5-grams of characters when classifying sentence-sized texts between six language groups. BIBREF265 used a probabilistic model similar to NB. BIBREF252 used NB and naive Bayes EM, which uses the Expectation–Maximization (“EM”) algorithm in a semi-supervised setting to improve accuracy. BIBREF4 used Gaussian naive Bayes (“GNB”, i.e. NB with Gaussian estimation over continuous variables) from scikit-learn.
In contrast to NB, in Bayesian networks the features are not assumed to be independent of each other. The network learns the dependencies between features in a training phase. BIBREF315 used a Bayesian Net classifier in two-staged (group first) over the open track of the DSL 2015 shared task. BIBREF130 similarly evaluated Bayesian Nets, but found them to perform worse than the other 11 algorithms they tested.
 BIBREF25 used the product of the Markovian probabilities of character bigrams. The language identifier created by BIBREF153 , BIBREF101 , “whatlang”, obtains 99.2% classification accuracy with smoothing for 65 character test strings, when distinguishing between 1,100 languages. The product of Markovian probabilities has recently also been used by BIBREF109 and BIBREF260 .
 BIBREF170 use a word-based backoff method called HeLI. Here, each language is represented by several different language models, only one of which is used for each word found in the test document. The language models for each language are: a word-level language model, and one or more models based on character of order 1– INLINEFORM0 . When a word that is not included in the word-level model is encountered in a test document, the method backs off to using character of the size INLINEFORM1 . If there is not even a partial coverage here, the method backs off to lower order and continues backing off until at least a partial coverage is obtained (potentially all the way to character unigrams). The system of BIBREF170 implementing the HeLI method attained shared first place in the closed track of the DSL 2016 shared task BIBREF317 , and was the best method tested by BIBREF148 for test documents longer than 30 characters.
The well-known method of BIBREF7 uses overlapping character of varying sizes based on words. The language models are created by tokenizing the training texts for each language INLINEFORM0 into words, and then padding each word with spaces, one before and four after. Each padded word is then divided into overlapping character of sizes 1–5, and the counts of every unique are calculated over the training corpus. The are ordered by frequency and INLINEFORM1 of the most frequent , INLINEFORM2 , are used as the domain of the language model INLINEFORM3 for the language INLINEFORM4 . The rank of an INLINEFORM5 in language INLINEFORM6 is determined by the frequency in the training corpus INLINEFORM7 and denoted INLINEFORM8 .
During , the test document INLINEFORM0 is treated in a similar way and a corresponding model INLINEFORM1 of the K most frequent is created. Then a distance score is calculated between the model of the test document and each of the language models. The value INLINEFORM2 is calculated as the difference in ranks between INLINEFORM3 and INLINEFORM4 of the INLINEFORM5 in the domain INLINEFORM6 of the model of the test document. If an is not found in a language model, a special penalty value INLINEFORM7 is added to the total score of the language for each missing . The penalty value should be higher than the maximum possible distance between ranks. DISPLAYFORM0 
The score INLINEFORM0 for each language INLINEFORM1 is the sum of values, as in sumvalues1. The language with the lowest score INLINEFORM2 is selected as the identified language. The method is equivalent to Spearman's measure of disarray BIBREF318 . The out-of-place method has been widely used in literature as a baseline. In the evaluations of BIBREF148 for 285 languages, the out-of-place method achieved an F-score of 95% for 35-character test documents. It was the fourth best of the seven evaluated methods for test document lengths over 20 characters.
Local Rank Distance BIBREF319 is a measure of difference between two strings. LRD is calculated by adding together the distances identical units (for example character ) are from each other between the two strings. The distance is only calculated within a local window of predetermined length. BIBREF122 and BIBREF320 used LRD with a Radial Basis Function (“RBF”) kernel (see RBF). For learning they experimented with both Kernel Discriminant Analysis (“KDA”) and Kernel Ridge Regression (“KRR”). BIBREF248 also used KDA.
 BIBREF224 calculated the Levenshtein distance between the language models and each word in the mystery text. The similary score for each language was the inverse of the sum of the Levenshtein distances. Their language identifier obtained 97.7% precision when classifying texts from two to four words between five languages. Later BIBREF216 used Levenshtein distance for Algerian dialect identification and BIBREF305 for query word identification.
 BIBREF321 , BIBREF322 , BIBREF323 , and BIBREF324 calculated the difference between probabilities as in Equation EQREF109 . DISPLAYFORM0 
where INLINEFORM0 is the probability for the feature INLINEFORM1 in the mystery text and INLINEFORM2 the corresponding probability in the language model of the language INLINEFORM3 . The language with the lowest score INLINEFORM4 is selected as the most likely language for the mystery text. BIBREF239 , BIBREF262 used the log probability difference and the absolute log probability difference. The log probability difference proved slightly better, obtaining a precision of 94.31% using both character and word when classifying 100 character texts between 53 language-encoding pairs.
Depending on the algorithm, it can be easier to view language models as vectors of weights over the target features. In the following methods, each language is represented by one or more feature vectors. Methods where each feature type is represented by only one feature vector are also sometimes referred to as centroid-based BIBREF58 or nearest prototype methods. Distance measures are generally applied to all features included in the feature vectors.
 BIBREF31 calculated the squared Euclidean distance between feature vectors. The Squared Euclidean distance can be calculated as: DISPLAYFORM0 
 BIBREF93 used the simQ similarity measure, which is closely related to the Squared Euclidean distance.
 BIBREF155 investigated the of multilingual documents using a Stochastic Learning Weak Estimator (“SLWE”) method. In SLWE, the document is processed one word at a time and the language of each word is identified using a feature vector representing the current word as well as the words processed so far. This feature vector includes all possible units from the language models – in their case mixed-order character from one to four. The vector is updated using the SLWE updating scheme to increase the probabilities of units found in the current word. The probabilities of units that have been found in previous words, but not in the current one, are on the other hand decreased. After processing each word, the distance of the feature vector to the probability distribution of each language is calculated, and the best-matching language is chosen as the language of the current word. Their language identifier obtained 96.0% accuracy when classifying sentences with ten words between three languages. They used the Euclidean distance as the distance measure as follows: DISPLAYFORM0 
 BIBREF325 compared the use of Euclidean distance with their own similarity functions. BIBREF112 calculated the cosine angle between the feature vector of the test document and the feature vectors acting as language models. This is also called the cosine similarity and is calculated as follows: DISPLAYFORM0 
The method of BIBREF112 was evaluated by BIBREF326 in the context of over multilingual documents. The cosine similarity was used recently by BIBREF131 . One common trick with cosine similarity is to pre-normalise the feature vectors to unit length (e.g. BIBREF36 ), in which case the calculation takes the form of the simple dot product: DISPLAYFORM0 
 BIBREF60 used chi-squared distance, calculated as follows: DISPLAYFORM0 
 BIBREF85 compared Manhattan, Bhattacharyya, chi-squared, Canberra, Bray Curtis, histogram intersection, correlation distances, and out-of-place distances, and found the out-of-place method to be the most accurate.
 BIBREF239 , BIBREF262 used cross-entropy and symmetric cross-entropy. Cross-entropy is calculated as follows, where INLINEFORM0 and INLINEFORM1 are the probabilities of the feature INLINEFORM2 in the the test document INLINEFORM3 and the corpus INLINEFORM4 : DISPLAYFORM0 
Symmetric cross-entropy is calculated as: DISPLAYFORM0 
For cross-entropy, distribution INLINEFORM0 must be smoothed, and for symmetric cross-entropy, both probability distributions must be smoothed. Cross-entropy was used recently by BIBREF161 . BIBREF301 used a cross-entropy estimating method they call the Mean of Matching Statistics (“MMS”). In MMS every possible suffix of the mystery text INLINEFORM1 is compared to the language model of each language and the average of the lengths of the longest possible units in the language model matching the beginning of each suffix is calculated.
 BIBREF327 and BIBREF32 calculated the relative entropy between the language models and the test document, as follows: DISPLAYFORM0 
This method is also commonly referred to as Kullback-Leibler (“KL”) distance or skew divergence. BIBREF60 compared relative entropy with the product of the relative frequencies for different-sized character , and found that relative entropy was only competitive when used with character bigrams. The product of relative frequencies gained clearly higher recall with higher-order when compared with relative entropy.
 BIBREF239 , BIBREF262 also used the RE and MRE measures, which are based on relative entropy. The RE measure is calculated as follows: DISPLAYFORM0 
MRE is the symmetric version of the same measure. In the tests performed by BIBREF239 , BIBREF262 , the RE measure with character outperformed other tested methods obtaining 98.51% precision when classifying 100 character texts between 53 language-encoding pairs.
 BIBREF304 used a logistic regression (“LR”) model (also commonly referred to as “maximum entropy” within NLP), smoothed with a Gaussian prior. BIBREF328 defined LR for character-based features as follows: DISPLAYFORM0 
where INLINEFORM0 is a normalization factor and INLINEFORM1 is the word count in the word-tokenized test document. BIBREF158 used an LR classifier and found it to be considerably faster than an SVM, with comparable results. Their LR classifier ranked 6 out of 9 on the closed submission track of the DSL 2015 shared task. BIBREF199 used Adaptive Logistic Regression, which automatically optimizes parameters. In recent years LR has been widely used for .
 BIBREF95 was the first to use perplexity for , in the manner of a language model. He calculated the perplexity for the test document INLINEFORM0 as follows: DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 were the Katz smoothed relative frequencies of word n-grams INLINEFORM1 of the length INLINEFORM2 . BIBREF146 and BIBREF148 evaluated the best performing method used by BIBREF145 . Character n-gram based perplexity was the best method for extremely short texts in the evaluations of BIBREF148 , but for longer sequences the methods of BIBREF36 and BIBREF60 proved to be better. Lately, BIBREF182 also used perplexity.
 BIBREF20 used Yule's characteristic K and the Kolmogorov-Smirnov goodness of fit test to categorize languages. Kolmogorov-Smirnov proved to be the better of the two, obtaining 89% recall for 53 characters (one punch card) of text when choosing between two languages. In the goodness of fit test, the ranks of features in the models of the languages and the test document are compared. BIBREF329 experimented with Jiang and Conrath's (JC) distance BIBREF330 and Lin's similarity measure BIBREF331 , as well as the out-of-place method. They conclude that Lin's similarity measure was consistently the most accurate of the three. JC-distance measure was later evaluated by BIBREF239 , BIBREF262 , and was outperformed by the RE measure. BIBREF39 and BIBREF332 calculated special ratios from the number of trigrams in the language models when compared with the text to be identified. BIBREF333 , BIBREF334 , BIBREF335 used the quadratic discrimination score to create the feature vectors representing the languages and the test document. They then calculated the Mahalanobis distance between the languages and the test document. Their language identifier obtained 98.9% precision when classifying texts of four “screen lines” between 19 languages. BIBREF336 used odds ratio to identify the language of parts of words when identifying between two languages. Odds ratio for language INLINEFORM0 when compared with language INLINEFORM1 for morph INLINEFORM2 is calculated as in Equation EQREF127 . DISPLAYFORM0 
The differences between languages can be stored in discriminant functions. The functions are then used to map the test document into an INLINEFORM0 -dimensional space. The distance of the test document to the languages known by the language identifier is calculated, and the nearest language is selected (in the manner of a nearest prototype classifier).
 BIBREF114 used multiple linear regression to calculate discriminant functions for two-way for Indian languages. BIBREF337 compared linear regression, NB, and LR. The precision for the three methods was very similar, with linear regression coming second in terms of precision after LR.
Multiple discriminant analysis was used for by BIBREF18 . He used two functions, the first separated Finnish from English and Swedish, and the second separated English and Swedish from each other. He used Mahalanobis' INLINEFORM0 as a distance measure. BIBREF113 used Multivariate Analysis (“MVA”) with Principal Component Analysis (“PCA”) for dimensionality reduction and . BIBREF59 compared discriminant analysis with SVM and NN using characters as features, and concluded that the SVM was the best method.
 BIBREF40 experimented with the Winnow 2 algorithm BIBREF338 , but the method was outperformed by other methods they tested.
With support vector machines (“SVMs”), a binary classifier is learned by learning a separating hyperplane between the two classes of instances which maximizes the margin between them. The simplest way to extend the basic SVM model into a multiclass classifier is via a suite of one-vs-rest classifiers, where the classifier with the highest score determines the language of the test document. One feature of SVMs that has made them particularly popular is their compatibility with kernels, whereby the separating hyperplane can be calculated via a non-linear projection of the original instance space. In the following paragraphs, we list the different kernels that have been used with SVMs for .
For with SVMs, the predominant approach has been a simple linear kernel SVM model. The linear kernel model has a weight vector INLINEFORM0 and the classification of a feature vector INLINEFORM1 , representing the test document INLINEFORM2 , is calculated as follows: DISPLAYFORM0 
where INLINEFORM0 is a scalar bias term. If INLINEFORM1 is equal to or greater than zero, INLINEFORM2 is categorized as INLINEFORM3 .
The first to use a linear kernel SVM were BIBREF339 , and generally speaking, linear-kernel SVMs have been widely used for , with great success across a range of shared tasks.
 BIBREF100 were the first to apply polynomial kernel SVMs to . With a polynomial kernel INLINEFORM0 can be calculated as: DISPLAYFORM0 
where INLINEFORM0 is the polynomial degree, and a hyperparameter of the model.
Another popular kernel is the RBF function, also known as a Gaussian or squared exponential kernel. With an RBF kernel INLINEFORM0 is calculated as: DISPLAYFORM0 
where INLINEFORM0 is a hyperparameter. BIBREF321 were the first to use an RBF kernel SVM for .
With sigmoid kernel SVMs, also known as hyperbolic tangent SVMs, INLINEFORM0 can be calculated as: DISPLAYFORM0 
 BIBREF340 were the first to use a sigmoid kernel SVM for , followed by BIBREF341 , who found the SVM to perform better than NB, Classification And Regression Tree (“CART”), or the sum of relative frequencies.
Other kernels that have been used with SVMs for include exponential kernels BIBREF178 and rational kernels BIBREF342 . BIBREF31 were the first to use SVMs for , in the form of string kernels using Ukkonen's algorithm. They used same string kernels with Euclidean distance, which did not perform as well as SVM. BIBREF87 compared SVMs with linear and on-line passive–aggressive kernels for , and found passive–aggressive kernels to perform better, but both SVMs to be inferior to NB and Log-Likelihood Ratio (sum of log-probabilities). BIBREF339 experimented with the Sequential Minimal Optimization (“SMO”) algorithm, but found a simple linear kernel SVM to perform better. BIBREF118 achieved the best results using the SMO algorithm, whereas BIBREF123 found CRFs to work better than SMO. BIBREF178 found that SMO was better than linear, exponential and polynomial kernel SVMs for Arabic tweet gender and dialect prediction.
MultipleKernelSVMarticlesTable lists articles where SVMs with different kernels have been compared. BIBREF343 evaluated three different SVM approaches using datasets from different DSL shared tasks. SVM-based approaches were the top performing systems in the 2014 and 2015 shared tasks.
 BIBREF277 used SVMs with the Margin Infused Relaxed Algorithm, which is an incremental version of SVM training. In their evaluation, this method achieved better results than off-the-shelf .
 BIBREF344 was the first to use Neural Networks (“NN”) for , in the form of a simple BackPropagation Neural Network (“BPNN”) BIBREF345 with a single layer of hidden units, which is also called a multi-layer perceptron (“MLP”) model. She used words as the input features for the neural network. BIBREF346 and BIBREF347 succesfully applied MLP to .
 BIBREF348 , BIBREF349 and BIBREF350 used radial basis function (RBF) networks for . BIBREF351 were the first to use adaptive resonance learning (“ART”) neural networks for . BIBREF85 used Neural Text Categorizer (“NTC”: BIBREF352 ) as a baseline. NTC is an MLP-like NN using string vectors instead of number vectors.
 BIBREF111 were the first to use a RNN for . They concluded that RNNs are less accurate than the simple sum of logarithms of counts of character bi- or trigrams, possibly due to the relatively modestly-sized dataset they experimented with. BIBREF221 compared NNs with the out-of-place method (see sec. UID104 ). Their results show that the latter, used with bigrams and trigrams of characters, obtains clearly higher identification accuracy when dealing with test documents shorter than 400 characters.
RNNs were more successfully used later by BIBREF245 who also incorporated character n-gram features in to the network architecture. BIBREF223 were the first to use a Long Short-Term Memory (“LSTM”) for BIBREF353 , and BIBREF354 was the first to use Gated Recurrent Unit networks (“GRUs”), both of which are RNN variants. BIBREF354 used byte-level representations of sentences as input for the networks. Recently, BIBREF89 and BIBREF176 also used LSTMs. Later, GRUs were successfully used for by BIBREF355 and BIBREF356 . In addition to GRUs, BIBREF354 also experimented with deep residual networks (“ResNets”) at DSL 2016.
During 2016 and 2017, there was a spike in the use of convolutional neural networks (CNNs) for , most successfully by BIBREF302 and BIBREF357 . Recently, BIBREF358 combined a CNN with adversarial learning to better generalize to unseen domains, surpassing the results of BIBREF151 based on the same training regime as .
 BIBREF275 used CBOW NN, achieving better results over the development set of DSL 2017 than RNN-based neural networks. BIBREF62 used deep averaging networks (DANs) based on word embeddings in language variety identification.
 BIBREF45 used the decision table majority classifier algorithm from the WEKA toolkit in English variety detection. The bagging algorithm using DTs was the best method they tested (73.86% accuracy), followed closely by the decision table with 73.07% accuracy.
 BIBREF359 were the first to apply hidden Markov models (HMM) to . More recently HMMs have been used by BIBREF214 , BIBREF288 , and BIBREF261 . BIBREF360 generated aggregate Markov models, which resulted in the best results when distinguishing between six languages, obtaining 74% accuracy with text length of ten characters. BIBREF156 used an extended Markov Model (“eMM”), which is essentially a standard HMM with modified emission probabilities. Their eMM used manually optimized weights to combine four scores (products of relative frequencies) into one score. BIBREF361 used Markov logic networks BIBREF362 to predict the language used in interlinear glossed text examples contained in linguistic papers.
 BIBREF363 evaluated the use of unsupervised Fuzzy C Means algorithm (“FCM”) in language identification. The unsupervised algorithm was used on the training data to create document clusters. Each cluster was tagged with the language having the most documents in the cluster. Then in the identification phase, the mystery text was mapped to the closest cluster and identified with its language. A supervised centroid classifier based on cosine similarity obtained clearly better results in their experiments (93% vs. 77% accuracy).
 BIBREF119 and BIBREF67 evaluated the extreme gradient boosting (“XGBoost”) method BIBREF364 . BIBREF119 found that gradient boosting gave better results than RFs, while conversely, BIBREF67 found that LR gave better results than gradient boosting.
 BIBREF365 used compression methods for , whereby a single test document is added to the training text of each language in turn, and the language with the smallest difference (after compression) between the sizes of the original training text file and the combined training and test document files is selected as the prediction. This has obvious disadvantages in terms of real-time computational cost for prediction, but is closely related to language modeling approaches to (with the obvious difference that the language model doesn't need to be retrained multiply for each test document). In terms of compression methods, BIBREF366 experimented with Maximal Tree Machines (“MTMs”), and BIBREF367 used LZW-based compression.
Very popular in text categorization and topic modeling, BIBREF368 , BIBREF23 , and BIBREF24 used Latent Dirichlet Allocation (“LDA”: BIBREF369 ) based features in classifying tweets between Arabic dialects, English, and French. Each tweet was assigned with an LDA topic, which was used as one of the features of an LR classifier.
 BIBREF249 used a Gaussian Process classifier with an RBF kernel in an ensemble with an LR classifier. Their ensemble achieved only ninth place in the “PAN” (Plagiarism Analysis, Authorship Identification, and Near-Duplicate Detection workshop) Author Profiling language variety shared task BIBREF370 and did not reach the results of the baseline for the task.
 BIBREF181 , BIBREF188 used a Passive Aggressive classifier, which proved to be almost as good as the SVMs in their evaluations between five different machine learning algorithms from the same package.
Ensemble methods are meta-classification methods capable of combining several base classifiers into a combined model via a “meta-classifier” over the outputs of the base classifiers, either explicitly trained or some heuristic. It is a simple and effective approach that is used widely in machine learning to boost results beyond those of the individual base classifiers, and particularly effective when applied to large numbers of individually uncorrelated base classifiers.
 BIBREF20 used simple majority voting to combine classifiers using different features and methods. In majority voting, the language of the test document is identified if a majority ( INLINEFORM0 ) of the classifiers in the ensemble vote for the same language. In plurality voting, the language with most votes is chosen as in the simple scoring method (simple1). Some authors also refer to plurality voting as majority voting.
 BIBREF371 used majority voting in tweet . BIBREF210 used majority voting with JSM classifiers. BIBREF265 and BIBREF269 used majority voting between SVM classifiers trained with different features. BIBREF266 used majority voting to combine four classifiers: RF, random tree, SVM, and DT. BIBREF372 and BIBREF152 used majority voting between three off-the-shelf language identifiers. BIBREF104 used majority voting between perplexity-based and other classifiers. BIBREF141 used majority voting between three sum of relative frequencies-based classifiers where values were weighted with different weighting schemes. BIBREF270 , BIBREF125 , BIBREF171 , BIBREF185 , BIBREF172 , and BIBREF260 used plurality voting with SVMs.
 BIBREF182 used voting between several perplexity-based classifiers with different features at the 2017 DSL shared task. A voting ensemble gave better results on the closed track than a singular word-based perplexity classifier (0.9025 weighted F1-score over 0.9013), but worse results on the open track (0.9016 with ensemble and 0.9065 without).
In a highest probability ensemble, the winner is simply the language which is given the highest probability by any of the individual classifiers in the ensemble. BIBREF96 used Gaussian Mixture Models (“GMM”) to give probabilities to the outputs of classifiers using different features. BIBREF372 used higher confidence between two off-the-shelf language identifiers. BIBREF265 used GMM to transform SVM prediction scores into probabilities. BIBREF270 , BIBREF125 used highest confidence over a range of base SVMs. BIBREF125 used an ensemble composed of low-dimension hash-based classifiers. According to their experiments, hashing provided up to 86% dimensionality reduction without negatively affecting performance. Their probability-based ensemble obtained 89.2% accuracy, while the voting ensemble got 88.7%. BIBREF166 combined an SVM and a LR classifier.
A mean probability ensemble can be used to combine classifiers that produce probabilities (or other mutually comparable values) for languages. The average of values for each language over the classifier results is used to determine the winner and the results are equal to the sum of values method (sumvalues1). BIBREF270 evaluated several ensemble methods and found that the mean probability ensemble attained better results than plurality voting, median probability, product, highest confidence, or Borda count ensembles.
In a median probability ensemble, the medians over the probabilities given by the individual classifiers are calculated for each language. BIBREF270 and BIBREF171 used a median probability rule ensemble over SVM classifiers. Consistent with the results of BIBREF270 , BIBREF171 found that a mean ensemble was better than a median ensemble, attaining 68% accuracy vs. 67% for the median ensemble.
A product rule ensemble takes the probabilities for the base classifiers and calculates their product (or sum of the log probabilities), with the effect of penalising any language where there is a particularly low probability from any of the base classifiers. BIBREF210 used log probability voting with JSM classifiers. BIBREF210 observed a small increase in average accuracy using the product ensemble over a majority voting ensemble.
In a INLINEFORM0 -best ensemble, several models are created for each language INLINEFORM1 by partitioning the corpus INLINEFORM2 into separate samples. The score INLINEFORM3 is calculated for each model. For each language, plurality voting is then applied to the INLINEFORM4 models with the best scores to predict the language of the test document INLINEFORM5 . BIBREF349 evaluated INLINEFORM6 -best with INLINEFORM7 based on several similarity measures. BIBREF54 compared INLINEFORM8 and INLINEFORM9 and concluded that there was no major difference in accuracy when distinguishing between six languages (100 character test set). BIBREF373 experimented with INLINEFORM10 -best classifiers, but they gave clearly worse results than the other classifiers they evaluated. BIBREF212 used INLINEFORM11 -best in two phases, first selecting INLINEFORM12 closest neighbors with simple similarity, and then using INLINEFORM13 with a more advanced similarity ranking.
In bagging, independent samples of the training data are generated by random sampling with replacement, individual classifiers are trained over each such training data sample, and the final classification is determined by plurality voting. BIBREF67 evaluated the use of bagging with an LR classifier in PAN 2017 language variety identification shared task, however, bagging did not improve the accuracy in the 10-fold cross-validation experiments on the training set. BIBREF374 used bagging with word convolutional neural networks (“W-CNN”). BIBREF45 used bagging with DTs in English national variety detection and found DT-based bagging to be the best evaluated method when all 60 different features (a wide selection of formal, POS, lexicon-based, and data-based features) were used, attaining 73.86% accuracy. BIBREF45 continued the experiments using the ReliefF feature selection algorithm from the WEKA toolkit to select the most efficient features, and achieved 77.32% accuracy over the reduced feature set using a NB classifier.
 BIBREF130 evaluated the Rotation Forest meta classifier for DTs. The method randomly splits the used features into a pre-determined number of subsets and then uses PCA for each subset. It obtained 66.6% accuracy, attaining fifth place among the twelve methods evaluated.
The AdaBoost algorithm BIBREF375 examines the performance of the base classifiers on the evaluation set and iteratively boosts the significance of misclassified training instances, with a restart mechanism to avoid local minima. AdaBoost was the best of the five machine learning techniques evaluated by BIBREF53 , faring better than C4.5, NB, RF, and linear SVM. BIBREF130 used the LogitBoost variation of AdaBoost. It obtained 67.0% accuracy, attaining third place among the twelve methods evaluated.
In stacking, a higher level classifier is explicitly trained on the output of several base classifiers. BIBREF96 used AdaBoost.ECC and CART to combine classifiers using different features. More recently, BIBREF127 used LR to combine the results of five RNNs. As an ensemble they produced better results than NB and LR, which were better than the individual RNNs. Also in 2017, BIBREF185 , BIBREF172 used RF to combine several linear SVMs with different features. The system used by BIBREF172 ranked first in the German dialect identification shared task, and the system by BIBREF185 came second (71.65% accuracy) in the Arabic dialect identification shared task.
In the previous two sections, we have alluded to issues of evaluation in research to date. In this section, we examine the literature more closely, providing a broad overview of the evaluation metrics that have been used, as well as the experimental settings in which research has been evaluated.
The most common approach is to treat the task as a document-level classification problem. Given a set of evaluation documents, each having a known correct label from a closed set of labels (often referred to as the “gold-standard”), and a predicted label for each document from the same set, the document-level accuracy is the proportion of documents that are correctly labeled over the entire evaluation collection. This is the most frequently reported metric and conveys the same information as the error rate, which is simply the proportion of documents that are incorrectly labeled (i.e. INLINEFORM0 ).
Authors sometimes provide a per-language breakdown of results. There are two distinct ways in which results are generally summarized per-language: (1) precision, in which documents are grouped according to their predicted language; and (2) recall, in which documents are grouped according to what language they are actually written in. Earlier work has tended to only provide a breakdown based on the correct label (i.e. only reporting per-language recall). This gives us a sense of how likely a document in any given language is to be classified correctly, but does not give an indication of how likely a prediction for a given language is of being correct. Under the monolingual assumption (i.e. each document is written in exactly one language), this is not too much of a problem, as a false negative for one language must also be a false positive for another language, so precision and recall are closely linked. Nonetheless, authors have recently tended to explicitly provide both precision and recall for clarity. It is also common practice to report an F-score INLINEFORM0 , which is the harmonic mean of precision and recall. The F-score (also sometimes called F1-score or F-measure) was developed in IR to measure the effectiveness of retrieval with respect to a user who attaches different relative importance to precision and recall BIBREF376 . When used as an evaluation metric for classification tasks, it is common to place equal weight on precision and recall (hence “F1”-score, in reference to the INLINEFORM1 hyper-parameter, which equally weights precision and recall when INLINEFORM2 ).
In addition to evaluating performance for each individual language, authors have also sought to convey the relationship between classification errors and specific sets of languages. Errors in systems are generally not random; rather, certain sets of languages are much more likely to be confused. The typical method of conveying this information is through the use of a confusion matrix, a tabulation of the distribution of (predicted language, actual language) pairs.
Presenting full confusion matrices becomes problematic as the number of languages considered increases, and as a result has become relatively uncommon in work that covers a broader range of languages. Per-language results are also harder to interpret as the number of languages increases, and so it is common to present only collection-level summary statistics. There are two conventional methods for summarizing across a whole collection: (1) giving each document equal weight; and (2) giving each class (i.e. language) equal weight. (1) is referred to as a micro-average, and (2) as a macro-average. For under the monolingual assumption, micro-averaged precision and recall are the same, since each instance of a false positive for one language must also be a false negative for another language. In other words, micro-averaged precision and recall are both simply the collection-level accuracy. On the other hand, macro-averaged precision and recall give equal weight to each language. In datasets where the number of documents per language is the same, this again works out to being the collection-level average. However, research has frequently dealt with datasets where there is a substantial skew between classes. In such cases, the collection-level accuracy is strongly biased towards more heavily-represented languages. To address this issue, in work on skewed document collections, authors tend to report both the collection-level accuracy and the macro-averaged precision/recall/F-score, in order to give a more complete picture of the characteristics of the method being studied.
Whereas the notions of macro-averaged precision and recall are clearly defined, there are two possible methods to calculate the macro-averaged F-score. The first is to calculate it as the harmonic mean of the macro-averaged precision and recall, and the second is to calculate it as the arithmetic mean of the per-class F-score.
The comparability of published results is also limited by the variation in size and source of the data used for evaluation. In work to date, authors have used data from a variety of different sources to evaluate the performance of proposed solutions. Typically, data for a number of languages is collected from a single source, and the number of languages considered varies widely. Earlier work tended to focus on a smaller number of Western European languages. Later work has shifted focus to supporting larger numbers of languages simultaneously, with the work of BIBREF101 pushing the upper bound, reporting a language identifier that supports over 1300 languages. The increased size of the language set considered is partly due to the increased availability of language-labeled documents from novel sources such as Wikipedia and Twitter. This supplements existing data from translations of the Universal Declaration of Human Rights, bible translations, as well as parallel texts from MT datasets such as OPUS and SETimes, and European Government data such as JRC-Acquis. These factors have led to a shift away from proprietary datasets such as the ECI multilingual corpus that were commonly used in earlier research. As more languages are considered simultaneously, the accuracy of systems decreases. A particularly striking illustration of this is the evaluation results by BIBREF148 for the logLIGA method BIBREF312 . BIBREF312 report an accuracy of 99.8% over tweets (averaging 80 characters) in six European languages as opposed to the 97.9% from the original LIGA method. The LIGA and logLIGA implementations by BIBREF148 have comparable accuracy for six languages, but the accuracy for 285 languages (with 70 character test length) is only slightly over 60% for logLIGA and the original LIGA method is at almost 85%. Many evaluations are not directly comparable as the test sizes, language sets, and hyper-parameters differ. A particularly good example is the method of BIBREF7 . The original paper reports an accuracy of 99.8% over eight European languages (>300 bytes test size). BIBREF150 report an accuracy of 68.6% for the method over a dataset of 67 languages (500 byte test size), and BIBREF148 report an accuracy of over 90% for 285 languages (25 character test size).
Separate to the question of the number and variety of languages included are issues regarding the quantity of training data used. A number of studies have examined the relationship between accuracy and quantity of training data through the use of learning curves. The general finding is that accuracy increases with more training data, though there are some authors that report an optimal amount of training data, where adding more training data decreases accuracy thereafter BIBREF377 . Overall, it is not clear whether there is a universal quantity of data that is “enough” for any language, rather this amount appears to be affected by the particular set of languages as well as the domain of the data. The breakdown presented by BIBREF32 shows that with less than 100KB per language, there are some languages where classification accuracy is near perfect, whereas there are others where it is very poor.
Another aspect that is frequently reported on is how long a sample of text needs to be before its language can be correctly detected. Unsurprisingly, the general consensus is that longer samples are easier to classify correctly. There is a strong interest in classifying short segments of text, as certain applications naturally involve short text documents, such as of microblog messages or search engine queries. Another area where of texts as short as one word has been investigated is in the context of dealing with documents that contain text in more than one language, where word-level has been proposed as a possible solution (see openissues:multilingual). These outstanding challenges have led to research focused specifically on of shorter segments of text, which we discuss in more detail in openissues:short.
From a practical perspective, knowing the rate at which a system can process and classify documents is useful as it allows a practitioner to predict the time required to process a document collection given certain computational resources. However, so many factors influence the rate at which documents are processed that comparison of absolute values across publications is largely meaningless. Instead, it is more valuable to consider publications that compare multiple systems under controlled conditions (same computer hardware, same evaluation data, etc.). The most common observations are that classification times between different algorithms can differ by orders of magnitude, and that the fastest methods are not always the most accurate. Beyond that, the diversity of systems tested and the variety in the test data make it difficult to draw further conclusions about the relative speed of algorithms.
Where explicit feature selection is used, the number of features retained is a parameter of interest, as it affects both the memory requirements of the system and its classification rate. In general, a smaller feature set results in a faster and more lightweight identifier. Relatively few authors give specific details of the relationship between the number of features selected and accuracy. A potential reason for this is that the improvement in accuracy plateaus with increasing feature count, though the exact number of features required varies substantially with the method and the data used. At the lower end of the scale, BIBREF7 report that 300–400 features per language is sufficient. Conversely BIBREF148 found that, for the same method, the best results for the evaluation set were attained with 20,000 features per language.
As discussed in standardevaluation, the objective comparison of different methods for is difficult due to the variation in the data that different authors have used to evaluate methods. BIBREF32 emphasize this by demonstrating how the performance of a system can vary according to the data used for evaluation. This implies that comparisons of results reported by different authors may not be meaningful, as a strong result in one paper may not translate into a strong result on the dataset used in a different paper. In other areas of research, authors have proposed standardized corpora to allow for the objective comparison of different methods.
Some authors have released datasets to accompany their work, to allow for direct replication of their experiments and encourage comparison and standardization. datasets lists a number of datasets that have been released to accompany specific publications. In this list, we only include corpora that were prepared specifically for research, and that include the full text of documents. Corpora of language-labelled Twitter messages that only provide document identifiers are also available, but reproducing the full original corpus is always an issue as the original Twitter messages are deleted or otherwise made unavailable.
One challenge in standardizing datasets for is that the codes used to label languages are not fully standardized, and a large proportion of labeling systems only cover a minor portion of the languages used in the world today BIBREF381 . BIBREF382 discuss this problem in detail, listing different language code sets, as well as the internal structure exhibited by some of the code sets. Some standards consider certain groups of “languages” as varieties of a single macro-language, whereas others consider them to be discrete languages. An example of this is found in South Slavic languages, where some language code sets refer to Serbo-Croatian, whereas others make distinctions between Bosnian, Serbian and Croatian BIBREF98 . The unclear boundaries between such languages make it difficult to build a reference corpus of documents for each language, or to compare language-specific results across datasets.
Another challenge in standardizing datasets for is the great deal of variation that can exist between data in the same language. We examine this in greater detail in openissues:encoding, where we discuss how the same language can use a number of different orthographies, can be digitized using a number of different encodings, and may also exist in transliterated forms. The issue of variation within a language complicates the development of standardized datasets, due to challenges in determining which variants of a language should be included. Since we have seen that the performance of systems can vary per-domain BIBREF32 , that research is often motivated by target applications (see applications), and that domain-specific information can be used to improve accuracy (see openissues:domainspecific), it is often unsound to use a generic dataset to develop a language identifier for a particular domain.
A third challenge in standardizing datasets for is the cost of obtaining correctly-labeled data. Manual labeling of data is usually prohibitively expensive, as it requires access to native speakers of all languages that the dataset aims to include. Large quantities of raw text data are available from sources such as web crawls or Wikipedia, but this data is frequently mislabeled (e.g. most non-English Wikipedias still include some English-language documents). In constructing corpora from such resources, it is common to use some form of automatic , but this makes such corpora unsuitable for evaluation purposes as they are biased towards documents that can be correctly identified by automatic systems BIBREF152 . Future work in this area could investigate other means of ensuring correct gold-standard labels while minimizing the annotation cost.
Despite these challenges, standardized datasets are critical for replicable and comparable research in . Where a subset of data is used from a larger collection, researchers should include details of the specific subset, including any breakdown into training and test data, or partitions for cross-validation. Where data from a new source is used, justification should be given for its inclusion, as well as some means for other researchers to replicate experiments on the same dataset.
To address specific sub-problems in , a number of shared tasks have been organized on problems such as in multilingual documents BIBREF378 , code-switched data BIBREF383 , discriminating between closely related languages BIBREF384 , and dialect and language variety identification in various languages BIBREF385 , BIBREF386 , BIBREF370 , BIBREF387 . Shared tasks are important for because they provide datasets and standardized evaluation methods that serve as benchmarks for the community. We summarize all shared tasks organized to date in sharedtasks.
Generally, datasets for shared tasks have been made publicly available after the conclusion of the task, and are a good source of standardized evaluation data. However, the shared tasks to date have tended to target specific sub-problems in , and no general, broad-coverage datasets have been compiled. Widespread interest in over closely-related languages has resulted in a number of shared tasks that specifically tackle the issue. Some tasks have focused on varieties of a specific language. For example, the DEFT2010 shared task BIBREF385 examined varieties of French, requiring participants to classify French documents with respect to their geographical source, in addition to the decade in which they were published. Another example is the Arabic Dialect Identification (“ADI”) shared task at the VarDial workshop BIBREF126 , BIBREF386 , and the Arabic Multi-Genre Broadcast (“MGB”) Challenge BIBREF387 .
Two shared tasks focused on a narrow group of languages using Twitter data. The first was TweetLID, a shared task on of Twitter messages according to six languages in common use in Spain, namely: Spanish, Portuguese, Catalan, English, Galician, and Basque (in order of the number of documents in the dataset) BIBREF388 , BIBREF389 . The organizers provided almost 35,000 Twitter messages, and in addition to the six monolingual tags, supported four additional categories: undetermined, multilingual (i.e. the message contains more than one language, without requiring the system to specify the component languages), ambiguous (i.e. the message is ambiguous between two or more of the six target languages), and other (i.e. the message is in a language other than the six target languages). The second shared task was the PAN lab on authorship profiling 2017 BIBREF370 . The PAN lab on authorship profiling is held annually and historically has focused on age, gender, and personality traits prediction in social media. In 2017 the competition introduced the inclusion of language varieties and dialects of Arabic, English, Spanish, and Portuguese,
More ambitiously, the four editions of the Discriminating between Similar Languages (DSL) BIBREF384 , BIBREF6 , BIBREF317 , BIBREF386 shared tasks required participants to discriminate between a set of languages in several language groups, each consisting of highly-similar languages or national varieties of that language. The dataset, entitled DSL Corpus Collection (“DSLCC”) BIBREF77 , and the languages included are summarized in dslcc. Historically the best-performing systems BIBREF265 , BIBREF390 , BIBREF43 have approached the task via hierarchical classification, first predicting the language group, then the language within that group.
There are various reasons to investigate . Studies in approach the task from different perspectives, and with different motivations and application goals in mind. In this section, we briefly summarize what these motivations are, and how their specific needs differ.
The oldest motivation for automatic is perhaps in conjunction with translation BIBREF27 . Automatic is used as a pre-processing step to determine what translation model to apply to an input text, whether it be by routing to a specific human translator or by applying MT. Such a use case is still very common, and can be seen in the Google Chrome web browser, where an built-in module is used to offer MT services to the user when the detected language of the web page being visited differs from the user's language settings.
NLP components such as POS taggers and parsers tend to make a strong assumption that the input text is monolingual in a given language. Similarly to the translation case, can play an obvious role in routing documents written in different languages to NLP components tailored to those languages. More subtle is the case of documents with mixed multilingual content, the most commonly-occurring instance of which is foreign inclusion, where a document is predominantly in a single language (e.g. German or Japanese) but is interspersed with words and phrases (often technical terms) from a language such as English. For example, BIBREF391 found that around 6% of word tokens in German text sourced from the Internet are English inclusions. In the context of POS tagging, one strategy for dealing with inclusions is to have a dedicated POS for all foreign words, and force the POS tagger to perform both foreign inclusion detection and POS tag these words in the target language; this is the approach taken in the Penn POS tagset, for example BIBREF392 . An alternative strategy is to have an explicit foreign inclusion detection pre-processor, and some special handling of foreign inclusions. For example, in the context of German parsing, BIBREF391 used foreign inclusion predictions to restrict the set of (German) POS tags used to form a parse tree, and found that this approach substantially improved parser accuracy.
Another commonly-mentioned use case is for multilingual document storage and retrieval. A document retrieval system (such as, but not limited to, a web search engine) may be required to index documents in multiple languages. In such a setting, it is common to apply at two points: (1) to the documents being indexed; and (2) to the queries being executed on the collection. Simple keyword matching techniques can be problematic in text-based document retrieval, because the same word can be valid in multiple languages. A classic example of such words (known as “false friends”) includes gift, which in German means “poison”. Performing on both the document and the query helps to avoid confusion between such terms, by taking advantage of the context in which it appears in order to infer the language. This has resulted in specific work in of web pages, as well as search engine queries. BIBREF393 and BIBREF394 give overviews of shared tasks specifically concentrating on language labeling of individual search query words. Having said this, in many cases, the search query itself does a sufficiently good job of selecting documents in a particular language, and overt is often not performed in mixed multilingual search contexts.
Automatic has also been used to facilitate linguistic and other text-based research. BIBREF34 report that their motivation for developing a language identifier was “to find out how many web pages are written in a particular language”. Automatic has been used in constructing web-based corpora. The Crúbadán project BIBREF395 and the Finno-Ugric Languages and the Internet project BIBREF396 make use of automated techniques to gather linguistic resources for under-resourced languages. Similarly, the Online Database of INterlinear text (“ODIN”: BIBREF397 ) uses automated as one of the steps in collecting interlinear glossed text from the web for purposes of linguistic search and bootstrapping NLP tools.
One challenge in collecting linguistic resources from the web is that documents can be multilingual (i.e. contain text in more than one language). This is problematic for standard methods, which assume that a document is written in a single language, and has prompted research into segmenting text by language, as well as word-level , to enable extraction of linguistic resources from multilingual documents. A number of shared tasks discussed in detail in evaluation:sharedtasks included data from social media. Examples are the TweetLID shared task on tweet held at SEPLN 2014 BIBREF388 , BIBREF389 , the data sets used in the first and second shared tasks on in code-switched data which were partially taken from Twitter BIBREF383 , BIBREF398 , and the third edition of the DSL shared task which contained two out-of-domain test sets consisting of tweets BIBREF317 . The 5th edition of the PAN at CLEF author profiling task included language variety identification for tweets BIBREF370 . There has also been research on identifying the language of private messages between eBay users BIBREF399 , presumably as a filtering step prior to more in-depth data analysis.
An “off-the-shelf” language identifier is software that is distributed with pre-trained models for a number of languages, so that a user is not required to provide training data before using the system. Such a setup is highly attractive to many end-users of automatic whose main interest is in utilizing the output of a language identifier rather than implementing and developing the technique. To this end, a number of off-the-shelf language identifiers have been released over time. Many authors have evaluated these off-the-shelf identifiers, including a recent evaluation involving 13 language identifiers which was carried out by BIBREF400 . In this section, we provide a brief summary of open-source or otherwise free systems that are available, as well as the key characteristics of each system. We have also included dates of when the software has been last updated as of October 2018.
TextCat is the most well-known Perl implementation of the out-of-place method, it lists models for 76 languages in its off-the-shelf configuration; the program is not actively maintained. TextCat is not the only example of an off-the-shelf implementation of the out-of-place method: other implementations include libtextcat with 76 language models, JTCL with 15 languages, and mguesser with 104 models for different language-encoding pairs. The main issue addressed by later implementations is classification speed: TextCat is implemented in Perl and is not optimized for speed, whereas implementations such as libtextcat and mguesser have been specifically written to be fast and efficient. whatlang-rs uses an algorithm based on character trigrams and refers the user to the BIBREF7 article. It comes pre-trained with 83 languages.
is the language identifier embedded in the Google Chrome web browser. It uses a NB classifier, and script-specific classification strategies. assumes that all the input is in UTF-8, and assigns the responsibility of encoding detection and transcoding to the user. uses Unicode information to determine the script of the input. also implements a number of pre-processing heuristics to help boost performance on its target domain (web pages), such as stripping character sequences like .jpg. The standard implementation supports 83 languages, and an extended model is also available, that supports 160 languages.
is a Java library that implements a language identifier based on a NB classifier trained over character . The software comes with pre-trained models for 53 languages, using data from Wikipedia. makes use of a range of normalization heuristics to improve the performance on particular languages, including: (1) clustering of Chinese/Japanese/Korean characters to reduce sparseness; (2) removal of “language-independent” characters, and other text normalization; and (3) normalization of Arabic characters.
is a Python implementation of the method described by BIBREF150 , which exploits training data for the same language across multiple different sources of text to identify sequences of characters that are strongly predictive of a given language, regardless of the source of the text. This feature set is combined with a NB classifier, and is distributed with a pre-trained model for 97 languages prepared using data from 5 different text sources. BIBREF151 provide an empirical comparison of to , and and find that it compares favorably both in terms of accuracy and classification speed. There are also implementations of the classifier component (but not the training portion) of in Java, C, and JavaScript.
 BIBREF153 uses a vector-space model with per-feature weighting on character sequences. One particular feature of is that it uses discriminative training in selecting features, i.e. it specifically makes use of features that are strong evidence against a particular language, which is generally not captured by NB models. Another feature of is that it uses inter-string smoothing to exploit sentence-level locality in making language predictions, under the assumption that adjacent sentences are likely to be in the same language. BIBREF153 reports that this substantially improves the accuracy of the identifier. Another distinguishing feature of is that it comes pre-trained with data for 1400 languages, which is the highest number by a large margin of any off-the-shelf system.
whatthelang is a recent language identifier written in Python, which utilizes the FastText NN-based text classification algorithm. It supports 176 languages.
implements an off-the-shelf classifier trained using Wikipedia data, covering 122 languages. Although not described as such, the actual classification algorithm used is a linear model, and is thus closely related to both NB and a cosine-based vector space model.
In addition to the above-mentioned general-purpose language identifiers, there have also been efforts to produce pre-trained language identifiers targeted specifically at Twitter messages. is a Twitter-specific tool with built-in models for 19 languages. It uses a document representation based on tries BIBREF401 . The algorithm is a LR classifier using all possible substrings of the data, which is important to maximize the available information from the relatively short Twitter messages.
 BIBREF152 provides a comparison of 8 off-the-shelf language identifiers applied without re-training to Twitter messages. One issue they report is that comparing the accuracy of off-the-shelf systems is difficult because of the different subset of languages supported by each system, which may also not fully cover the languages present in the target data. The authors choose to compare accuracy over the full set of languages, arguing that this best reflects the likely use-case of applying an off-the-shelf system to new data. They find that the best individual systems are , and , but that slightly higher accuracy can be attained by a simple voting-based ensemble classifier involving these three systems.
In addition to this, commercial or other closed-source language identifiers and language identifier services exist, of which we name a few. The Polyglot 3000 and Lextek Language Identifier are standalone language identifiers for Windows. Open Xerox Language Identifier is a web service with available REST and SOAP APIs.
Several papers have catalogued open issues in BIBREF327 , BIBREF382 , BIBREF1 , BIBREF334 , BIBREF32 , BIBREF324 , BIBREF317 . Some of the issues, such as text representation (features) and choice of algorithm (methods), have already been covered in detail in this survey. In this section, we synthesize the remaining issues into a single section, and also add new issues that have not been discussed in previous work. For each issue, we review related work and suggest promising directions for future work.
Text preprocessing (also known as normalization) is an umbrella term for techniques where an automatic transformation is applied to text before it is presented to a classifier. The aim of such a process is to eliminate sources of variation that are expected to be confounding factors with respect to the target task. Text preprocessing is slightly different from data cleaning, as data cleaning is a transformation applied only to training data, whereas normalization is applied to both training and test data. BIBREF1 raise text preprocessing as an outstanding issue in , arguing that its effects on the task have not been sufficiently investigated. In this section, we summarize the normalization strategies that have been proposed in the literature.
Case folding is the elimination of capitalization, replacing characters in a text with either their lower-case or upper-case forms. Basic approaches generally map between [a-z] and [A-Z] in the ASCII encoding, but this approach is insufficient for extended Latin encodings, where diacritics must also be appropriately handled. A resource that makes this possible is the Unicode Character Database (UCD) which defines uppercase, lowercase and titlecase properties for each character, enabling automatic case folding for documents in a Unicode encoding such as UTF-8.
Range compression is the grouping of a range of characters into a single logical set for counting purposes, and is a technique that is commonly used to deal with the sparsity that results from character sets for ideographic languages, such as Chinese, that may have thousands of unique “characters”, each of which is observed with relatively low frequency. BIBREF402 use such a technique where all characters in a given range are mapped into a single “bucket”, and the frequency of items in each bucket is used as a feature to represent the document. Byte-level representations of encodings that use multi-byte sequences to represent codepoints achieve a similar effect by “splitting” codepoints. In encodings such as UTF-8, the codepoints used by a single language are usually grouped together in “code planes”, where each codepoint in a given code plane shares the same upper byte. Thus, even though the distribution over codepoints may be quite sparse, when the byte-level representation uses byte sequences that are shorter than the multi-byte sequence of a codepoint, the shared upper byte will be predictive of specific languages.
Cleaning may also be applied, where heuristic rules are used to remove some data that is perceived to hinder the accuracy of the language identifier. For example, BIBREF34 identify HTML entities as a candidate for removal in document cleaning, on the basis that classifiers trained on data which does not include such entities may drop in accuracy when applied to raw HTML documents. includes heuristics such as expanding HTML entities, deleting digits and punctuation, and removing SGML-like tags. Similarly, also removes “language-independent characters” such as numbers, symbols, URLs, and email addresses. It also removes words that are all-capitals and tries to remove other acronyms and proper names using heuristics.
In the domain of Twitter messages, BIBREF313 remove links, usernames, smilies, and hashtags (a Twitter-specific “tagging” feature), arguing that these entities are language independent and thus should not feature in the model. BIBREF136 address of web pages, and report removing HTML formatting, and applying stopping using a small stopword list. BIBREF59 carry out experiments on the ECI multilingual corpus and report removing punctuation, space characters, and digits.
The idea of preprocessing text to eliminate domain-specific “noise” is closely related to the idea of learning domain-independent characteristics of a language BIBREF150 . One difference is that normalization is normally heuristic-driven, where a manually-specified set of rules is used to eliminate unwanted elements of the text, whereas domain-independent text representations are data-driven, where text from different sources is used to identify the characteristics that a language shares between different sources. Both approaches share conceptual similarities with problems such as content extraction for web pages. In essence, the aim is to isolate the components of the text that actually represent language, and suppress the components that carry other information. One application is the language-aware extraction of text strings embedded in binary files, which has been shown to perform better than conventional heuristic approaches BIBREF36 . Future work in this area could focus specifically on the application of language-aware techniques to content extraction, using models of language to segment documents into textual and non-textual components. Such methods could also be used to iteratively improve itself by improving the quality of training data.
is further complicated when we consider that some languages can be written in different orthographies (e.g. Bosnian and Serbian can be written in both Latin and Cyrillic script). Transliteration is another phenomenon that has a similar effect, whereby phonetic transcriptions in another script are produced for particular languages. These transcriptions can either be standardized and officially sanctioned, such as the use of Hanyu Pinyin for Chinese, or may also emerge irregularly and organically as in the case of arabizi for Arabic BIBREF403 . BIBREF1 identify variation in the encodings and scripts used by a given language as an open issue in , pointing out that early work tended to focus on languages written using a romanized script, and suggesting that dealing with issues of encoding and orthography adds substantial complexity to the task. BIBREF34 discuss the relative difficulties of discriminating between languages that vary in any combination of encoding, script and language family, and give examples of pairs of languages that fall into each category.
across orthographies and transliteration is an area that has not received much attention in work to date, but presents unique and interesting challenges that are suitable targets for future research. An interesting and unexplored question is whether it is possible to detect that documents in different encodings or scripts are written in the same language, or what language a text is transliterated from, without any a-priori knowledge of the encoding or scripts used. One possible approach to this could be to take advantage of standard orderings of alphabets in a language – the pattern of differences between adjacent characters should be consistent across encodings, though whether this is characteristic of any given language requires exploration.
 BIBREF1 paint a fairly bleak picture of the support for low-resource languages in automatic . This is supported by the arguments of BIBREF382 who detail specific issues in building hugely multilingual datasets. BIBREF404 also specifically called for research into automatic for low-density languages. Ethnologue BIBREF0 lists a total of 7099 languages. BIBREF382 describe the Ethnologue in more detail, and discuss the role that plays in other aspects of supporting minority languages, including detecting and cataloging resources. The problem is circular: methods are typically supervised, and need training data for each language to be covered, but the most efficient way to recover such data is through methods.
A number of projects are ongoing with the specific aim of gathering linguistic data from the web, targeting as broad a set of languages as possible. One such project is the aforementioned ODIN BIBREF361 , BIBREF397 , which aims to collect parallel snippets of text from Linguistics articles published on the web. ODIN specifically targets articles containing Interlinear Glossed Text (IGT), a semi-structured format for presenting text and a corresponding gloss that is commonly used in Linguistics.
Other projects that exist with the aim of creating text corpora for under-resourced languages by crawling the web are the Crúbadán project BIBREF395 and SeedLing BIBREF405 . The Crúbadán crawler uses seed data in a target language to generate word lists that in turn are used as queries for a search engine. The returned documents are then compared with the seed resource via an automatic language identifier, which is used to eliminate false positives. BIBREF395 reports that corpora for over 400 languages have been built using this method. The SeedLing project crawls texts from several web sources which has resulted in a total of 1451 languages from 105 language families. According to the authors, this represents 19% of the world's languages.
Much recent work on multilingual documents (openissues:multilingual) has been done with support for minority languages as a key goal. One of the common problems with gathering linguistic data from the web is that the data in the target language is often embedded in a document containing data in another language. This has spurred recent developments in text segmentation by language and word-level . BIBREF326 present a method to detect documents that contain text in more than one language and identify the languages present with their relative proportions in the document. The method is evaluated on real-world data from a web crawl targeted to collect documents for specific low-density languages.
for low-resource languages is a promising area for future work. One of the key questions that has not been clearly answered is how much data is needed to accurately model a language for purposes of . Work to date suggests that there may not be a simple answer to this question as accuracy varies according to the number and variety of languages modeled BIBREF32 , as well as the diversity of data available to model a specific language BIBREF150 .
Early research in tended to focus on a very limited number of languages (sometimes as few as 2). This situation has improved somewhat with many current off-the-shelf language identifiers supporting on the order of 50–100 languages (ots). The standout in this regard is BIBREF101 , supporting 1311 languages in its default configuration. However, evaluation of the identifier of BIBREF153 on a different domain found that the system suffered in terms of accuracy because it detected many languages that were not present in the test data BIBREF152 .
 BIBREF397 describe the construction of web crawlers specifically targeting IGT, as well as the identification of the languages represented in the IGT snippets. for thousands of languages from very small quantities of text is one of the issues that they have had to tackle. They list four specific challenges for in ODIN: (1) the large number of languages; (2) “unseen” languages that appear in the test data but not in training data; (3) short target sentences; and (4) (sometimes inconsistent) transliteration into Latin text. Their solution to this task is to take advantage of a domain-specific feature: they assume that the name of the language that they are extracting must appear in the document containing the IGT, and hence treat this as a co-reference resolution problem. They report that this approach significantly outperforms the text-based approach in this particular problem setting.
An interesting area to explore is the trade-off between the number of languages supported and the accuracy per-language. From existing results it is not clear if it is possible to continue increasing the number of languages supported without adversely affecting the average accuracy, but it would be useful to quantify if this is actually the case across a broad range of text sources. mostlanguages lists the articles where the with more than 30 languages has been investigated.
“Unseen” languages are languages that we do not have training data for but may nonetheless be encountered by a system when applied to real-world data. Dealing with languages for which we do not have training data has been identified as an issue by BIBREF1 and has also been mentioned by BIBREF361 as a specific challenge in harvesting linguistic data from the web. BIBREF233 use an unlabeled training set with a labeled evaluation set for token-level code switching identification between Modern Standard Arabic (MSA) and dialectal Arabic. They utilize existing dictionaries and also a morphological analyzer for MSA, so the system is supported by extensive external knowledge sources. The possibility to use unannotated training material is nonetheless a very useful feature.
Some authors have attempted to tackle the unseen language problem through attempts at unsupervised labeling of text by language. BIBREF225 uses an unsupervised clustering algorithm to separate a multilingual corpus into groups corresponding to languages. She uses singular value decomposition (SVD) to first identify the words that discriminate between documents and then to separate the terms into highly correlating groups. The documents grouped together by these discriminating terms are merged and the process is repeated until the wanted number of groups (corresponding to languages) is reached. BIBREF412 also presents an approach to unseen language problem, building graphs of co-occurrences of words in sentences, and then partitioning the graph using a custom graph-clustering algorithm which labels each word in the cluster with a single label. The number of labels is initialized to be the same as the number of words, and decreases as the algorithm is recursively applied. After a small number of iterations (the authors report 20), the labels become relatively stable and can be interpreted as cluster labels. Smaller clusters are then discarded, and the remaining clusters are interpreted as groups of words for each language. BIBREF413 compared the Chinese Whispers algorithm of BIBREF412 and Graclus clustering on unsupervised Tweet . They conclude that Chinese Whispers is better suited to . BIBREF414 used Fuzzy ART NNs for unsupervised language clustering for documents in Arabic, Persian, and Urdu. In Fuzzy ART, the clusters are also dynamically updated during the identification process.
 BIBREF415 also tackle the unseen language problem through clustering. They use a character representation for text, and a clustering algorithm that consists of an initial INLINEFORM0 -means phase, followed by particle-swarm optimization. This produces a large number of small clusters, which are then labeled by language through a separate step. BIBREF240 used co-occurrences of words with INLINEFORM1 -means clustering in word-level unsupervised . They used a Dirichlet process Gaussian mixture model (“DPGMM”), a non-parametric variant of a GMM, to automatically determine the number of clusters, and manually labeled the language of each cluster. BIBREF249 also used INLINEFORM2 -means clustering, and BIBREF416 used the INLINEFORM3 -means clustering algorithm in a custom framework. BIBREF244 utilized unlabeled data to improve their system by using a CRF autoencoder, unsupervised word embeddings, and word lists.
A different partial solution to the issue of unseen languages is to design the classifier to be able to output “unknown” as a prediction for language. This helps to alleviate one of the problems commonly associated with the presence of unseen languages – classifiers without an “unknown” facility are forced to pick a language for each document, and in the case of unseen languages, the choice may be arbitrary and unpredictable BIBREF412 . When is used for filtering purposes, i.e. to select documents in a single language, this mislabeling can introduce substantial noise into the data extracted; furthermore, it does not matter what or how many unseen languages there are, as long as they are consistently rejected. Therefore the “unknown” output provides an adequate solution to the unseen language problem for purposes of filtering.
The easiest way to implement unknown language detection is through thresholding. Most systems internally compute a score for each language for an unknown text, so thresholding can be applied either with a global threshold BIBREF33 , a per-language threshold BIBREF34 , or by comparing the score for the top-scoring INLINEFORM0 -languages. The problem of unseen languages and open-set recognition was also considered by BIBREF270 , BIBREF84 , and BIBREF126 . BIBREF126 experiments with one-class classification (“OCC”) and reaches an F-score on 98.9 using OC-SVMs (SVMs trained only with data from one language) to discriminate between 10 languages.
Another possible method for unknown language detection that has not been explored extensively in the literature, is the use of non-parametric mixture models based on Hierarchical Dirichlet Processes (“HDP”). Such models have been successful in topic modeling, where an outstanding issue with the popular LDA model is the need to specify the number of topics in advance. BIBREF326 introduced an approach to detecting multilingual documents that uses a model very similar to LDA, where languages are analogous to topics in the LDA model. Using a similar analogy, an HDP-based model may be able to detect documents that are written in a language that is not currently modeled by the system. BIBREF24 used LDA to cluster unannotated tweets. Recently BIBREF417 used LDA in unsupervised sentence-level . They manually identified the languages of the topics created with LDA. If there were more topics than languages then the topics in the same language were merged.
Filtering, a task that we mentioned earlier in this section, is a very common application of , and it is therefore surprising that there is little research on filtering for specific languages. Filtering is a limit case of with unseen languages, where all languages but one can be considered unknown. Future work could examine how useful different types of negative evidence are for filtering – if we want to detect English documents, e.g., are there empirical advantages in having distinct models of Italian and German (even if we don't care about the distinction between the two languages), or can we group them all together in a single “negative” class? Are we better off including as many languages as possible in the negative class, or can we safely exclude some?
Multilingual documents are documents that contain text in more than one language. In constructing the hrWac corpus, BIBREF97 found that 4% of the documents they collected contained text in more than one language. BIBREF329 report that web pages in many languages contain formulaic strings in English that do not actually contribute to the content of the page, but may nonetheless confound attempts to identify multilingual documents. Recent research has investigated how to make use of multilingual documents from sources such as web crawls BIBREF40 , forum posts BIBREF263 , and microblog messages BIBREF418 . However, most methods assume that a document contains text from a single language, and so are not directly applicable to multilingual documents.
Handling of multilingual documents has been named as an open research question BIBREF1 . Most NLP techniques presuppose monolingual input data, so inclusion of data in foreign languages introduces noise, and can degrade the performance of NLP systems. Automatic detection of multilingual documents can be used as a pre-filtering step to improve the quality of input data. Detecting multilingual documents is also important for acquiring linguistic data from the web, and has applications in mining bilingual texts for statistical MT from online resources BIBREF418 , or to study code-switching phenomena in online communications. There has also been interest in extracting text resources for low-density languages from multilingual web pages containing both the low-density language and another language such as English.
The need to handle multilingual documents has prompted researchers to revisit the granularity of . Many researchers consider document-level to be relatively easy, and that sentence-level and word-level are more suitable targets for further research. However, word-level and sentence-level tokenization are not language-independent tasks, and for some languages are substantially harder than others BIBREF419 .
 BIBREF112 is a language identifier that supports identification of multilingual documents. The system is based on a vector space model using cosine similarity. for multilingual documents is performed through the use of virtual mixed languages. BIBREF112 shows how to construct vectors representative of particular combinations of languages independent of the relative proportions, and proposes a method for choosing combinations of languages to consider for any given document. One weakness of this approach is that for exhaustive coverage, this method is factorial in the number of languages, and as such intractable for a large set of languages. Furthermore, calculating the parameters for the virtual mixed languages becomes infeasibly complex for mixtures of more than 3 languages.
As mentioned previously, BIBREF326 propose an LDA-inspired method for multilingual documents that is able to identify that a document is multilingual, identify the languages present and estimate the relative proportions of the document written in each language. To remove the need to specify the number of topics (or in this case, languages) in advance, BIBREF326 use a greedy heuristic that attempts to find the subset of languages that maximizes the posterior probability of a target document. One advantage of this approach is that it is not constrained to 3-language combinations like the method of BIBREF112 . Language set identification has also been considered by BIBREF34 , BIBREF407 , and BIBREF420 , BIBREF276 .
To encourage further research on for multilingual documents, in the aforementioned shared task hosted by the Australiasian Language Technology Workshop 2010, discussed in evaluation:sharedtasks, participants were required to predict the language(s) present in a held-out test set containing monolingual and bilingual documents BIBREF378 . The dataset was prepared using data from Wikipedia, and bilingual documents were produced using a segment from an article in one language and a segment from the equivalent article in another language. Equivalence between articles was determined using the cross-language links embedded within each Wikipedia article. The winning entry BIBREF421 first built monolingual models from multilingual training data, and then applied them to a chunked version of the test data, making the final prediction a function of the prediction over chunks.
Another approach to handling multilingual documents is to attempt to segment them into contiguous monolingual segments. In addition to identifying the languages present, this requires identifying the locations of boundaries in the text which mark the transition from one language to another. Several methods for supervised language segmentation have been proposed. BIBREF33 generalized a algorithm for monolingual documents by adding a dynamic programming algorithm based on a simple Markov model of multilingual documents. More recently, multilingual algorithms have also been presented by BIBREF140 , BIBREF73 , BIBREF74 , BIBREF106 , and BIBREF82 .
of short strings is known to be challenging for existing techniques. BIBREF37 tested four different classification methods, and found that all have substantially lower accuracy when applied to texts of 25 characters compared with texts of 125 characters. These findings were later strengthened, for example, by BIBREF145 and BIBREF148 .
 BIBREF195 describes a method specifically targeted at short texts that augments a dictionary with an affix table, which was tested over synthetic data derived from a parallel bible corpus. BIBREF145 focus on messages of 5–21 characters, using language models over data drawn the from Universal Declaration of Human Rights (UDHR). We would expect that generic methods for of short texts should be effective in any domain where short texts are found, such as search engine queries or microblog messages. However, BIBREF195 and BIBREF145 both only test their systems in a single domain: bible texts in the former case, and texts from the UDHR in the latter case. Other research has shown that results do not trivially generalize across domains BIBREF32 , and found that in UDHR documents is relatively easy BIBREF301 . For both bible and UDHR data, we expect that the linguistic content is relatively grammatical and well-formed, an expectation that does not carry across to domains such as search engine queries and microblogs. Another “short text” domain where has been studied is of proper names. BIBREF306 identify this as an issue. BIBREF422 found that of names is more accurate than of generic words of equivalent length.
 BIBREF299 raise an important criticism of work on Twitter messages to date: only a small number of European languages has been considered. BIBREF299 expand the scope of for Twitter, covering nine languages across Cyrillic, Arabic and Devanagari scripts. BIBREF152 expand the evaluation further, introducing a dataset of language-labeled Twitter messages across 65 languages constructed using a semi-automatic method that leverages user identity to avoid inducing a bias in the evaluation set towards messages that existing systems are able to identify correctly. BIBREF152 also test a 1300-language model based on BIBREF153 , but find that it performs relatively poorly in the target domain due to a tendency to over-predict low-resource languages.
Work has also been done on of single words in a document, where the task is to label each word in the document with a specific language. Work to date in this area has assumed that word tokenization can be carried out on the basis of whitespace. BIBREF35 explore word-level in the context of segmenting a multilingual document into monolingual segments. Other work has assumed that the languages present in the document are known in advance.
Conditional random fields (“CRFs”: BIBREF423 ) are a sequence labeling method most often used in for labeling the language of individual words in a multilingual text. CRFs can be thought of as a finite state model with probabilistic transition probabilities optimised over pre-defined cliques. They can use any observations made from the test document as features, including language labels given by monolingual language identifiers for words. BIBREF40 used a CRF trained with generalized expectation criteria, and found it to be the most accurate of all methods tested (NB, LR, HMM, CRF) at word-level . BIBREF40 introduce a technique to estimate the parameters using only monolingual data, an important consideration as there is no readily-available collection of manually-labeled multilingual documents with word-level annotations. BIBREF263 present a two-pass approach to processing Turkish-Dutch bilingual documents, where the first pass labels each word independently and the second pass uses the local context of a word to further refine the predictions. BIBREF263 achieved 97,6% accuracy on distinguishing between the two languages using a linear-chain CRF. BIBREF180 are the only ones so far to use a CRF for of monolingual texts. With a CRF, they attained a higher F-score in German dialect identification than NB or an ensemble consisting of NB, CRF, and SVM. Lately CRFs were also used for by BIBREF52 and BIBREF44 . BIBREF296 investigate of individual words in the context of code switching. They find that smoothing of models substantially improves accuracy of a language identifier based on a NB classifier when applied to individual words.
While one line of research into has focused on pushing the boundaries of how many languages are supported simultaneously by a single system BIBREF382 , BIBREF36 , BIBREF153 , another has taken a complementary path and focused on in groups of similar languages. Research in this area typically does not make a distinction between languages, varieties and dialects, because such terminological differences tend to be politically rather than linguistically motivated BIBREF424 , BIBREF382 , BIBREF5 , and from an NLP perspective the challenges faced are very similar.
for closely-related languages, language varieties, and dialects has been studied for Malay–Indonesian BIBREF332 , Indian languages BIBREF114 , South Slavic languages BIBREF377 , BIBREF98 , BIBREF4 , BIBREF425 , Serbo-Croatian dialects BIBREF426 , English varieties BIBREF278 , BIBREF45 , Dutch–Flemish BIBREF53 , Dutch dialects (including a temporal dimension) BIBREF427 , German Dialects BIBREF428 Mainland–Singaporean–Taiwanese Chinese BIBREF429 , Portuguese varieties BIBREF5 , BIBREF259 , Spanish varieties BIBREF70 , BIBREF147 , French varieties BIBREF430 , BIBREF431 , BIBREF432 , languages of the Iberian Peninsula BIBREF388 , Romanian dialects BIBREF120 , and Arabic dialects BIBREF41 , BIBREF78 , BIBREF433 , BIBREF75 , BIBREF434 , the last of which we discuss in more detail in this section. As to off-the-shelf tools which can identify closely-related languages, BIBREF79 released a system trained to identify 27 languages, including 10 language varieties. Closely-related languages, language varieties, and dialects have also been the focus of a number of shared tasks in recent years as discussed in evaluation:sharedtasks.
Similar languages are a known problem for existing language identifiers BIBREF332 , BIBREF435 . BIBREF34 identify language pairs from the same language family that also share a common script and the same encoding, as the most difficult to discriminate. BIBREF98 report that achieves only 45% accuracy when trained and tested on 3-way Bosnian/Serbian/Croatian dataset. BIBREF278 found that methods are not competitive with conventional word-based document categorization methods in distinguishing between national varieties of English. BIBREF332 reports that a character trigram model is able to distinguish Malay/Indonesian from English, French, German, and Dutch, but handcrafted rules are needed to distinguish between Malay and Indonesian. One kind of rule is the use of “exclusive words” that are known to occur in only one of the languages. A similar idea is used by BIBREF98 , in automatically learning a “blacklist” of words that have a strong negative correlation with a language – i.e. their presence implies that the text is not written in a particular language. In doing so, they achieve an overall accuracy of 98%, far surpassing the 45% of . BIBREF153 also adopts such “discriminative training” to make use of negative evidence in .
 BIBREF435 observed that general-purpose approaches to typically use a character representation of text, but successful approaches for closely-related languages, varieties, and dialects seem to favor a word-based representation or higher-order (e.g. 4-grams, 5-grams, and even 6-grams) that often cover whole words BIBREF429 , BIBREF98 , BIBREF278 , BIBREF343 . The study compared character with word-based representations for over varieties of Spanish, Portuguese and French, and found that word-level models performed better for varieties of Spanish, but character models perform better in the case of Portuguese and French.
To train accurate and robust systems that discriminate between language varieties or similar languages, models should ideally be able to capture not only lexical but more abstract systemic differences between languages. One way to achieve this, is by using features that use de-lexicalized text representations (e.g. by substituting named entities or content words by placeholders), or at a higher level of abstraction, using POS tags or other morphosyntactic information BIBREF70 , BIBREF390 , BIBREF43 , or even adversarial machine learning to modify the learned representations to remove such artefacts BIBREF358 . Finally, an interesting research direction could be to combine work on closely-related languages with the analysis of regional or dialectal differences in language use BIBREF436 , BIBREF437 , BIBREF438 , BIBREF432 .
In recent years, there has been a significant increase of interest in the computational processing of Arabic. This is evidenced by a number of research papers in several NLP tasks and applications including the identification/discrimination of Arabic dialects BIBREF41 , BIBREF78 . Arabic is particularly interesting for researchers interested in language variation due to the fact that the language is often in a diaglossic situation, in which the standard form (Modern Standard Arabic or “MSA”) coexists with several regional dialects which are used in everyday communication.
Among the studies published on the topic of Arabic , BIBREF41 proposed a supervised approach to distinguish between MSA and Egyptian Arabic at the sentence level, and achieved up to 85.5% accuracy over an Arabic online commentary dataset BIBREF379 . BIBREF433 achieved higher results over the same dataset using a linear-kernel SVM classifier.
 BIBREF78 compiled a dataset containing MSA, Egyptian Arabic, Gulf Arabic and Levantine Arabic, and used it to investigate three classification tasks: (1) MSA and dialectal Arabic; (2) four-way classification – MSA, Egyptian Arabic, Gulf Arabic, and Levantine Arabic; and (3) three-way classification – Egyptian Arabic, Gulf Arabic, and Levantine Arabic.
 BIBREF439 explores the use of sentence-level Arabic dialect identification as a pre-processor for MT, in customizing the selection of the MT model used to translate a given sentence to the dialect it uses. In performing dialect-specific MT, the authors achieve an improvement of 1.0% BLEU score compared with a baseline system which does not differentiate between Arabic dialects.
Finally, in addition to the above-mentioned dataset of BIBREF379 , there are a number of notable multi-dialect corpora of Arabic: a multi-dialect corpus of broadcast speeches used in the ADI shared task BIBREF440 ; a multi-dialect corpus of (informal) written Arabic containing newspaper comments and Twitter data BIBREF441 ; a parallel corpus of 2,000 sentences in MSA, Egyptian Arabic, Tunisian Arabic, Jordanian Arabic, Palestinian Arabic, and Syrian Arabic, in addition to English BIBREF442 ; a corpus of sentences in 18 Arabic dialects (corresponding to 18 different Arabic-speaking countries) based on data manually sourced from web forums BIBREF75 ; and finally two recently compiled multi-dialect corpora containing microblog posts from Twitter BIBREF241 , BIBREF443 .
While not specifically targeted at identifying language varieties, BIBREF355 made the critical observation that when naively trained, systems tend to perform most poorly over language varieties from the lowest socio-economic demographics (focusing particularly on the case of English), as they tend to be most under-represented in training corpora. If, as a research community, we are interested in the social equitability of our systems, it is critical that we develop datasets that are truly representative of the global population, to better quantify and remove this effect. To this end, BIBREF355 detail a method for constructing a more representative dataset, and demonstrate the impact of training on such a dataset in terms of alleviating socio-economic bias.
One approach to is to build a generic language identifier that aims to correctly identify the language of a text without any information about the source of the text. Some work has specifically targeted across multiple domains, learning characteristics of languages that are consistent between different sources of text BIBREF150 . However, there are often domain-specific features that are useful for identifying the language of a text. In this survey, our primary focus has been on of digitally-encoded text, using only the text itself as evidence on which to base the prediction of the language. Within a text, there can sometimes be domain-specific peculiarities that can be used for . For example, BIBREF399 investigates of user-to-user messages in the eBay e-commerce portal. He finds that using only the first two and last two words of a message is sufficient for identifying the language of a message.
This article has presented a comprehensive survey on language identification of digitally-encoded text. We have shown that is a rich, complex, and multi-faceted problem that has engaged a wide variety of research communities. accuracy is critical as it is often the first step in longer text processing pipelines, so errors made in will propagate and degrade the performance of later stages. Under controlled conditions, such as limiting the number of languages to a small set of Western European languages and using long, grammatical, and structured text such as government documents as training data, it is possible to achieve near-perfect accuracy. This led many researchers to consider a solved problem, as argued by BIBREF2 . However, becomes much harder when taking into account the peculiarities of real-world data, such as very short documents (e.g. search engine queries), non-linguistic “noise” (e.g. HTML markup), non-standard use of language (e.g. as seen in social media data), and mixed-language documents (e.g. forum posts in multilingual web forums).
Modern approaches to are generally data-driven and are based on comparing new documents with models of each target language learned from data. The types of models as well as the sources of training data used in the literature are diverse, and work to date has not compared and evaluated these in a systematic manner, making it difficult to draw broader conclusions about what the “best” method for actually is. We have attempted to synthesize results to date to identify a set of “best practices”, but these should be treated as guidelines and should always be considered in the broader context of a target application.
Existing work on serves to illustrate that the scope and depth of the problem are much greater than they may first appear. In openissues, we discussed open issues in , identifying the key challenges, and outlining opportunities for future research. Far from being a solved problem, aspects of make it an archetypal learning task with subtleties that could be tackled by future work on supervised learning, representation learning, multi-task learning, domain adaptation, multi-label classification and other subfields of machine learning. We hope that this paper can serve as a reference point for future work in the area, both for providing insight into work to date, as well as pointing towards the key aspects that merit further investigation.
This research was supported in part by the Australian Research Council, the Kone Foundation and the Academy of Finland. We would like to thank Kimmo Koskenniemi for many valuable discussions and comments concerning the early phases of the features and the methods sections.

Figure and table captions:
Table 1: Excerpts from Wikipedia articles on NLP in different languages.
Table 2: List of articles (2013–2017) where relative frequencies of character n-grams have been used as features. The columns indicate the length of the n-grams used. “***” indicates the empirically best n-gram length in that paper, and “**” the secondbest n-gram length. “*” indicates less effective n-gram lengths. “...” indicates that even higher n-grams were used.
Table 3: List of recent articles where Markovian character n-grams have been used as features. The columns indicate the length of the n-grams used. “***” indicates the best and “**” the second-best n-gram length as reported in the article in question. “*” indicates less effective n-gram lengths.
Table 4: Recent papers (2016–) where the frequency of character n-grams has been used to generate feature vectors. The columns indicate the length of the n-grams used, and the machine learning method(s) used. The relevant section numbers for the methods are mentioned in parentheses. “...” indicates that even more methods were used.
Table 5: List of articles (2016-) where character n-grams of differing sizes have been used as features. The numbered columns indicate the length of the n-grams used. The method column indicates the method used with the n-grams. The relevant section numbers are mentioned in parentheses. “...” indicates that even more methods were used.
Table 6: References (2016-) where prefixes and suffixes collected from a training corpus have been used for LI. The columns indicate the length of the prefixes and suffixes. The method column indicates the method used. The relevant section numbers are mentioned in parentheses. “...” indicates that even more methods were used.
Table 7: Word characteristics used by Mustonen (1965).
Table 8: References (2015–) where word n-grams have been used as features. The numbered columns indicate the length of the n-grams used. “***” indicates the best and “**” the second best n-gram length, as evaluated in the article in question. “*” indicates less effective n-gram lengths. Identical numbers of asterisks indicate that there was no clear order of effectiveness, or that all the n-gram lengths were used simultaneously. The method column indicates the method used. The relevant section numbers are mentioned in parentheses. “...” indicates that even more methods were used.
Table 9: List of articles where word and character n-grams have been used as features. The numbered columns indicate the length of the word n-grams and char-column the length of character n-grams used. The method column indicates the method used. The relevant section numbers are mentioned in parentheses. “...” indicates that even more methods were used.
Table 10: References where SVMs have been tested with different kernels. The columns indicate the kernels used. “dn” stands for polynomial kernel.
Table 11: Published LI Datasets
Table 12: List of LI shared tasks.
Table 13: DSLCC: the languages included in each version of the corpus collection, grouped by language similarity.
Table 14: Empirical evaluations with more than 30 languages.

Questions about this paper:
what are the off-the-shelf systems discussed in the paper?

Answers:
Answer with content missing: (Names of many identifiers missing) TextCat, ChromeCLD, LangDetect, langid.py, whatlang, whatthelang, YALI, LDIG, Polyglot 3000, Lextek Language Identifier and Open Xerox Language Identifier.