Language identification (“”) is the task of determining the natural language that a document or part thereof is written in. Recognizing text in a specific language comes naturally to a human reader familiar with the language. intro:langid presents excerpts from Wikipedia articles in different languages on the topic of Natural Language Processing (“NLP”), labeled according to the language they are written in. Without referring to the labels, readers of this article will certainly have recognized at least one language in intro:langid, and many are likely to be able to identify all the languages therein.Research into aims to mimic this human ability to recognize specific languages. Over the years, a number of computational approaches have been developed that, through the use of specially-designed algorithms and indexing structures, are able to infer the language being used without the need for human intervention. The capability of such systems could be described as super-human: an average person may be able to identify a handful of languages, and a trained linguist or translator may be familiar with many dozens, but most of us will have, at some point, encountered written texts in languages they cannot place. However, research aims to develop systems that are able to identify any human language, a set which numbers in the thousands BIBREF0 .In a broad sense, applies to any modality of language, including speech, sign language, and handwritten text, and is relevant for all means of information storage that involve language, digital or otherwise. However, in this survey we limit the scope of our discussion to of written text stored in a digitally-encoded form.Research to date on has traditionally focused on monolingual documents BIBREF1 (we discuss for multilingual documents in openissues:multilingual). In monolingual , the task is to assign each document a unique language label. Some work has reported near-perfect accuracy for of large documents in a small number of languages, prompting some researchers to label it a “solved task” BIBREF2 . However, in order to attain such accuracy, simplifying assumptions have to be made, such as the aforementioned monolinguality of each document, as well as assumptions about the type and quantity of data, and the number of languages considered.The ability to accurately detect the language that a document is written in is an enabling technology that increases accessibility of data and has a wide variety of applications. For example, presenting information in a user's native language has been found to be a critical factor in attracting website visitors BIBREF3 . Text processing techniques developed in natural language processing and Information Retrieval (“IR”) generally presuppose that the language of the input text is known, and many techniques assume that all documents are in the same language. In order to apply text processing techniques to real-world data, automatic is used to ensure that only documents in relevant languages are subjected to further processing. In information storage and retrieval, it is common to index documents in a multilingual collection by the language that they are written in, and is necessary for document collections where the languages of documents are not known a-priori, such as for data crawled from the World Wide Web. Another application of that predates computational methods is the detection of the language of a document for routing to a suitable translator. This application has become even more prominent due to the advent of Machine Translation (“MT”) methods: in order for MT to be applied to translate a document to a target language, it is generally necessary to determine the source language of the document, and this is the task of . also plays a part in providing support for the documentation and use of low-resource languages. One area where is frequently used in this regard is in linguistic corpus creation, where is used to process targeted web crawls to collect text resources for low-resource languages.A large part of the motivation for this article is the observation that lacks a “home discipline”, and as such, the literature is fragmented across a number of fields, including NLP, IR, machine learning, data mining, social medial analysis, computer science education, and systems science. This has hampered the field, in that there have been many instances of research being carried out with only partial knowledge of other work on the topic, and the myriad of published systems and datasets.Finally, it should be noted that this survey does not make a distinction between languages, language varieties, and dialects. Whatever demarcation is made between languages, varieties and dialects, a system is trained to identify the associated document classes. Of course, the more similar two classes are, the more challenging it is for a system to discriminate between them. Training a system to discriminate between similar languages such as Croatian and Serbian BIBREF4 , language varieties like Brazilian and European Portuguese BIBREF5 , or a set of Arabic dialects BIBREF6 is more challenging than training systems to discriminate between, for example, Japanese and Finnish. Even so, as evidenced in this article, from a computational perspective, the algorithms and features used to discriminate between languages, language varieties, and dialects are identical.is in some ways a special case of text categorization, and previous research has examined applying standard text categorization methods to BIBREF7 , BIBREF8 . BIBREF9 provides a definition of text categorization, which can be summarized as the task of mapping a document onto a pre-determined set of classes. This is a very broad definition, and indeed one that is applicable to a wide variety of tasks, amongst which falls modern-day . The archetypal text categorization task is perhaps the classification of newswire articles according to the topics that they discuss, exemplified by the Reuters-21578 dataset BIBREF10 . However, has particular characteristics that make it different from typical text categorization tasks:These distinguishing characteristics present unique challenges and offer particular opportunities, so much so that research in has generally proceeded independently of text categorization research. In this survey, we will examine the common themes and ideas that underpin research in . We begin with a brief history of research that has led to modern (history), and then proceed to review the literature, first introducing the mathematical notation used in the article (notation), and then providing synthesis and analysis of existing research, focusing specifically on the representation of text (features) and the learning algorithms used (methods). We examine the methods for evaluating the quality of the systems (evaluation) as well as the areas where has been applied (applications), and then provide an overview of “off-the-shelf” systems (ots). We conclude the survey with a discussion of the open issues in (openissues), enumerating issues and existing efforts to address them, as well as charting the main directions where further research in is required.Although there are some dedicated survey articles, these tend to be relatively short; there have not been any comprehensive surveys of research in automated LI of text to date. The largest survey so far can be found in the literature review of Marco Lui's PhD thesis BIBREF11 , which served as an early draft and starting point for the current article. BIBREF12 provides a historical overview of language identification focusing on the use of language models. BIBREF13 gives a brief overview of some of the methods used for , and BIBREF14 provide a review of some of the techniques and applications used previously. BIBREF15 gives a short overview of some of the challenges, algorithms and available tools for . BIBREF16 provides a brief summary of , how it relates to other research areas, and some outstanding challenges, but only does so in general terms and does not go into any detail about existing work in the area. Another brief article about is BIBREF17 , which covers both of spoken language as well as of written documents, and also discusses of documents stored as images rather than digitally-encoded text.as a task predates computational methods – the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional program for text is by BIBREF18 , a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, BIBREF19 considered the problem of automatic . According to BIBREF20 and the available abstract of Nakamura's article, his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual . BIBREF20 wrote his master's thesis “Language Identification by Statistical Analysis” for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use of text in military intelligence settings is evidenced by the recent articles of, for example, BIBREF21 , BIBREF22 , BIBREF23 , and BIBREF24 . As features for , BIBREF20 used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic ( INLINEFORM0 ), he managed to achieve 89% accuracy over 53 characters when distinguishing between English and Spanish. His thesis actually includes the identifier program code (for the IBM System/360 Model 67 mainframe), and even the language models in printed form.Much of the earliest work on automatic was focused on identification of spoken language, or did not make a distinction between written and spoken language. For example, the work of BIBREF25 is primarily focused on of spoken utterances, but makes a broader contribution in demonstrating the feasibility of on the basis of a statistical model of broad phonetic information. However, their experiments do not use actual speech data, but rather “synthetic” data in the form of phonetic transcriptions derived from written text.Another subfield of speech technology, speech synthesis, has also generated a considerable amount of research in the of text, starting from the 1980s. In speech synthesis, the need to know the source language of individual words is crucial in determining how they should be pronounced. BIBREF26 uses the relative frequencies of character trigrams as probabilities and determines the language of words using a Bayesian model. Church explains the method – that has since been widely used in LI – as a small part of an article concentrating on many aspects of letter stress assignment in speech synthesis, which is probably why BIBREF27 is usually attributed to being the one to have introduced the aforementioned method to of text. As Beesley's article concentrated solely on the problem of LI, this single focus probably enabled his research to have greater visibility. The role of the program implementing his method was to route documents to MT systems, and Beesley's paper more clearly describes what has later come to be known as a character model. The fact that the distribution of characters is relatively consistent for a given language was already well known.The highest-cited early work on automatic is BIBREF7 . Cavnar and Trenkle's method (which we describe in detail in outofplace) builds up per-document and per-language profiles, and classifies a document according to which language profile it is most similar to, using a rank-order similarity metric. They evaluate their system on 3478 documents in eight languages obtained from USENET newsgroups, reporting a best overall accuracy of 99.8%. Gertjan van Noord produced an implementation of the method of Cavnar and Trenkle named , which has become eponymous with the method itself. is packaged with pre-trained models for a number of languages, and so it is likely that the strong results reported by Cavnar and Trenkle, combined with the ready availability of an “off-the-shelf” implementation, has resulted in the exceptional popularity of this particular method. BIBREF7 can be considered a milestone in automatic , as it popularized the use of automatic methods on character models for , and to date the method is still considered a benchmark for automatic .This section introduces the notation used throughout this article to describe methods. We have translated the notation in the original papers to our notation, to make it easier to see the similarities and differences between the methods presented in the literature. The formulas presented could be used to implement language identifiers and re-evaluate the studies they were originally presented in.A corpus INLINEFORM0 consists of individual tokens INLINEFORM1 which may be bytes, characters or words. INLINEFORM2 is comprised of a finite sequence of individual tokens, INLINEFORM3 . The total count of individual tokens INLINEFORM4 in INLINEFORM5 is denoted by INLINEFORM6 . In a corpus INLINEFORM7 with non-overlapping segments INLINEFORM8 , each segment is referred to as INLINEFORM9 , which may be a short document or a word or some other way of segmenting the corpus. The number of segments is denoted as INLINEFORM10 .A feature INLINEFORM0 is some countable characteristic of the corpus INLINEFORM1 . When referring to the set of all features INLINEFORM2 in a corpus INLINEFORM3 , we use INLINEFORM4 , and the number of features is denoted by INLINEFORM5 . A set of unique features in a corpus INLINEFORM6 is denoted by INLINEFORM7 . The number of unique features is referred to as INLINEFORM8 . The count of a feature INLINEFORM9 in the corpus INLINEFORM10 is referred to as INLINEFORM11 . If a corpus is divided into segments INLINEFORM12 , the count of a feature INLINEFORM13 in INLINEFORM14 is defined as the sum of counts over the segments of the corpus, i.e. INLINEFORM15 . Note that the segmentation may affect the count of a feature in INLINEFORM16 as features do not cross segment borders.A frequently-used feature is an , which consists of a contiguous sequence of INLINEFORM0 individual tokens. An starting at position INLINEFORM1 in a corpus segment is denoted INLINEFORM2 , where positions INLINEFORM3 remain within the same segment of the corpus as INLINEFORM4 . If INLINEFORM5 , INLINEFORM6 is an individual token. When referring to all of length INLINEFORM7 in a corpus INLINEFORM8 , we use INLINEFORM9 and the count of all such is denoted by INLINEFORM10 . The count of an INLINEFORM11 in a corpus segment INLINEFORM12 is referred to as INLINEFORM13 and is defined by count: DISPLAYFORM0 The set of languages is INLINEFORM0 , and INLINEFORM1 denotes the number of languages. A corpus INLINEFORM2 in language INLINEFORM3 is denoted by INLINEFORM4 . A language model INLINEFORM5 based on INLINEFORM6 is denoted by INLINEFORM7 . The features given values by the model INLINEFORM8 are the domain INLINEFORM9 of the model. In a language model, a value INLINEFORM10 for the feature INLINEFORM11 is denoted by INLINEFORM12 . For each potential language INLINEFORM13 of a corpus INLINEFORM14 in an unknown language, a resulting score INLINEFORM15 is calculated. A corpus in an unknown language is also referred to as a test document.The design of a supervised language identifier can generally be deconstructed into four key steps:A representation of text is selectedA model for each language is derived from a training corpus of labelled documentsA function is defined that determines the similarity between a document and each languageThe language of a document is predicted based on the highest-scoring modelThe theoretical description of some of the methods leaves room for interpretation on how to implement them. BIBREF28 define an algorithm to be any well-defined computational procedure. BIBREF29 introduces a three-tiered classification where programs implement algorithms and algorithms implement functions. The examples of functions given by BIBREF29 , sort and find max differ from our identify language as they are always solvable and produce the same results. In this survey, we have considered two methods to be the same if they always produce exactly the same results from exactly the same inputs. This would not be in line with the definition of an algorithm by BIBREF29 , as in his example there are two different algorithms mergesort and quicksort that implement the function sort, always producing identical results with the same input. What we in this survey call a method, is actually a function in the tiers presented by BIBREF29 .In this section, we present an extensive list of features used in , some of which are not self-evident. The equations written in the unified notation defined earlier show how the values INLINEFORM0 used in the language models are calculated from the tokens INLINEFORM1 . For each feature type, we generally introduce the first published article that used that feature type, as well as more recent articles where the feature type has been considered.In , text is typically modeled as a stream of characters. However, there is a slight mismatch between this view and how text is actually stored: documents are digitized using a particular encoding, which is a mapping from characters (e.g. a character in an alphabet), onto the actual sequence of bytes that is stored and transmitted by computers. Encodings vary in how many bytes they use to represent each character. Some encodings use a fixed number of bytes for each character (e.g. ASCII), whereas others use a variable-length encoding (e.g. UTF-8). Some encodings are specific to a given language (e.g. GuoBiao 18030 or Big5 for Chinese), whereas others are specifically designed to represent as many languages as possible (e.g. the Unicode family of encodings). Languages can often be represented in a number of different encodings (e.g. UTF-8 and Shift-JIS for Japanese), and sometimes encodings are specifically designed to share certain codepoints (e.g. all single-byte UTF-8 codepoints are exactly the same as ASCII). Most troubling for , isomorphic encodings can be used to encode different languages, meaning that the determination of the encoding often doesn't help in honing in on the language. Infamous examples of this are the ISO-8859 and EUC encoding families. Encodings pose unique challenges for practical applications: a given language can often be encoded in different forms, and a given encoding can often map onto multiple languages.Some research has included an explicit encoding detection step to resolve bytes to the characters they represent BIBREF30 , effectively transcoding the document into a standardized encoding before attempting to identify the language. However, transcoding is computationally expensive, and other research suggests that it may be possible to ignore encoding and build a single per-language model covering multiple encodings simultaneously BIBREF31 , BIBREF32 . Another solution is to treat each language-encoding pair as a separate category BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 . The disadvantage of this is that it increases the computational cost by modeling a larger number of classes. Most of the research has avoided issues of encoding entirely by assuming that all documents use the same encoding BIBREF37 . This may be a reasonable assumption in some settings, such as when processing data from a single source (e.g. all data from Twitter and Wikipedia is UTF-8 encoded). In practice, a disadvantage of this approach may be that some encodings are only applicable to certain languages (e.g. S-JIS for Japanese and Big5 for Chinese), so knowing that a document is in a particular encoding can provide information that would be lost if the document is transcoded to a universal encoding such as UTF-8. BIBREF38 used a parallel state machine to detect which encoding scheme a file could potentially have been encoded with. The knowledge of the encoding, if detected, is then used to narrow down the possible languages.Most features and methods do not make a distinction between bytes or characters, and because of this we will present feature and method descriptions in terms of characters, even if byte tokenization was actually used in the original research.In this section, we review how individual character tokens have been used as features in . BIBREF39 used the formatting of numbers when distinguishing between Malay and Indonesian. BIBREF40 used the presence of non-alphabetic characters between the current word and the words before and after as features. BIBREF41 used emoticons (or emojis) in Arabic dialect identification with Naive Bayes (“NB”; see product). Non-alphabetic characters have also been used by BIBREF42 , BIBREF43 , BIBREF44 , and BIBREF45 . BIBREF46 used knowledge of alphabets to exclude languages where a language-unique character in a test document did not appear. BIBREF47 used alphabets collected from dictionaries to check if a word might belong to a language. BIBREF48 used the Unicode database to get the possible languages of individual Unicode characters. Lately, the knowledge of relevant alphabets has been used for also by BIBREF49 and BIBREF44 .Capitalization is mostly preserved when calculating character frequencies, but in contexts where it is possible to identify the orthography of a given document and where capitalization exists in the orthography, lowercasing can be used to reduce sparseness. In recent work, capitalization was used as a special feature by BIBREF42 , BIBREF43 , and BIBREF45 . BIBREF50 was the first to use the length of words in . BIBREF51 used the length of full person names comprising several words. Lately, the number of characters in words has been used for by BIBREF52 , BIBREF53 , BIBREF44 , and BIBREF45 . BIBREF52 also used the length of the two preceding words. BIBREF54 used character frequencies as feature vectors. In a feature vector, each feature INLINEFORM0 has its own integer value. The raw frequency – also called term frequency (TF) – is calculated for each language INLINEFORM1 as: DISPLAYFORM0  BIBREF20 was the first to use the probability of characters. He calculated the probabilities as relative frequencies, by dividing the frequency of a feature found in the corpus by the total count of features of the same type in the corpus. When the relative frequency of a feature INLINEFORM0 is used as a value, it is calculated for each language INLINEFORM1 as: DISPLAYFORM0  BIBREF55 calculated the relative frequencies of one character prefixes, and BIBREF56 did the same for one character suffixes. BIBREF57 calculated character frequency document frequency (“LFDF”) values. BIBREF58 compared their own Inverse Class Frequency (“ICF”) method with the Arithmetic Average Centroid (“AAC”) and the Class Feature Centroid (“CFC”) feature vector updating methods. In ICF a character appearing frequently only in some language gets more positive weight for that language. The values differ from Inverse Document Frequency (“IDF”, artemenko1), as they are calculated using also the frequencies of characters in other languages. Their ICF-based vectors generally performed better than those based on AAC or CFC. BIBREF59 explored using the relative frequencies of characters with similar discriminating weights. BIBREF58 also used Mutual Information (“MI”) and chi-square weighting schemes with characters. BIBREF32 compared the identification results of single characters with the use of character bigrams and trigrams when classifying over 67 languages. Both bigrams and trigrams generally performed better than unigrams. BIBREF60 also found that the identification results from identifiers using just characters are generally worse than those using character sequences.In this section we consider the different combinations of characters used in the literature. Character mostly consist of all possible characters in a given encoding, but can also consist of only alphabetic or ideographic characters. BIBREF56 calculated the co-occurrence ratios of any two characters, as well as the ratio of consonant clusters of different sizes to the total number of consonants. BIBREF61 used the combination of every bigram and their counts in words. BIBREF53 used the proportions of question and exclamation marks to the total number of the end of sentence punctuation as features with several machine learning algorithms. BIBREF62 used FastText to generate character n-gram embeddings BIBREF63 . Neural network generated embeddings are explained in cooccurrencesofwords. BIBREF20 used the relative frequencies of vowels following vowels, consonants following vowels, vowels following consonants and consonants following consonants. BIBREF52 used vowel-consonant ratios as one of the features with Support Vector Machines (“SVMs”, supportvectormachines), Decision Trees (“DTs”, decisiontrees), and Conditional Random Fields (“CRFs”, openissues:short). BIBREF41 used the existence of word lengthening effects and repeated punctuation as features. BIBREF64 used the presence of characters repeating more than twice in a row as a feature with simple scoring (simple1). BIBREF65 used more complicated repetitions identified by regular expressions. BIBREF66 used letter and character bigram repetition with a CRF. BIBREF67 used the count of character sequences with three or more identical characters, using several machine learning algorithms.Character are continuous sequences of characters of length INLINEFORM0 . They can be either consecutive or overlapping. Consecutive character bigrams created from the four character sequence door are do and or, whereas the overlapping bigrams are do, oo, and or. Overlapping are most often used in the literature. Overlapping produces a greater number and variety of from the same amount of text. BIBREF20 was the first to use combinations of any two characters. He calculated the relative frequency of each bigram. RFTable2 lists more recent articles where relative frequencies of of characters have been used. BIBREF20 also used the relative frequencies of two character combinations which had one unknown character between them, also known as gapped bigrams. BIBREF68 used a modified relative frequency of character unigrams and bigrams.Character trigram frequencies relative to the word count were used by BIBREF92 , who calculated the values INLINEFORM0 as in vega1. Let INLINEFORM1 be the word-tokenized segmentation of the corpus INLINEFORM2 of character tokens, then: DISPLAYFORM0 where INLINEFORM0 is the count of character trigrams INLINEFORM1 in INLINEFORM2 , and INLINEFORM3 is the total word count in the corpus. Later frequencies relative to the word count were used by BIBREF93 for character bigrams and trigrams. BIBREF25 divided characters into five phonetic groups and used a Markovian method to calculate the probability of each bigram consisting of these phonetic groups. In Markovian methods, the probability of a given character INLINEFORM0 is calculated relative to a fixed-size character context INLINEFORM1 in corpus INLINEFORM2 , as follows: DISPLAYFORM0 where INLINEFORM0 is an prefix of INLINEFORM1 of length INLINEFORM2 . In this case, the probability INLINEFORM3 is the value INLINEFORM4 , where INLINEFORM5 , in the model INLINEFORM6 . BIBREF94 used 4-grams with recognition weights which were derived from Markovian probabilities. MarkovianTable lists some of the more recent articles where Markovian character have been used. BIBREF110 was the first author to propose a full-fledged probabilistic language identifier. He defines the probability of a trigram INLINEFORM0 being written in the language INLINEFORM1 to be: DISPLAYFORM0 He considers the prior probabilities of each language INLINEFORM0 to be equal, which leads to: DISPLAYFORM0  BIBREF110 used the probabilities INLINEFORM0 as the values INLINEFORM1 in the language models. BIBREF111 used a list of the most frequent bigrams and trigrams with logarithmic weighting. BIBREF112 was the first to use direct frequencies of character as feature vectors. BIBREF113 used Principal Component Analysis (“PCA”) to select only the most discriminating bigrams in the feature vectors representing languages. BIBREF114 used the most frequent and discriminating byte unigrams, bigrams, and trigrams among their feature functions. They define the most discriminating features as those which have the most differing relative frequencies between the models of the different languages. BIBREF115 tested from two to five using frequencies as feature vectors, frequency ordered lists, relative frequencies, and Markovian probabilities. FrequencyVectorTable lists the more recent articles where the frequency of character have been used as features. In the method column, “RF” refers to Random Forest (cf. decisiontrees), “LR” to Logistic Regression (discriminantfunctions), “KRR” to Kernel Ridge Regression (vectors), “KDA” to Kernel Discriminant Analysis (vectors), and “NN” to Neural Networks (neuralnetworks). BIBREF47 used the last two and three characters of open class words. BIBREF34 used an unordered list of distinct trigrams with the simple scoring method (Simplescoring). BIBREF132 used Fisher's discriminant function to choose the 1000 most discriminating trigrams. BIBREF133 used unique 4-grams of characters with positive Decision Rules (Decisionrule). BIBREF134 used the frequencies of bi- and trigrams in words unique to a language. BIBREF135 used lists of the most frequent trigrams. BIBREF38 divided possible character bigrams into those that are commonly used in a language and to those that are not. They used the ratio of the commonly used bigrams to all observed bigrams to give a confidence score for each language. BIBREF136 used the difference between the ISO Latin-1 code values of two consecutive characters as well as two characters separated by another character, also known as gapped character bigrams. BIBREF137 used the IDF and the transition probability of trigrams. They calculated the IDF values INLINEFORM0 of trigrams INLINEFORM1 for each language INLINEFORM2 , as in artemenko1, where INLINEFORM3 is the number of trigrams INLINEFORM4 in the corpus of the language INLINEFORM5 and INLINEFORM6 is the number of languages in which the trigram INLINEFORM7 is found, where INLINEFORM8 is the language-segmented training corpus with each language as a single segment. DISPLAYFORM0  INLINEFORM0 is defined as: DISPLAYFORM0  BIBREF138 used from one to four, which were weighted with “TF-IDF” (Term Frequency–Inverse Document Frequency). TF-IDF was calculated as: DISPLAYFORM0 TF-IDF weighting or close variants have been widely used for . BIBREF139 used “CF-IOF” (Class Frequency-Inverse Overall Frequency) weighted 3- and 4-grams. BIBREF140 used the logarithm of the ratio of the counts of character bigrams and trigrams in the English and Hindi dictionaries. BIBREF141 used a feature weighting scheme based on mutual information (“MI”). They also tried weighting schemes based on the “GSS” (Galavotti, Sebastiani, and Simi) and “NGL” (Ng, Goh, and Low) coefficients, but using the MI-based weighting scheme proved the best in their evaluations when they used the sum of values method (sumvalues1). BIBREF67 used punctuation trigrams, where the first character has to be a punctuation mark (but not the other two characters). BIBREF142 used consonant bi- and trigrams which were generated from words after the vowels had been removed.The language models mentioned earlier consisted only of of the same size INLINEFORM0 . If from one to four were used, then there were four separate language models. BIBREF7 created ordered lists of the most frequent for each language. BIBREF143 used similar lists with symmetric cross-entropy. BIBREF144 used a Markovian method to calculate the probability of byte trigrams interpolated with byte unigrams. BIBREF145 created a language identifier based on character of different sizes over 281 languages, and obtained an identification accuracy of 62.8% for extremely short samples (5–9 characters). Their language identifier was used or evaluated by BIBREF146 , BIBREF147 , and BIBREF148 . BIBREF146 managed to improve the identification results by feeding the raw language distance calculations into an SVM.DifferingNgramTable3 lists recent articles where character of differing sizes have been used. “LR” in the methods column refer to Logistic Regression (maxent), “LSTM RNN” to Long Short-Term Memory Recurrent Neural Networks (neuralnetworks), and “DAN” to Deep Averaging Networks (neuralnetworks). BIBREF30 used up to the four last characters of words and calculated their relative frequencies. BIBREF149 used frequencies of 2–7-grams, normalized relative to the total number of in all the language models as well as the current language model. BIBREF60 compared the use of different sizes of in differing combinations, and found that combining of differing sizes resulted in better identification scores. BIBREF150 , BIBREF151 , BIBREF152 used mixed length domain-independent language models of byte from one to three or four.Mixed length language models were also generated by BIBREF36 and later by BIBREF153 , BIBREF101 , who used the most frequent and discriminating longer than two bytes, up to a maximum of 12 bytes, based on their weighted relative frequencies. INLINEFORM0 of the most frequent were extracted from training corpora for each language, and their relative frequencies were calculated. In the tests reported in BIBREF153 , INLINEFORM1 varied from 200 to 3,500 . Later BIBREF154 also evaluated different combinations of character as well as their combinations with words. BIBREF155 used mixed-order frequencies relative to the total number of in the language model. BIBREF61 used frequencies of from one to five and gapped 3- and 4-grams as features with an SVM. As an example, some gapped 4-grams from the word Sterneberg would be Senb, tree, enbr, and reeg. BIBREF156 used character as a backoff from Markovian word . BIBREF157 used the frequencies of word initial ranging from 3 to the length of the word minus 1. BIBREF158 used the most relevant selected using the absolute value of the Pearson correlation. BIBREF159 used only the first 10 characters from a longer word to generate the , while the rest were ignored. BIBREF160 used only those which had the highest TF-IDF scores. BIBREF43 used character weighted by means of the “BM25” (Best Match 25) weighting scheme. BIBREF161 used byte up to length 25. BIBREF61 used consonant sequences generated from words. BIBREF189 used the presence of vowel sequences as a feature with a NB classifier (see naivebayes) when distinguishing between English and transliterated Indian languages. BIBREF190 used a basic dictionary (basicdictionary) composed of the 400 most common character 4-grams. BIBREF46 and BIBREF110 used character combinations (of different sizes) that either existed in only one language or did not exist in one or more languages. BIBREF191 used the suffixes of lexical words derived from untagged corpora. BIBREF192 used prefixes and suffixes determined using linguistic knowledge of the Arabic language. BIBREF193 used suffixes and prefixes in rule-based . BIBREF134 used morphemes and morpheme trigrams (morphotactics) constructed by Creutz's algorithm BIBREF194 . BIBREF195 used prefixes and suffixes constructed by his own algorithm, which was later also used by BIBREF196 . BIBREF197 used morpheme lexicons in . BIBREF196 compared the use of morphological features with the use of variable sized character . When choosing between ten European languages, the morphological features obtained only 26.0% accuracy while the reached 82.7%. BIBREF198 lemmatized Malay words in order to get the base forms. BIBREF199 used a morphological analyzer of Arabic. BIBREF70 used morphological information from a part-of-speech (POS) tagger. BIBREF189 and BIBREF64 used manually selected suffixes as features. BIBREF200 created morphological grammars to distinguish between Croatian and Serbian. BIBREF201 used morphemes created by Morfessor, but they also used manually created morphological rules. BIBREF102 used a suffix module containing the most frequent suffixes. BIBREF202 and BIBREF159 used word suffixes as features with CRFs. BIBREF119 used an unsupervised method to learn morphological features from training data. The method collects candidate affixes from a dictionary built using the training data. If the remaining part of a word is found from the dictionary after removing a candidate affix, the candidate affix is considered to be a morpheme. BIBREF119 used 5% of the most frequent affixes in language identification. BIBREF183 used character classified into different types, which included prefixes and suffixes. PrefixSuffixTable lists some of the more recent articles where prefixes and suffixes collected from a training corpus has been used for . BIBREF206 used trigrams composed of syllables. BIBREF198 used Markovian syllable bigrams for between Malay and English. Later BIBREF207 also experimented with syllable uni- and trigrams. BIBREF114 used the most frequent as well as the most discriminating Indian script syllables, called aksharas. They used single aksharas, akshara bigrams, and akshara trigrams. Syllables would seem to be especially apt in situations where distinction needs to be made between two closely-related languages. BIBREF96 used the trigrams of non-syllable chunks that were based on MI. BIBREF198 experimented also with Markovian bigrams using both character and grapheme bigrams, but the syllable bigrams proved to work better. Graphemes in this case are the minimal units of the writing system, where a single character may be composed of several graphemes (e.g. in the case of the Hangul or Thai writing systems). Later, BIBREF207 also used grapheme uni- and trigrams. BIBREF207 achieved their best results combining word unigrams and syllable bigrams with a grapheme back-off. BIBREF208 used the MADAMIRA toolkit for D3 decliticization and then used D3-token 5-grams. D3 decliticization is a way to preprocess Arabic words presented by BIBREF209 .Graphones are sequences of characters linked to sequences of corresponding phonemes. They are automatically deduced from a bilingual corpus which consists of words and their correct pronunciations using Joint Sequence Models (“JSM”). BIBREF210 used language tags instead of phonemes when generating the graphones and then used Markovian graphone from 1 to 8 in . BIBREF211 used the position of the current word in word-level . The position of words in sentences has also been used as a feature in code-switching detection by BIBREF52 . It had predictive power greater than the language label or length of the previous word. BIBREF18 used the characteristics of words as parts of discriminating functions. BIBREF212 used the string edit distance and overlap between the word to be identified and words in dictionaries. Similarly BIBREF140 used a modified edit distance, which considers the common spelling substitutions when Hindi is written using latin characters. BIBREF213 used the Minimum Edit Distance (“MED”).Basic dictionaries are unordered lists of words belonging to a language. Basic dictionaries do not include information about word frequency, and are independent of the dictionaries of other languages. BIBREF110 used a dictionary for as a part of his speech synthesizer. Each word in a dictionary had only one possible “language”, or pronunciation category. More recently, a basic dictionary has been used for by BIBREF214 , BIBREF52 , and BIBREF90 .Unique word dictionaries include only those words of the language, that do not belong to the other languages targeted by the language identifier. BIBREF215 used unique short words (from one to three characters) to differentiate between languages. Recently, a dictionary of unique words was used for by BIBREF116 , BIBREF216 , and BIBREF67 . BIBREF47 used exhaustive lists of function words collected from dictionaries. BIBREF217 used stop words – that is non-content or closed-class words – as a training corpus. Similarly, BIBREF218 used words from closed word classes, and BIBREF97 used lists of function words. BIBREF219 used a lexicon of Arabic words and phrases that convey modality. Common to these features is that they are determined based on linguistic knowledge. BIBREF220 used the most relevant words for each language. BIBREF221 used unique or nearly unique words. BIBREF80 used Information Gain Word-Patterns (“IG-WP”) to select the words with the highest information gain. BIBREF222 made an (unordered) list of the most common words for each language, as, more recently, did BIBREF223 , BIBREF83 , and BIBREF85 . BIBREF224 encoded the most common words to root forms with the Soundex algorithm. BIBREF225 collected the frequencies of words into feature vectors. BIBREF112 compared the use of character from 2 to 5 with the use of words. Using words resulted in better identification results than using character bigrams (test document sizes of 20, 50, 100 or 200 characters), but always worse than character 3-, 4- or 5-grams. However, the combined use of words and character 4-grams gave the best results of all tested combinations, obtaining 95.6% accuracy for 50 character sequences when choosing between 13 languages. BIBREF158 used TF-IDF scores of words to distinguish between language groups. Recently, the frequency of words has also been used for by BIBREF180 , BIBREF183 , BIBREF129 , and BIBREF142 . BIBREF226 and BIBREF227 were the first to use relative frequencies of words in . As did BIBREF112 for word frequencies, also BIBREF60 found that combining the use of character with the use of words provided the best results. His language identifier obtained 99.8% average recall for 50 character sequences for the 10 evaluated languages (choosing between the 13 languages known by the language identifier) when using character from 1 to 6 combined with words. BIBREF98 calculated the relative frequency of words over all the languages. BIBREF137 calculated the IDF of words, following the approach outlined in artemenko1. BIBREF177 calculated the Pointwise Mutual Information (“PMI”) for words and used it to group words to Chinese dialects or dialect groups. Recently, the relative frequency of words has also been used for by BIBREF184 , BIBREF148 and BIBREF91  BIBREF228 used the relative frequency of words with less than six characters. Recently, BIBREF83 also used short words, as did BIBREF45 . BIBREF229 used the relative frequency calculated from Google searches. Google was later also used by BIBREF96 and BIBREF230 . BIBREF231 created probability maps for words for German dialect identification between six dialects. In a word probability map, each predetermined geographic point has a probability for each word form. Probabilities were derived using a linguistic atlas and automatically-induced dialect lexicons. BIBREF232 used commercial spelling checkers, which utilized lexicons and morphological analyzers. The language identifier of BIBREF232 obtained 97.9% accuracy when classifying one-line texts between 11 official South African languages. BIBREF233 used the ALMORGEANA analyzer to check if the word had an analysis in modern standard Arabic. They also used sound change rules to use possible phonological variants with the analyzer. BIBREF234 used spellchecking and morphological analyzers to detect English words from Hindi–English mixed search queries. BIBREF235 used spelling checkers to distinguish between 15 languages, extending the work of BIBREF232 with dynamic model selection in order to gain better performance. BIBREF157 used a similarity count to find if mystery words were misspelled versions of words in a dictionary. BIBREF236 used an “LBG-VQ” (Linde, Buzo & Gray algorithm for Vector Quantization) approach to design a codebook for each language BIBREF237 . The codebook contained a predetermined number of codevectors. Each codeword represented the word it was generated from as well as zero or more words close to it in the vector space. BIBREF41 used the number of words in a sentence with NB. BIBREF53 and BIBREF45 used the sentence length calculated in both words and characters with several machine learning algorithms. BIBREF53 used the ratio to the total number of words of: once-occurring words, twice-occurring words, short words, long words, function words, adjectives and adverbs, personal pronouns, and question words. They also used the word-length distribution for words of 1–20 characters. BIBREF193 used at least the preceding and proceeding words with manual rules in word-level for text-to-speech synthesis. BIBREF238 used Markovian word with a Hidden Markov Model (“HMM”) tagger (othermethods). WordNgramTable lists more recent articles where word or similar constructs have been used. “PPM” in the methods column refers to Prediction by Partial Matching (smoothing), and “kNN” to INLINEFORM0 Nearest Neighbor classification (ensemble). BIBREF239 used word trigrams simultaneously with character 4-grams. He concluded that word-based models can be used to augment the results from character when they are not providing reliable identification results. WordCharacterNgramTable lists articles where both character and word have been used together. “CBOW” in the methods column refer to Continuous Bag of Words neural network (neuralnetworks), and “MIRA” to Margin Infused Relaxed Algorithm (supportvectormachines). BIBREF154 evaluated different combinations of word and character with SVMs. The best combination for language variety identification was using all the features simultaneously. BIBREF187 used normal and gapped word and character simultaneously. BIBREF240 uses word embeddings consisting of Positive Pointwise Mutual Information (“PPMI”) counts to represent each word type. Then they use Truncated Singular Value Decomposition (“TSVD”) to reduce the dimension of the word vectors to 100. BIBREF241 used INLINEFORM0 -means clustering when building dialectal Arabic corpora. BIBREF242 used features provided by Latent Semantic Analysis (“LSA”) with SVMs and NB. BIBREF243 present two models, the CBOW model and the continuous skip-gram model. The CBOW model can be used to generate a word given it's context and the skip-gram model can generate the context given a word. The projection matrix, which is the weight matrix between the input layer and the hidden layer, can be divided into vectors, one vector for each word in the vocabulary. These word-vectors are also referred to as word embeddings. The embeddings can be used as features in other tasks after the neural network has been trained. BIBREF244 , BIBREF245 , BIBREF80 , BIBREF246 , BIBREF247 , BIBREF248 , BIBREF62 , and BIBREF130 used word embeddings generated by the word2vec skip-gram model BIBREF243 as features in . BIBREF249 used word2vec word embeddings and INLINEFORM0 -means clustering. BIBREF250 , BIBREF251 , and BIBREF44 also used word embeddings created with word2vec. BIBREF167 trained both character and word embeddings using FastText text classification method BIBREF63 on the Discriminating between Similar Languages (“DSL”) 2016 shared task, where it reached low accuracy when compared with the other methods. BIBREF205 used FastText to train word vectors including subword information. Then he used these word vectors together with some additional word features to train a CRF-model which was used for codeswitching detection. BIBREF212 extracted features from the hidden layer of a Recurrent Neural Network (“RNN”) that had been trained to predict the next character in a string. They used the features with a SVM classifier. BIBREF229 evaluated methods for detecting foreign language inclusions and experimented with a Conditional Markov Model (“CMM”) tagger, which had performed well on Named Entity Recognition (“NER”). BIBREF229 was able to produce the best results by incorporating her own English inclusion classifier's decision as a feature for the tagger, and not using the taggers POS tags. BIBREF197 used syntactic parsers together with dictionaries and morpheme lexicons. BIBREF278 used composed of POS tags and function words. BIBREF173 used labels from a NER system, cluster prefixes, and Brown clusters BIBREF279 . BIBREF214 used POS tag from one to three and BIBREF43 from one to five, and BIBREF67 used POS tag trigrams with TF-IDF weighting. BIBREF203 , BIBREF42 , BIBREF53 , and BIBREF45 have also recently used POS tags. BIBREF80 used POS tags with emotion-labeled graphs in Spanish variety identification. In emotion-labeled graphs, each POS-tag was connected to one or more emotion nodes if a relationship between the original word and the emotion was found from the Spanish Emotion Lexicon. They also used POS-tags with IG-WP. BIBREF208 used the MADAMIRA tool for morphological analysis disambiguation. The polySVOX text analysis module described by BIBREF197 uses two-level rules and morpheme lexicons on sub-word level and separate definite clause grammars (DCGs) on word, sentence, and paragraph levels. The language of sub-word units, words, sentences, and paragraphs in multilingual documents is identified at the same time as performing syntactic analysis for the document. BIBREF280 converted sentences into POS-tag patterns using a word-POS dictionary for Malay. The POS-tag patterns were then used by a neural network to indicate whether the sentences were written in Malay or not. BIBREF281 used Jspell to detect differences in the grammar of Portuguese variants. BIBREF200 used a syntactic grammar to recognize verb-da-verb constructions, which are characteristic of the Serbian language. The syntactic grammar was used together with several morphological grammars to distinguish between Croatian and Serbian. BIBREF193 used the weighted scores of the words to the left and right of the word to be classified. BIBREF238 used language labels within an HMM. BIBREF282 used the language labels of other words in the same sentence to determine the language of the ambiguous word. The languages of the other words had been determined by the positive Decision Rules (Decisionrule), using dictionaries of unique words when possible. BIBREF213 , BIBREF71 used the language tags of the previous three words with an SVM. BIBREF283 used language labels of surrounding words with NB. BIBREF82 used the language probabilities of the previous word to determining weights for languages. BIBREF156 used unigram, bigram and trigram language label transition probabilities. BIBREF284 used the language labels for the two previous words as well as knowledge of whether code-switching had already been detected or not. BIBREF285 used the language label of the previous word to determine the language of an ambiguous word. BIBREF286 also used the language label of the previous word. BIBREF287 used the language identifications of 2–4 surrounding words for post-identification correction in word-level . BIBREF109 used language labels with a CRF. BIBREF52 used language labels of the current and two previous words in code-switching point prediction. Their predictive strength was lower than the count of code-switches, but better than the length or position of the word. All of the features were used together with NB, DT and SVM. BIBREF288 used language label bigrams with an HMM. BIBREF41 used the word-level language labels obtained with the approach of BIBREF289 on sentence-level dialect identification.Feature smoothing is required in order to handle the cases where not all features INLINEFORM0 in a test document have been attested in the training corpora. Thus, it is used especially when the count of features is high, or when the amount of training data is low. Smoothing is usually handled as part of the method, and not pre-calculated into the language models. Most of the smoothing methods evaluated by BIBREF290 have been used in , and we follow the order of methods in that article.In Laplace smoothing, an extra number of occurrences is added to every possible feature in the language model. BIBREF291 used Laplace's sample size correction (add-one smoothing) with the product of Markovian probabilities. BIBREF292 experimented with additive smoothing of 0.5, and noted that it was almost as good as Good-Turing smoothing. BIBREF290 calculate the values for each as: DISPLAYFORM0 where INLINEFORM0 is the probability estimate of INLINEFORM1 in the model and INLINEFORM2 its frequency in the training corpus. INLINEFORM3 is the total number of of length INLINEFORM4 and INLINEFORM5 the number of distinct in the training corpus. INLINEFORM6 is the Lidstone smoothing parameter. When using Laplace smoothing, INLINEFORM7 is equal to 1 and with Lidstone smoothing, the INLINEFORM8 is usually set to a value between 0 and 1.The penalty values used by BIBREF170 with the HeLI method function as a form of additive smoothing. BIBREF145 evaluated additive, Katz, absolute discounting, and Kneser-Ney smoothing methods. Additive smoothing produced the least accurate results of the four methods. BIBREF293 and BIBREF258 evaluated NB with several different Lidstone smoothing values. BIBREF107 used additive smoothing with character as a baseline classifier, which they were unable to beat with Convolutional Neural Networks (“CNNs”). BIBREF292 used Good-Turing smoothing with the product of Markovian probabilities. BIBREF290 define the Good-Turing smoothed count INLINEFORM0 as: DISPLAYFORM0 where INLINEFORM0 is the number of features occurring exactly INLINEFORM1 times in the corpus INLINEFORM2 . Lately Good-Turing smoothing has been used by BIBREF294 and BIBREF88 . BIBREF220 used Jelinek-Mercer smoothing correction over the relative frequencies of words, calculated as follows: DISPLAYFORM0 where INLINEFORM0 is a smoothing parameter, which is usually some small value like 0.1. BIBREF105 used character 1–8 grams with Jelinek-Mercer smoothing. Their language identifier using character 5-grams achieved 3rd place (out of 12) in the TweetLID shared task constrained track. BIBREF95 and BIBREF145 used the Katz back-off smoothing BIBREF295 from the SRILM toolkit, with perplexity. Katz smoothing is an extension of Good-Turing discounting. The probability mass left over from the discounted is then distributed over unseen via a smoothing factor. In the smoothing evaluations by BIBREF145 , Katz smoothing performed almost as well as absolute discounting, which produced the best results. BIBREF296 evaluated Witten-Bell, Katz, and absolute discounting smoothing methods. Witten-Bell got 87.7%, Katz 87.5%, and absolute discounting 87.4% accuracy with character 4-grams. BIBREF297 used the PPM-C algorithm for . PPM-C is basically a product of Markovian probabilities with an escape scheme. If an unseen context is encountered for the character being processed, the escape probability is used together with a lower-order model probability. In PPM-C, the escape probability is the sum of the seen contexts in the language model. PPM-C was lately used by BIBREF165 . The PPM-D+ algorithm was used by BIBREF298 . BIBREF299 and BIBREF300 used a PPM-A variant. BIBREF301 also used PPM. The language identifier of BIBREF301 obtained 91.4% accuracy when classifying 100 character texts between 277 languages. BIBREF302 used Witten-Bell smoothing with perplexity. BIBREF303 used a Chunk-Based Language Model (“CBLM”), which is similar to PPM models. BIBREF145 used several smoothing techniques with Markovian probabilities. Absolute discounting from the VariKN toolkit performed the best. BIBREF145 define the smoothing as follows: a constant INLINEFORM0 is subtracted from the counts INLINEFORM1 of all observed INLINEFORM2 and the held-out probability mass is distributed between the unseen in relation to the probabilities of lower order INLINEFORM3 , as follows: DISPLAYFORM0 where INLINEFORM0 is a scaling factor that makes the conditional distribution sum to one. Absolute discounting with Markovian probabilities from the VariKN toolkit was later also used by BIBREF146 , BIBREF147 , and BIBREF148 .The original Kneser-Ney smoothing is based on absolute discounting with an added back-off function to lower-order models BIBREF145 . BIBREF290 introduced a modified version of the Kneser-Ney smoothing using interpolation instead of back-off. BIBREF304 used the Markovian probabilities with Witten-Bell and modified Kneser-Ney smoothing. BIBREF88 , BIBREF166 , and BIBREF261 also recently used modified Kneser-Ney discounting. BIBREF119 used both original and modified Kneser-Ney smoothings. In the evaluations of BIBREF145 , Kneser-Ney smoothing fared better than additive, but somewhat worse than the Katz and absolute discounting smoothing. Lately BIBREF109 also used Kneser-Ney smoothing. BIBREF86 , BIBREF87 evaluated several smoothing techniques with character and word : Laplace/Lidstone, Witten-Bell, Good-Turing, and Kneser-Ney. In their evaluations, additive smoothing with 0.1 provided the best results. Good-Turing was not as good as additive smoothing, but better than Witten-Bell and Kneser-Ney smoothing. Witten-Bell proved to be clearly better than Kneser-Ney.In recent years there has been a tendency towards attempting to combine several different types of features into one classifier or classifier ensemble. Many recent studies use readily available classifier implementations and simply report how well they worked with the feature set used in the context of their study. There are many methods presented in this article that are still not available as out of the box implementations, however. There are many studies which have not been re-evaluated at all, going as far back as BIBREF18 . Our hope is that this article will inspire new studies and many previously unseen ways of combining features and methods. In the following sections, the reviewed articles are grouped by the methods used for . BIBREF46 used a positive Decision Rules with unique characters and character , that is, if a unique character or character was found, the language was identified. The positive Decision Rule (unique features) for the test document INLINEFORM0 and the training corpus INLINEFORM1 can be formulated as follows: DISPLAYFORM0 where INLINEFORM0 is the set of unique features in INLINEFORM1 , INLINEFORM2 is the corpus for language INLINEFORM3 , and INLINEFORM4 is a corpus of any other language INLINEFORM5 . Positive decision rules can also be used with non-unique features when the decisions are made in a certain order. For example, BIBREF52 presents the pseudo code for her dictionary lookup tool, where these kind of decisions are part of an if-then-else statement block. Her (manual) rule-based dictionary lookup tool works better for Dutch–English code-switching detection than the SVM, DT, or CRF methods she experiments with. The positive Decision Rule has also been used recently by BIBREF85 , BIBREF190 , BIBREF287 , BIBREF216 , BIBREF305 , BIBREF169 , and BIBREF214 .In the negative Decision Rule, if a character or character combination that was found in INLINEFORM0 does not exist in a particular language, that language is omitted from further identification. The negative Decision Rule can be expressed as: DISPLAYFORM0 where INLINEFORM0 is the corpus for language INLINEFORM1 . The negative Decision Rule was first used by BIBREF47 in . BIBREF118 evaluated the JRIP classifier from the Waikato Environment for Knowledge Analysis (“WEKA”). JRIP is an implementation of the propositional rule learner. It was found to be inferior to the SVM, NB and DT algorithms.In isolation the desicion rules tend not to scale well to larger numbers of languages (or very short test documents), and are thus mostly used in combination with other methods or as a Decision Tree. BIBREF306 were the earliest users of Decision Trees (“DT”) in . They used DT based on characters and their context without any frequency information. In training the DT, each node is split into child nodes according to an information theoretic optimization criterion. For each node a feature is chosen, which maximizes the information gain at that node. The information gain is calculated for each feature and the feature with the highest gain is selected for the node. In the identification phase, the nodes are traversed until only one language is left (leaf node). Later, BIBREF196 , BIBREF307 , and BIBREF308 have been especially successful in using DTs.Random Forest (RF) is an ensemble classifier generating many DTs. It has been succesfully used in by BIBREF140 , BIBREF201 , BIBREF309 , and BIBREF185 , BIBREF172 .In simple scoring, each feature in the test document is checked against the language model for each language, and languages which contain that feature are given a point, as follows: DISPLAYFORM0 where INLINEFORM0 is the INLINEFORM1 th feature found in the test document INLINEFORM2 . The language scoring the most points is the winner. Simple scoring is still a good alternative when facing an easy problem such as preliminary language group identification. It was recently used for this purpose by BIBREF246 with a basic dictionary. They achieved 99.8% accuracy when identifying between 6 language groups. BIBREF310 use a version of simple scoring as a distance measure, assigning a penalty value to features not found in a model. In this version, the language scoring the least amount of points is the winner. Their language identifier obtained 100% success rate with character 4-grams when classifying relatively large documents (from 1 to 3 kilobytes), between 10 languages. Simple scoring was also used lately by BIBREF166 , BIBREF311 , and BIBREF90 .The sum of values can be expressed as: DISPLAYFORM0 where INLINEFORM0 is the INLINEFORM1 th feature found in the test document INLINEFORM2 , and INLINEFORM3 is the value for the feature in the language model of the language INLINEFORM4 . The language with the highest score is the winner.The simplest case of sumvalues1 is when the text to be identified contains only one feature. An example of this is BIBREF157 who used the frequencies of short words as values in word-level identification. For longer words, he summed up the frequencies of different-sized found in the word to be identified. BIBREF210 first calculated the language corresponding to each graphone. They then summed up the predicted languages, and the language scoring the highest was the winner. When a tie occurred, they used the product of the Markovian graphone . Their method managed to outperform SVMs in their tests. BIBREF46 used the average of all the relative frequencies of the in the text to be identified. BIBREF312 evaluated several variations of the LIGA algorithm introduced by BIBREF313 . BIBREF308 and BIBREF148 also used LIGA and logLIGA methods. The average or sum of relative frequencies was also used recently by BIBREF85 and BIBREF108 . BIBREF57 summed up LFDF values (see characters), obtaining 99.75% accuracy when classifying document sized texts between four languages using Arabic script. BIBREF110 calculates the score of the language for the test document INLINEFORM0 as the average of the probability estimates of the features, as follows: DISPLAYFORM0 where INLINEFORM0 is the number of features in the test document INLINEFORM1 . BIBREF153 summed weighted relative frequencies of character , and normalized the score by dividing by the length (in characters) of the test document. Taking the average of the terms in the sums does not change the order of the scored languages, but it gives comparable results between different lengths of test documents. BIBREF92 , BIBREF314 summed up the feature weights and divided them by the number of words in the test document in order to set a threshold to detect unknown languages. Their language identifier obtained 89% precision and 94% recall when classifying documents between five languages. BIBREF192 used a weighting method combining alphabets, prefixes, suffixes and words. BIBREF233 summed up values from a word trigram ranking, basic dictionary and morphological analyzer lookup. BIBREF282 summed up language labels of the surrounding words to identify the language of the current word. BIBREF200 summed up points awarded by the presence of morphological and syntactic features. BIBREF102 used inverse rank positions as values. BIBREF158 computed the sum of keywords weighted with TF-IDF. BIBREF315 summed up the TF-IDF derived probabilities of words.The product of values can be expressed as follows: DISPLAYFORM0 where INLINEFORM0 is the INLINEFORM1 th feature found in test document INLINEFORM2 , and INLINEFORM3 is the value for the feature in the language model of language INLINEFORM4 . The language with the highest score is the winner. Some form of feature smoothing is usually required with the product of values method to avoid multiplying by zero. BIBREF26 was the first to use the product of relative frequencies and it has been widely used ever since; recent examples include BIBREF86 , BIBREF87 , BIBREF161 , and BIBREF148 . Some of the authors use a sum of log frequencies rather than a product of frequencies to avoid underflow issues over large numbers of features, but the two methods yield the same relative ordering, with the proviso that the maximum of multiplying numbers between 0 and 1 becomes the minimum of summing their negative logarithms, as can be inferred from: DISPLAYFORM0 When (multinomial) NB is used in , each feature used has a probability to indicate each language. The probabilities of all features found in the test document are multiplied for each language, and the language with the highest probability is selected, as in productvalues1. Theoretically the features are assumed to be independent of each other, but in practice using features that are functionally dependent can improve classification accuracy BIBREF316 .NB implementations have been widely used for , usually with a more varied set of features than simple character or word of the same type and length. The features are typically represented as feature vectors given to a NB classifier. BIBREF283 trained a NB classifier with language labels of surrounding words to help predict the language of ambiguous words first identified using an SVM. The language identifier used by BIBREF77 obtained 99.97% accuracy with 5-grams of characters when classifying sentence-sized texts between six language groups. BIBREF265 used a probabilistic model similar to NB. BIBREF252 used NB and naive Bayes EM, which uses the Expectation–Maximization (“EM”) algorithm in a semi-supervised setting to improve accuracy. BIBREF4 used Gaussian naive Bayes (“GNB”, i.e. NB with Gaussian estimation over continuous variables) from scikit-learn.In contrast to NB, in Bayesian networks the features are not assumed to be independent of each other. The network learns the dependencies between features in a training phase. BIBREF315 used a Bayesian Net classifier in two-staged (group first) over the open track of the DSL 2015 shared task. BIBREF130 similarly evaluated Bayesian Nets, but found them to perform worse than the other 11 algorithms they tested. BIBREF25 used the product of the Markovian probabilities of character bigrams. The language identifier created by BIBREF153 , BIBREF101 , “whatlang”, obtains 99.2% classification accuracy with smoothing for 65 character test strings, when distinguishing between 1,100 languages. The product of Markovian probabilities has recently also been used by BIBREF109 and BIBREF260 . BIBREF170 use a word-based backoff method called HeLI. Here, each language is represented by several different language models, only one of which is used for each word found in the test document. The language models for each language are: a word-level language model, and one or more models based on character of order 1– INLINEFORM0 . When a word that is not included in the word-level model is encountered in a test document, the method backs off to using character of the size INLINEFORM1 . If there is not even a partial coverage here, the method backs off to lower order and continues backing off until at least a partial coverage is obtained (potentially all the way to character unigrams). The system of BIBREF170 implementing the HeLI method attained shared first place in the closed track of the DSL 2016 shared task BIBREF317 , and was the best method tested by BIBREF148 for test documents longer than 30 characters.The well-known method of BIBREF7 uses overlapping character of varying sizes based on words. The language models are created by tokenizing the training texts for each language INLINEFORM0 into words, and then padding each word with spaces, one before and four after. Each padded word is then divided into overlapping character of sizes 1–5, and the counts of every unique are calculated over the training corpus. The are ordered by frequency and INLINEFORM1 of the most frequent , INLINEFORM2 , are used as the domain of the language model INLINEFORM3 for the language INLINEFORM4 . The rank of an INLINEFORM5 in language INLINEFORM6 is determined by the frequency in the training corpus INLINEFORM7 and denoted INLINEFORM8 .During , the test document INLINEFORM0 is treated in a similar way and a corresponding model INLINEFORM1 of the K most frequent is created. Then a distance score is calculated between the model of the test document and each of the language models. The value INLINEFORM2 is calculated as the difference in ranks between INLINEFORM3 and INLINEFORM4 of the INLINEFORM5 in the domain INLINEFORM6 of the model of the test document. If an is not found in a language model, a special penalty value INLINEFORM7 is added to the total score of the language for each missing . The penalty value should be higher than the maximum possible distance between ranks. DISPLAYFORM0 The score INLINEFORM0 for each language INLINEFORM1 is the sum of values, as in sumvalues1. The language with the lowest score INLINEFORM2 is selected as the identified language. The method is equivalent to Spearman's measure of disarray BIBREF318 . The out-of-place method has been widely used in literature as a baseline. In the evaluations of BIBREF148 for 285 languages, the out-of-place method achieved an F-score of 95% for 35-character test documents. It was the fourth best of the seven evaluated methods for test document lengths over 20 characters.Local Rank Distance BIBREF319 is a measure of difference between two strings. LRD is calculated by adding together the distances identical units (for example character ) are from each other between the two strings. The distance is only calculated within a local window of predetermined length. BIBREF122 and BIBREF320 used LRD with a Radial Basis Function (“RBF”) kernel (see RBF). For learning they experimented with both Kernel Discriminant Analysis (“KDA”) and Kernel Ridge Regression (“KRR”). BIBREF248 also used KDA. BIBREF224 calculated the Levenshtein distance between the language models and each word in the mystery text. The similary score for each language was the inverse of the sum of the Levenshtein distances. Their language identifier obtained 97.7% precision when classifying texts from two to four words between five languages. Later BIBREF216 used Levenshtein distance for Algerian dialect identification and BIBREF305 for query word identification. BIBREF321 , BIBREF322 , BIBREF323 , and BIBREF324 calculated the difference between probabilities as in Equation EQREF109 . DISPLAYFORM0 where INLINEFORM0 is the probability for the feature INLINEFORM1 in the mystery text and INLINEFORM2 the corresponding probability in the language model of the language INLINEFORM3 . The language with the lowest score INLINEFORM4 is selected as the most likely language for the mystery text. BIBREF239 , BIBREF262 used the log probability difference and the absolute log probability difference. The log probability difference proved slightly better, obtaining a precision of 94.31% using both character and word when classifying 100 character texts between 53 language-encoding pairs.Depending on the algorithm, it can be easier to view language models as vectors of weights over the target features. In the following methods, each language is represented by one or more feature vectors. Methods where each feature type is represented by only one feature vector are also sometimes referred to as centroid-based BIBREF58 or nearest prototype methods. Distance measures are generally applied to all features included in the feature vectors. BIBREF31 calculated the squared Euclidean distance between feature vectors. The Squared Euclidean distance can be calculated as: DISPLAYFORM0  BIBREF93 used the simQ similarity measure, which is closely related to the Squared Euclidean distance. BIBREF155 investigated the of multilingual documents using a Stochastic Learning Weak Estimator (“SLWE”) method. In SLWE, the document is processed one word at a time and the language of each word is identified using a feature vector representing the current word as well as the words processed so far. This feature vector includes all possible units from the language models – in their case mixed-order character from one to four. The vector is updated using the SLWE updating scheme to increase the probabilities of units found in the current word. The probabilities of units that have been found in previous words, but not in the current one, are on the other hand decreased. After processing each word, the distance of the feature vector to the probability distribution of each language is calculated, and the best-matching language is chosen as the language of the current word. Their language identifier obtained 96.0% accuracy when classifying sentences with ten words between three languages. They used the Euclidean distance as the distance measure as follows: DISPLAYFORM0  BIBREF325 compared the use of Euclidean distance with their own similarity functions. BIBREF112 calculated the cosine angle between the feature vector of the test document and the feature vectors acting as language models. This is also called the cosine similarity and is calculated as follows: DISPLAYFORM0 The method of BIBREF112 was evaluated by BIBREF326 in the context of over multilingual documents. The cosine similarity was used recently by BIBREF131 . One common trick with cosine similarity is to pre-normalise the feature vectors to unit length (e.g. BIBREF36 ), in which case the calculation takes the form of the simple dot product: DISPLAYFORM0  BIBREF60 used chi-squared distance, calculated as follows: DISPLAYFORM0  BIBREF85 compared Manhattan, Bhattacharyya, chi-squared, Canberra, Bray Curtis, histogram intersection, correlation distances, and out-of-place distances, and found the out-of-place method to be the most accurate. BIBREF239 , BIBREF262 used cross-entropy and symmetric cross-entropy. Cross-entropy is calculated as follows, where INLINEFORM0 and INLINEFORM1 are the probabilities of the feature INLINEFORM2 in the the test document INLINEFORM3 and the corpus INLINEFORM4 : DISPLAYFORM0 Symmetric cross-entropy is calculated as: DISPLAYFORM0 For cross-entropy, distribution INLINEFORM0 must be smoothed, and for symmetric cross-entropy, both probability distributions must be smoothed. Cross-entropy was used recently by BIBREF161 . BIBREF301 used a cross-entropy estimating method they call the Mean of Matching Statistics (“MMS”). In MMS every possible suffix of the mystery text INLINEFORM1 is compared to the language model of each language and the average of the lengths of the longest possible units in the language model matching the beginning of each suffix is calculated. BIBREF327 and BIBREF32 calculated the relative entropy between the language models and the test document, as follows: DISPLAYFORM0 This method is also commonly referred to as Kullback-Leibler (“KL”) distance or skew divergence. BIBREF60 compared relative entropy with the product of the relative frequencies for different-sized character , and found that relative entropy was only competitive when used with character bigrams. The product of relative frequencies gained clearly higher recall with higher-order when compared with relative entropy. BIBREF239 , BIBREF262 also used the RE and MRE measures, which are based on relative entropy. The RE measure is calculated as follows: DISPLAYFORM0 MRE is the symmetric version of the same measure. In the tests performed by BIBREF239 , BIBREF262 , the RE measure with character outperformed other tested methods obtaining 98.51% precision when classifying 100 character texts between 53 language-encoding pairs. BIBREF304 used a logistic regression (“LR”) model (also commonly referred to as “maximum entropy” within NLP), smoothed with a Gaussian prior. BIBREF328 defined LR for character-based features as follows: DISPLAYFORM0 where INLINEFORM0 is a normalization factor and INLINEFORM1 is the word count in the word-tokenized test document. BIBREF158 used an LR classifier and found it to be considerably faster than an SVM, with comparable results. Their LR classifier ranked 6 out of 9 on the closed submission track of the DSL 2015 shared task. BIBREF199 used Adaptive Logistic Regression, which automatically optimizes parameters. In recent years LR has been widely used for . BIBREF95 was the first to use perplexity for , in the manner of a language model. He calculated the perplexity for the test document INLINEFORM0 as follows: DISPLAYFORM0 DISPLAYFORM1 where INLINEFORM0 were the Katz smoothed relative frequencies of word n-grams INLINEFORM1 of the length INLINEFORM2 . BIBREF146 and BIBREF148 evaluated the best performing method used by BIBREF145 . Character n-gram based perplexity was the best method for extremely short texts in the evaluations of BIBREF148 , but for longer sequences the methods of BIBREF36 and BIBREF60 proved to be better. Lately, BIBREF182 also used perplexity. BIBREF20 used Yule's characteristic K and the Kolmogorov-Smirnov goodness of fit test to categorize languages. Kolmogorov-Smirnov proved to be the better of the two, obtaining 89% recall for 53 characters (one punch card) of text when choosing between two languages. In the goodness of fit test, the ranks of features in the models of the languages and the test document are compared. BIBREF329 experimented with Jiang and Conrath's (JC) distance BIBREF330 and Lin's similarity measure BIBREF331 , as well as the out-of-place method. They conclude that Lin's similarity measure was consistently the most accurate of the three. JC-distance measure was later evaluated by BIBREF239 , BIBREF262 , and was outperformed by the RE measure. BIBREF39 and BIBREF332 calculated special ratios from the number of trigrams in the language models when compared with the text to be identified. BIBREF333 , BIBREF334 , BIBREF335 used the quadratic discrimination score to create the feature vectors representing the languages and the test document. They then calculated the Mahalanobis distance between the languages and the test document. Their language identifier obtained 98.9% precision when classifying texts of four “screen lines” between 19 languages. BIBREF336 used odds ratio to identify the language of parts of words when identifying between two languages. Odds ratio for language INLINEFORM0 when compared with language INLINEFORM1 for morph INLINEFORM2 is calculated as in Equation EQREF127 . DISPLAYFORM0 The differences between languages can be stored in discriminant functions. The functions are then used to map the test document into an INLINEFORM0 -dimensional space. The distance of the test document to the languages known by the language identifier is calculated, and the nearest language is selected (in the manner of a nearest prototype classifier). BIBREF114 used multiple linear regression to calculate discriminant functions for two-way for Indian languages. BIBREF337 compared linear regression, NB, and LR. The precision for the three methods was very similar, with linear regression coming second in terms of precision after LR.Multiple discriminant analysis was used for by BIBREF18 . He used two functions, the first separated Finnish from English and Swedish, and the second separated English and Swedish from each other. He used Mahalanobis' INLINEFORM0 as a distance measure. BIBREF113 used Multivariate Analysis (“MVA”) with Principal Component Analysis (“PCA”) for dimensionality reduction and . BIBREF59 compared discriminant analysis with SVM and NN using characters as features, and concluded that the SVM was the best method. BIBREF40 experimented with the Winnow 2 algorithm BIBREF338 , but the method was outperformed by other methods they tested.With support vector machines (“SVMs”), a binary classifier is learned by learning a separating hyperplane between the two classes of instances which maximizes the margin between them. The simplest way to extend the basic SVM model into a multiclass classifier is via a suite of one-vs-rest classifiers, where the classifier with the highest score determines the language of the test document. One feature of SVMs that has made them particularly popular is their compatibility with kernels, whereby the separating hyperplane can be calculated via a non-linear projection of the original instance space. In the following paragraphs, we list the different kernels that have been used with SVMs for .For with SVMs, the predominant approach has been a simple linear kernel SVM model. The linear kernel model has a weight vector INLINEFORM0 and the classification of a feature vector INLINEFORM1 , representing the test document INLINEFORM2 , is calculated as follows: DISPLAYFORM0 where INLINEFORM0 is a scalar bias term. If INLINEFORM1 is equal to or greater than zero, INLINEFORM2 is categorized as INLINEFORM3 .The first to use a linear kernel SVM were BIBREF339 , and generally speaking, linear-kernel SVMs have been widely used for , with great success across a range of shared tasks. BIBREF100 were the first to apply polynomial kernel SVMs to . With a polynomial kernel INLINEFORM0 can be calculated as: DISPLAYFORM0 where INLINEFORM0 is the polynomial degree, and a hyperparameter of the model.Another popular kernel is the RBF function, also known as a Gaussian or squared exponential kernel. With an RBF kernel INLINEFORM0 is calculated as: DISPLAYFORM0 where INLINEFORM0 is a hyperparameter. BIBREF321 were the first to use an RBF kernel SVM for .With sigmoid kernel SVMs, also known as hyperbolic tangent SVMs, INLINEFORM0 can be calculated as: DISPLAYFORM0  BIBREF340 were the first to use a sigmoid kernel SVM for , followed by BIBREF341 , who found the SVM to perform better than NB, Classification And Regression Tree (“CART”), or the sum of relative frequencies.Other kernels that have been used with SVMs for include exponential kernels BIBREF178 and rational kernels BIBREF342 . BIBREF31 were the first to use SVMs for , in the form of string kernels using Ukkonen's algorithm. They used same string kernels with Euclidean distance, which did not perform as well as SVM. BIBREF87 compared SVMs with linear and on-line passive–aggressive kernels for , and found passive–aggressive kernels to perform better, but both SVMs to be inferior to NB and Log-Likelihood Ratio (sum of log-probabilities). BIBREF339 experimented with the Sequential Minimal Optimization (“SMO”) algorithm, but found a simple linear kernel SVM to perform better. BIBREF118 achieved the best results using the SMO algorithm, whereas BIBREF123 found CRFs to work better than SMO. BIBREF178 found that SMO was better than linear, exponential and polynomial kernel SVMs for Arabic tweet gender and dialect prediction.MultipleKernelSVMarticlesTable lists articles where SVMs with different kernels have been compared. BIBREF343 evaluated three different SVM approaches using datasets from different DSL shared tasks. SVM-based approaches were the top performing systems in the 2014 and 2015 shared tasks. BIBREF277 used SVMs with the Margin Infused Relaxed Algorithm, which is an incremental version of SVM training. In their evaluation, this method achieved better results than off-the-shelf . BIBREF344 was the first to use Neural Networks (“NN”) for , in the form of a simple BackPropagation Neural Network (“BPNN”) BIBREF345 with a single layer of hidden units, which is also called a multi-layer perceptron (“MLP”) model. She used words as the input features for the neural network. BIBREF346 and BIBREF347 succesfully applied MLP to . BIBREF348 , BIBREF349 and BIBREF350 used radial basis function (RBF) networks for . BIBREF351 were the first to use adaptive resonance learning (“ART”) neural networks for . BIBREF85 used Neural Text Categorizer (“NTC”: BIBREF352 ) as a baseline. NTC is an MLP-like NN using string vectors instead of number vectors. BIBREF111 were the first to use a RNN for . They concluded that RNNs are less accurate than the simple sum of logarithms of counts of character bi- or trigrams, possibly due to the relatively modestly-sized dataset they experimented with. BIBREF221 compared NNs with the out-of-place method (see sec. UID104 ). Their results show that the latter, used with bigrams and trigrams of characters, obtains clearly higher identification accuracy when dealing with test documents shorter than 400 characters.RNNs were more successfully used later by BIBREF245 who also incorporated character n-gram features in to the network architecture. BIBREF223 were the first to use a Long Short-Term Memory (“LSTM”) for BIBREF353 , and BIBREF354 was the first to use Gated Recurrent Unit networks (“GRUs”), both of which are RNN variants. BIBREF354 used byte-level representations of sentences as input for the networks. Recently, BIBREF89 and BIBREF176 also used LSTMs. Later, GRUs were successfully used for by BIBREF355 and BIBREF356 . In addition to GRUs, BIBREF354 also experimented with deep residual networks (“ResNets”) at DSL 2016.During 2016 and 2017, there was a spike in the use of convolutional neural networks (CNNs) for , most successfully by BIBREF302 and BIBREF357 . Recently, BIBREF358 combined a CNN with adversarial learning to better generalize to unseen domains, surpassing the results of BIBREF151 based on the same training regime as . BIBREF275 used CBOW NN, achieving better results over the development set of DSL 2017 than RNN-based neural networks. BIBREF62 used deep averaging networks (DANs) based on word embeddings in language variety identification. BIBREF45 used the decision table majority classifier algorithm from the WEKA toolkit in English variety detection. The bagging algorithm using DTs was the best method they tested (73.86% accuracy), followed closely by the decision table with 73.07% accuracy. BIBREF359 were the first to apply hidden Markov models (HMM) to . More recently HMMs have been used by BIBREF214 , BIBREF288 , and BIBREF261 . BIBREF360 generated aggregate Markov models, which resulted in the best results when distinguishing between six languages, obtaining 74% accuracy with text length of ten characters. BIBREF156 used an extended Markov Model (“eMM”), which is essentially a standard HMM with modified emission probabilities. Their eMM used manually optimized weights to combine four scores (products of relative frequencies) into one score. BIBREF361 used Markov logic networks BIBREF362 to predict the language used in interlinear glossed text examples contained in linguistic papers. BIBREF363 evaluated the use of unsupervised Fuzzy C Means algorithm (“FCM”) in language identification. The unsupervised algorithm was used on the training data to create document clusters. Each cluster was tagged with the language having the most documents in the cluster. Then in the identification phase, the mystery text was mapped to the closest cluster and identified with its language. A supervised centroid classifier based on cosine similarity obtained clearly better results in their experiments (93% vs. 77% accuracy). BIBREF119 and BIBREF67 evaluated the extreme gradient boosting (“XGBoost”) method BIBREF364 . BIBREF119 found that gradient boosting gave better results than RFs, while conversely, BIBREF67 found that LR gave better results than gradient boosting. BIBREF365 used compression methods for , whereby a single test document is added to the training text of each language in turn, and the language with the smallest difference (after compression) between the sizes of the original training text file and the combined training and test document files is selected as the prediction. This has obvious disadvantages in terms of real-time computational cost for prediction, but is closely related to language modeling approaches to (with the obvious difference that the language model doesn't need to be retrained multiply for each test document). In terms of compression methods, BIBREF366 experimented with Maximal Tree Machines (“MTMs”), and BIBREF367 used LZW-based compression.Very popular in text categorization and topic modeling, BIBREF368 , BIBREF23 , and BIBREF24 used Latent Dirichlet Allocation (“LDA”: BIBREF369 ) based features in classifying tweets between Arabic dialects, English, and French. Each tweet was assigned with an LDA topic, which was used as one of the features of an LR classifier. BIBREF249 used a Gaussian Process classifier with an RBF kernel in an ensemble with an LR classifier. Their ensemble achieved only ninth place in the “PAN” (Plagiarism Analysis, Authorship Identification, and Near-Duplicate Detection workshop) Author Profiling language variety shared task BIBREF370 and did not reach the results of the baseline for the task. BIBREF181 , BIBREF188 used a Passive Aggressive classifier, which proved to be almost as good as the SVMs in their evaluations between five different machine learning algorithms from the same package.Ensemble methods are meta-classification methods capable of combining several base classifiers into a combined model via a “meta-classifier” over the outputs of the base classifiers, either explicitly trained or some heuristic. It is a simple and effective approach that is used widely in machine learning to boost results beyond those of the individual base classifiers, and particularly effective when applied to large numbers of individually uncorrelated base classifiers. BIBREF20 used simple majority voting to combine classifiers using different features and methods. In majority voting, the language of the test document is identified if a majority ( INLINEFORM0 ) of the classifiers in the ensemble vote for the same language. In plurality voting, the language with most votes is chosen as in the simple scoring method (simple1). Some authors also refer to plurality voting as majority voting. BIBREF371 used majority voting in tweet . BIBREF210 used majority voting with JSM classifiers. BIBREF265 and BIBREF269 used majority voting between SVM classifiers trained with different features. BIBREF266 used majority voting to combine four classifiers: RF, random tree, SVM, and DT. BIBREF372 and BIBREF152 used majority voting between three off-the-shelf language identifiers. BIBREF104 used majority voting between perplexity-based and other classifiers. BIBREF141 used majority voting between three sum of relative frequencies-based classifiers where values were weighted with different weighting schemes. BIBREF270 , BIBREF125 , BIBREF171 , BIBREF185 , BIBREF172 , and BIBREF260 used plurality voting with SVMs. BIBREF182 used voting between several perplexity-based classifiers with different features at the 2017 DSL shared task. A voting ensemble gave better results on the closed track than a singular word-based perplexity classifier (0.9025 weighted F1-score over 0.9013), but worse results on the open track (0.9016 with ensemble and 0.9065 without).In a highest probability ensemble, the winner is simply the language which is given the highest probability by any of the individual classifiers in the ensemble. BIBREF96 used Gaussian Mixture Models (“GMM”) to give probabilities to the outputs of classifiers using different features. BIBREF372 used higher confidence between two off-the-shelf language identifiers. BIBREF265 used GMM to transform SVM prediction scores into probabilities. BIBREF270 , BIBREF125 used highest confidence over a range of base SVMs. BIBREF125 used an ensemble composed of low-dimension hash-based classifiers. According to their experiments, hashing provided up to 86% dimensionality reduction without negatively affecting performance. Their probability-based ensemble obtained 89.2% accuracy, while the voting ensemble got 88.7%. BIBREF166 combined an SVM and a LR classifier.A mean probability ensemble can be used to combine classifiers that produce probabilities (or other mutually comparable values) for languages. The average of values for each language over the classifier results is used to determine the winner and the results are equal to the sum of values method (sumvalues1). BIBREF270 evaluated several ensemble methods and found that the mean probability ensemble attained better results than plurality voting, median probability, product, highest confidence, or Borda count ensembles.In a median probability ensemble, the medians over the probabilities given by the individual classifiers are calculated for each language. BIBREF270 and BIBREF171 used a median probability rule ensemble over SVM classifiers. Consistent with the results of BIBREF270 , BIBREF171 found that a mean ensemble was better than a median ensemble, attaining 68% accuracy vs. 67% for the median ensemble.A product rule ensemble takes the probabilities for the base classifiers and calculates their product (or sum of the log probabilities), with the effect of penalising any language where there is a particularly low probability from any of the base classifiers. BIBREF210 used log probability voting with JSM classifiers. BIBREF210 observed a small increase in average accuracy using the product ensemble over a majority voting ensemble.In a INLINEFORM0 -best ensemble, several models are created for each language INLINEFORM1 by partitioning the corpus INLINEFORM2 into separate samples. The score INLINEFORM3 is calculated for each model. For each language, plurality voting is then applied to the INLINEFORM4 models with the best scores to predict the language of the test document INLINEFORM5 . BIBREF349 evaluated INLINEFORM6 -best with INLINEFORM7 based on several similarity measures. BIBREF54 compared INLINEFORM8 and INLINEFORM9 and concluded that there was no major difference in accuracy when distinguishing between six languages (100 character test set). BIBREF373 experimented with INLINEFORM10 -best classifiers, but they gave clearly worse results than the other classifiers they evaluated. BIBREF212 used INLINEFORM11 -best in two phases, first selecting INLINEFORM12 closest neighbors with simple similarity, and then using INLINEFORM13 with a more advanced similarity ranking.In bagging, independent samples of the training data are generated by random sampling with replacement, individual classifiers are trained over each such training data sample, and the final classification is determined by plurality voting. BIBREF67 evaluated the use of bagging with an LR classifier in PAN 2017 language variety identification shared task, however, bagging did not improve the accuracy in the 10-fold cross-validation experiments on the training set. BIBREF374 used bagging with word convolutional neural networks (“W-CNN”). BIBREF45 used bagging with DTs in English national variety detection and found DT-based bagging to be the best evaluated method when all 60 different features (a wide selection of formal, POS, lexicon-based, and data-based features) were used, attaining 73.86% accuracy. BIBREF45 continued the experiments using the ReliefF feature selection algorithm from the WEKA toolkit to select the most efficient features, and achieved 77.32% accuracy over the reduced feature set using a NB classifier. BIBREF130 evaluated the Rotation Forest meta classifier for DTs. The method randomly splits the used features into a pre-determined number of subsets and then uses PCA for each subset. It obtained 66.6% accuracy, attaining fifth place among the twelve methods evaluated.The AdaBoost algorithm BIBREF375 examines the performance of the base classifiers on the evaluation set and iteratively boosts the significance of misclassified training instances, with a restart mechanism to avoid local minima. AdaBoost was the best of the five machine learning techniques evaluated by BIBREF53 , faring better than C4.5, NB, RF, and linear SVM. BIBREF130 used the LogitBoost variation of AdaBoost. It obtained 67.0% accuracy, attaining third place among the twelve methods evaluated.In stacking, a higher level classifier is explicitly trained on the output of several base classifiers. BIBREF96 used AdaBoost.ECC and CART to combine classifiers using different features. More recently, BIBREF127 used LR to combine the results of five RNNs. As an ensemble they produced better results than NB and LR, which were better than the individual RNNs. Also in 2017, BIBREF185 , BIBREF172 used RF to combine several linear SVMs with different features. The system used by BIBREF172 ranked first in the German dialect identification shared task, and the system by BIBREF185 came second (71.65% accuracy) in the Arabic dialect identification shared task.In the previous two sections, we have alluded to issues of evaluation in research to date. In this section, we examine the literature more closely, providing a broad overview of the evaluation metrics that have been used, as well as the experimental settings in which research has been evaluated.The most common approach is to treat the task as a document-level classification problem. Given a set of evaluation documents, each having a known correct label from a closed set of labels (often referred to as the “gold-standard”), and a predicted label for each document from the same set, the document-level accuracy is the proportion of documents that are correctly labeled over the entire evaluation collection. This is the most frequently reported metric and conveys the same information as the error rate, which is simply the proportion of documents that are incorrectly labeled (i.e. INLINEFORM0 ).Authors sometimes provide a per-language breakdown of results. There are two distinct ways in which results are generally summarized per-language: (1) precision, in which documents are grouped according to their predicted language; and (2) recall, in which documents are grouped according to what language they are actually written in. Earlier work has tended to only provide a breakdown based on the correct label (i.e. only reporting per-language recall). This gives us a sense of how likely a do

Q: what evaluation methods are discussed?

Q: what are the off-the-shelf systems discussed in the paper?
A: Answer with content missing: (Names of many identifiers missing) TextCat, ChromeCLD, LangDetect, langid.py, whatlang, whatthelang, YALI, LDIG, Polyglot 3000, Lextek Language Identifier and Open Xerox Language Identifier.