The types of models as well as the sources of training data used in the literature are diverse, and work to date has not compared and evaluated these in a systematic manner, making it difficult to draw broader conclusions about what the “best” method for actually is. We have attempted to synthesize results to date to identify a set of “best practices”, but these should be treated as guidelines and should always be considered in the broader context of a target application. Existing work on serves to illustrate that the scope and depth of the problem are much greater than they may first appear. In openissues, we discussed open issues in , identifying the key challenges, and outlining opportunities for future research. Far from being a solved problem, aspects of make it an archetypal learning task with subtleties that could be tackled by future work on supervised learning, representation learning, multi-task learning, domain adaptation, multi-label classification and other subfields of machine learning. We hope that this paper can serve as a reference point for future work in the area, both for providing insight into work to date, as well as pointing towards the key aspects that merit further investigation.

We examine the methods for evaluating the quality of the systems (evaluation) as well as the areas where has been applied (applications), and then provide an overview of “off-the-shelf” systems (ots). We conclude the survey with a discussion of the open issues in (openissues), enumerating issues and existing efforts to address them, as well as charting the main directions where further research in is required.

The role of the program implementing his method was to route documents to MT systems, and Beesley's paper more clearly describes what has later come to be known as a character model. The fact that the distribution of characters is relatively consistent for a given language was already well known. The highest-cited early work on automatic is  . Cavnar and Trenkle's method (which we describe in detail in outofplace) builds up per-document and per-language profiles, and classifies a document according to which language profile it is most similar to, using a rank-order similarity metric. They evaluate their system on 3478 documents in eight languages obtained from USENET newsgroups, reporting a best overall accuracy of 99.8%. Gertjan van Noord produced an implementation of the method of Cavnar and Trenkle named , which has become eponymous with the method itself. is packaged with pre-trained models for a number of languages, and so it is likely that the strong results reported by Cavnar and Trenkle, combined with the ready availability of an “off-the-shelf” implementation, has resulted in the exceptional popularity of this particular method.

Corpora Used for Evaluation	As discussed in standardevaluation, the objective comparison of different methods for is difficult due to the variation in the data that different authors have used to evaluate methods.  emphasize this by demonstrating how the performance of a system can vary according to the data used for evaluation. This implies that comparisons of results reported by different authors may not be meaningful, as a strong result in one paper may not translate into a strong result on the dataset used in a different paper. In other areas of research, authors have proposed standardized corpora to allow for the objective comparison of different methods. Some authors have released datasets to accompany their work, to allow for direct replication of their experiments and encourage comparison and standardization. datasets lists a number of datasets that have been released to accompany specific publications. In this list, we only include corpora that were prepared specifically for research, and that include the full text of documents. Corpora of language-labelled Twitter messages that only provide document identifiers are also available, but reproducing the full original corpus is always an issue as the original Twitter messages are deleted or otherwise made unavailable.

Automatic Language Identification in Texts: A Survey	Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

Standardized Evaluation for 	The most common approach is to treat the task as a document-level classification problem. Given a set of evaluation documents, each having a known correct label from a closed set of labels (often referred to as the “gold-standard”), and a predicted label for each document from the same set, the document-level accuracy is the proportion of documents that are correctly labeled over the entire evaluation collection. This is the most frequently reported metric and conveys the same information as the error rate, which is simply the proportion of documents that are incorrectly labeled (i.e. INLINEFORM0 ). Authors sometimes provide a per-language breakdown of results. There are two distinct ways in which results are generally summarized per-language: (1) precision, in which documents are grouped according to their predicted language; and (2) recall, in which documents are grouped according to what language they are actually written in. Earlier work has tended to only provide a breakdown based on the correct label (i.e. only reporting per-language recall). This gives us a sense of how likely a document in any given language is to be classified correctly, but does not give an indication of how likely a prediction for a given language is of being correct.

for closely-related languages, language varieties, and dialects has been studied for Malay–Indonesian  , Indian languages  , South Slavic languages  ,  ,  ,  , Serbo-Croatian dialects  , English varieties  ,  , Dutch–Flemish  , Dutch dialects (including a temporal dimension)  , German Dialects  Mainland–Singaporean–Taiwanese Chinese  , Portuguese varieties  ,  , Spanish varieties  ,  , French varieties  ,  ,  , languages of the Iberian Peninsula  , Romanian dialects  , and Arabic dialects  ,  ,  ,  ,  , the last of which we discuss in more detail in this section. As to off-the-shelf tools which can identify closely-related languages,  released a system trained to identify 27 languages, including 10 language varieties. Closely-related languages, language varieties, and dialects have also been the focus of a number of shared tasks in recent years as discussed in evaluation:sharedtasks. Similar languages are a known problem for existing language identifiers  ,  .

On the Equivalence of Methods	The theoretical description of some of the methods leaves room for interpretation on how to implement them.  define an algorithm to be any well-defined computational procedure.  introduces a three-tiered classification where programs implement algorithms and algorithms implement functions. The examples of functions given by  , sort and find max differ from our identify language as they are always solvable and produce the same results. In this survey, we have considered two methods to be the same if they always produce exactly the same results from exactly the same inputs. This would not be in line with the definition of an algorithm by  , as in his example there are two different algorithms mergesort and quicksort that implement the function sort, always producing identical results with the same input. What we in this survey call a method, is actually a function in the tiers presented by  .

This research was supported in part by the Australian Research Council, the Kone Foundation and the Academy of Finland. We would like to thank Kimmo Koskenniemi for many valuable discussions and comments concerning the early phases of the features and the methods sections.

Furthermore, calculating the parameters for the virtual mixed languages becomes infeasibly complex for mixtures of more than 3 languages. As mentioned previously,  propose an LDA-inspired method for multilingual documents that is able to identify that a document is multilingual, identify the languages present and estimate the relative proportions of the document written in each language. To remove the need to specify the number of topics (or in this case, languages) in advance,  use a greedy heuristic that attempts to find the subset of languages that maximizes the posterior probability of a target document. One advantage of this approach is that it is not constrained to 3-language combinations like the method of  . Language set identification has also been considered by  ,  , and  ,  . To encourage further research on for multilingual documents, in the aforementioned shared task hosted by the Australiasian Language Technology Workshop 2010, discussed in evaluation:sharedtasks, participants were required to predict the language(s) present in a held-out test set containing monolingual and bilingual documents  . The dataset was prepared using data from Wikipedia, and bilingual documents were produced using a segment from an article in one language and a segment from the equivalent article in another language.

Application Areas	There are various reasons to investigate . Studies in approach the task from different perspectives, and with different motivations and application goals in mind. In this section, we briefly summarize what these motivations are, and how their specific needs differ. The oldest motivation for automatic is perhaps in conjunction with translation  . Automatic is used as a pre-processing step to determine what translation model to apply to an input text, whether it be by routing to a specific human translator or by applying MT. Such a use case is still very common, and can be seen in the Google Chrome web browser, where an built-in module is used to offer MT services to the user when the detected language of the web page being visited differs from the user's language settings. NLP components such as POS taggers and parsers tend to make a strong assumption that the input text is monolingual in a given language. Similarly to the translation case, can play an obvious role in routing documents written in different languages to NLP components tailored to those languages. More subtle is the case of documents with mixed multilingual content, the most commonly-occurring instance of which is foreign inclusion, where a document is predominantly in a single language (e.g.

An interesting and unexplored question is whether it is possible to detect that documents in different encodings or scripts are written in the same language, or what language a text is transliterated from, without any a-priori knowledge of the encoding or scripts used. One possible approach to this could be to take advantage of standard orderings of alphabets in a language – the pattern of differences between adjacent characters should be consistent across encodings, though whether this is characteristic of any given language requires exploration.

Research Directions and Open Issues in 	Several papers have catalogued open issues in  ,  ,  ,  ,  ,  ,  . Some of the issues, such as text representation (features) and choice of algorithm (methods), have already been covered in detail in this survey. In this section, we synthesize the remaining issues into a single section, and also add new issues that have not been discussed in previous work. For each issue, we review related work and suggest promising directions for future work.

The Crúbadán project  and the Finno-Ugric Languages and the Internet project  make use of automated techniques to gather linguistic resources for under-resourced languages. Similarly, the Online Database of INterlinear text (“ODIN”:  ) uses automated as one of the steps in collecting interlinear glossed text from the web for purposes of linguistic search and bootstrapping NLP tools. One challenge in collecting linguistic resources from the web is that documents can be multilingual (i.e. contain text in more than one language). This is problematic for standard methods, which assume that a document is written in a single language, and has prompted research into segmenting text by language, as well as word-level , to enable extraction of linguistic resources from multilingual documents. A number of shared tasks discussed in detail in evaluation:sharedtasks included data from social media. Examples are the TweetLID shared task on tweet held at SEPLN 2014  ,  , the data sets used in the first and second shared tasks on in code-switched data which were partially taken from Twitter  ,  , and the third edition of the DSL shared task which contained two out-of-domain test sets consisting of tweets  .

When is used for filtering purposes, i.e. to select documents in a single language, this mislabeling can introduce substantial noise into the data extracted; furthermore, it does not matter what or how many unseen languages there are, as long as they are consistently rejected. Therefore the “unknown” output provides an adequate solution to the unseen language problem for purposes of filtering. The easiest way to implement unknown language detection is through thresholding. Most systems internally compute a score for each language for an unknown text, so thresholding can be applied either with a global threshold  , a per-language threshold  , or by comparing the score for the top-scoring INLINEFORM0 -languages. The problem of unseen languages and open-set recognition was also considered by  ,  , and  .  experiments with one-class classification (“OCC”) and reaches an F-score on 98.9 using OC-SVMs (SVMs trained only with data from one language) to discriminate between 10 languages. Another possible method for unknown language detection that has not been explored extensively in the literature, is the use of non-parametric mixture models based on Hierarchical Dirichlet Processes (“HDP”).

These outstanding challenges have led to research focused specifically on of shorter segments of text, which we discuss in more detail in openissues:short. From a practical perspective, knowing the rate at which a system can process and classify documents is useful as it allows a practitioner to predict the time required to process a document collection given certain computational resources. However, so many factors influence the rate at which documents are processed that comparison of absolute values across publications is largely meaningless. Instead, it is more valuable to consider publications that compare multiple systems under controlled conditions (same computer hardware, same evaluation data, etc.). The most common observations are that classification times between different algorithms can differ by orders of magnitude, and that the fastest methods are not always the most accurate. Beyond that, the diversity of systems tested and the variety in the test data make it difficult to draw further conclusions about the relative speed of algorithms. Where explicit feature selection is used, the number of features retained is a parameter of interest, as it affects both the memory requirements of the system and its classification rate. In general, a smaller feature set results in a faster and more lightweight identifier. Relatively few authors give specific details of the relationship between the number of features selected and accuracy.

evaluated the use of bagging with an LR classifier in PAN 2017 language variety identification shared task, however, bagging did not improve the accuracy in the 10-fold cross-validation experiments on the training set.  used bagging with word convolutional neural networks (“W-CNN”).  used bagging with DTs in English national variety detection and found DT-based bagging to be the best evaluated method when all 60 different features (a wide selection of formal, POS, lexicon-based, and data-based features) were used, attaining 73.86% accuracy.  continued the experiments using the ReliefF feature selection algorithm from the WEKA toolkit to select the most efficient features, and achieved 77.32% accuracy over the reduced feature set using a NB classifier.   evaluated the Rotation Forest meta classifier for DTs. The method randomly splits the used features into a pre-determined number of subsets and then uses PCA for each subset. It obtained 66.6% accuracy, attaining fifth place among the twelve methods evaluated. The AdaBoost algorithm  examines the performance of the base classifiers on the evaluation set and iteratively boosts the significance of misclassified training instances, with a restart mechanism to avoid local minima.

However, research aims to develop systems that are able to identify any human language, a set which numbers in the thousands  . In a broad sense, applies to any modality of language, including speech, sign language, and handwritten text, and is relevant for all means of information storage that involve language, digital or otherwise. However, in this survey we limit the scope of our discussion to of written text stored in a digitally-encoded form. Research to date on has traditionally focused on monolingual documents  (we discuss for multilingual documents in openissues:multilingual). In monolingual , the task is to assign each document a unique language label. Some work has reported near-perfect accuracy for of large documents in a small number of languages, prompting some researchers to label it a “solved task”  . However, in order to attain such accuracy, simplifying assumptions have to be made, such as the aforementioned monolinguality of each document, as well as assumptions about the type and quantity of data, and the number of languages considered. The ability to accurately detect the language that a document is written in is an enabling technology that increases accessibility of data and has a wide variety of applications.

Supporting Low-Resource Languages	  paint a fairly bleak picture of the support for low-resource languages in automatic . This is supported by the arguments of  who detail specific issues in building hugely multilingual datasets.  also specifically called for research into automatic for low-density languages. Ethnologue  lists a total of 7099 languages.  describe the Ethnologue in more detail, and discuss the role that plays in other aspects of supporting minority languages, including detecting and cataloging resources. The problem is circular: methods are typically supervised, and need training data for each language to be covered, but the most efficient way to recover such data is through methods. A number of projects are ongoing with the specific aim of gathering linguistic data from the web, targeting as broad a set of languages as possible. One such project is the aforementioned ODIN  ,  , which aims to collect parallel snippets of text from Linguistics articles published on the web. ODIN specifically targets articles containing Interlinear Glossed Text (IGT), a semi-structured format for presenting text and a corresponding gloss that is commonly used in Linguistics.

One challenge in standardizing datasets for is that the codes used to label languages are not fully standardized, and a large proportion of labeling systems only cover a minor portion of the languages used in the world today  .  discuss this problem in detail, listing different language code sets, as well as the internal structure exhibited by some of the code sets. Some standards consider certain groups of “languages” as varieties of a single macro-language, whereas others consider them to be discrete languages. An example of this is found in South Slavic languages, where some language code sets refer to Serbo-Croatian, whereas others make distinctions between Bosnian, Serbian and Croatian  . The unclear boundaries between such languages make it difficult to build a reference corpus of documents for each language, or to compare language-specific results across datasets. Another challenge in standardizing datasets for is the great deal of variation that can exist between data in the same language. We examine this in greater detail in openissues:encoding, where we discuss how the same language can use a number of different orthographies, can be digitized using a number of different encodings, and may also exist in transliterated forms.

as Text Categorization	is in some ways a special case of text categorization, and previous research has examined applying standard text categorization methods to  ,  .   provides a definition of text categorization, which can be summarized as the task of mapping a document onto a pre-determined set of classes. This is a very broad definition, and indeed one that is applicable to a wide variety of tasks, amongst which falls modern-day . The archetypal text categorization task is perhaps the classification of newswire articles according to the topics that they discuss, exemplified by the Reuters-21578 dataset  . However, has particular characteristics that make it different from typical text categorization tasks: These distinguishing characteristics present unique challenges and offer particular opportunities, so much so that research in has generally proceeded independently of text categorization research. In this survey, we will examine the common themes and ideas that underpin research in . We begin with a brief history of research that has led to modern (history), and then proceed to review the literature, first introducing the mathematical notation used in the article (notation), and then providing synthesis and analysis of existing research, focusing specifically on the representation of text (features) and the learning algorithms used (methods).

When using Laplace smoothing, INLINEFORM7 is equal to 1 and with Lidstone smoothing, the INLINEFORM8 is usually set to a value between 0 and 1. The penalty values used by  with the HeLI method function as a form of additive smoothing.  evaluated additive, Katz, absolute discounting, and Kneser-Ney smoothing methods. Additive smoothing produced the least accurate results of the four methods.  and  evaluated NB with several different Lidstone smoothing values.  used additive smoothing with character as a baseline classifier, which they were unable to beat with Convolutional Neural Networks (“CNNs”).   used Good-Turing smoothing with the product of Markovian probabilities.  define the Good-Turing smoothed count INLINEFORM0 as: DISPLAYFORM0  where INLINEFORM0 is the number of features occurring exactly INLINEFORM1 times in the corpus INLINEFORM2 . Lately Good-Turing smoothing has been used by  and  .

used an LR classifier and found it to be considerably faster than an SVM, with comparable results. Their LR classifier ranked 6 out of 9 on the closed submission track of the DSL 2015 shared task.  used Adaptive Logistic Regression, which automatically optimizes parameters. In recent years LR has been widely used for .   was the first to use perplexity for , in the manner of a language model. He calculated the perplexity for the test document INLINEFORM0 as follows: DISPLAYFORM0 DISPLAYFORM1  where INLINEFORM0 were the Katz smoothed relative frequencies of word n-grams INLINEFORM1 of the length INLINEFORM2 .  and  evaluated the best performing method used by  . Character n-gram based perplexity was the best method for extremely short texts in the evaluations of  , but for longer sequences the methods of  and  proved to be better. Lately,  also used perplexity.   used Yule's characteristic K and the Kolmogorov-Smirnov goodness of fit test to categorize languages.

Previous Surveys	Although there are some dedicated survey articles, these tend to be relatively short; there have not been any comprehensive surveys of research in automated LI of text to date. The largest survey so far can be found in the literature review of Marco Lui's PhD thesis  , which served as an early draft and starting point for the current article.  provides a historical overview of language identification focusing on the use of language models.  gives a brief overview of some of the methods used for , and  provide a review of some of the techniques and applications used previously.  gives a short overview of some of the challenges, algorithms and available tools for .  provides a brief summary of , how it relates to other research areas, and some outstanding challenges, but only does so in general terms and does not go into any detail about existing work in the area. Another brief article about is  , which covers both of spoken language as well as of written documents, and also discusses of documents stored as images rather than digitally-encoded text.

If an is not found in a language model, a special penalty value INLINEFORM7 is added to the total score of the language for each missing . The penalty value should be higher than the maximum possible distance between ranks. DISPLAYFORM0  The score INLINEFORM0 for each language INLINEFORM1 is the sum of values, as in sumvalues1. The language with the lowest score INLINEFORM2 is selected as the identified language. The method is equivalent to Spearman's measure of disarray  . The out-of-place method has been widely used in literature as a baseline. In the evaluations of  for 285 languages, the out-of-place method achieved an F-score of 95% for 35-character test documents. It was the fourth best of the seven evaluated methods for test document lengths over 20 characters. Local Rank Distance  is a measure of difference between two strings. LRD is calculated by adding together the distances identical units (for example character ) are from each other between the two strings. The distance is only calculated within a local window of predetermined length.  and  used LRD with a Radial Basis Function (“RBF”) kernel (see RBF).

A potential reason for this is that the improvement in accuracy plateaus with increasing feature count, though the exact number of features required varies substantially with the method and the data used. At the lower end of the scale,  report that 300–400 features per language is sufficient. Conversely  found that, for the same method, the best results for the evaluation set were attained with 20,000 features per language.

Methods	In recent years there has been a tendency towards attempting to combine several different types of features into one classifier or classifier ensemble. Many recent studies use readily available classifier implementations and simply report how well they worked with the feature set used in the context of their study. There are many methods presented in this article that are still not available as out of the box implementations, however. There are many studies which have not been re-evaluated at all, going as far back as  . Our hope is that this article will inspire new studies and many previously unseen ways of combining features and methods. In the following sections, the reviewed articles are grouped by the methods used for .

In datasets where the number of documents per language is the same, this again works out to being the collection-level average. However, research has frequently dealt with datasets where there is a substantial skew between classes. In such cases, the collection-level accuracy is strongly biased towards more heavily-represented languages. To address this issue, in work on skewed document collections, authors tend to report both the collection-level accuracy and the macro-averaged precision/recall/F-score, in order to give a more complete picture of the characteristics of the method being studied. Whereas the notions of macro-averaged precision and recall are clearly defined, there are two possible methods to calculate the macro-averaged F-score. The first is to calculate it as the harmonic mean of the macro-averaged precision and recall, and the second is to calculate it as the arithmetic mean of the per-class F-score. The comparability of published results is also limited by the variation in size and source of the data used for evaluation. In work to date, authors have used data from a variety of different sources to evaluate the performance of proposed solutions.

They also tried weighting schemes based on the “GSS” (Galavotti, Sebastiani, and Simi) and “NGL” (Ng, Goh, and Low) coefficients, but using the MI-based weighting scheme proved the best in their evaluations when they used the sum of values method (sumvalues1).  used punctuation trigrams, where the first character has to be a punctuation mark (but not the other two characters).  used consonant bi- and trigrams which were generated from words after the vowels had been removed. The language models mentioned earlier consisted only of of the same size INLINEFORM0 . If from one to four were used, then there were four separate language models.  created ordered lists of the most frequent for each language.  used similar lists with symmetric cross-entropy.  used a Markovian method to calculate the probability of byte trigrams interpolated with byte unigrams.  created a language identifier based on character of different sizes over 281 languages, and obtained an identification accuracy of 62.8% for extremely short samples (5–9 characters). Their language identifier was used or evaluated by  ,  , and  .

Empirical Evaluation	In the previous two sections, we have alluded to issues of evaluation in research to date. In this section, we examine the literature more closely, providing a broad overview of the evaluation metrics that have been used, as well as the experimental settings in which research has been evaluated.

present a method to detect documents that contain text in more than one language and identify the languages present with their relative proportions in the document. The method is evaluated on real-world data from a web crawl targeted to collect documents for specific low-density languages. for low-resource languages is a promising area for future work. One of the key questions that has not been clearly answered is how much data is needed to accurately model a language for purposes of . Work to date suggests that there may not be a simple answer to this question as accuracy varies according to the number and variety of languages modeled  , as well as the diversity of data available to model a specific language  .

AdaBoost was the best of the five machine learning techniques evaluated by  , faring better than C4.5, NB, RF, and linear SVM.  used the LogitBoost variation of AdaBoost. It obtained 67.0% accuracy, attaining third place among the twelve methods evaluated. In stacking, a higher level classifier is explicitly trained on the output of several base classifiers.  used AdaBoost.ECC and CART to combine classifiers using different features. More recently,  used LR to combine the results of five RNNs. As an ensemble they produced better results than NB and LR, which were better than the individual RNNs. Also in 2017,  ,  used RF to combine several linear SVMs with different features. The system used by  ranked first in the German dialect identification shared task, and the system by  came second (71.65% accuracy) in the Arabic dialect identification shared task.

report an accuracy of 99.8% over tweets (averaging 80 characters) in six European languages as opposed to the 97.9% from the original LIGA method. The LIGA and logLIGA implementations by  have comparable accuracy for six languages, but the accuracy for 285 languages (with 70 character test length) is only slightly over 60% for logLIGA and the original LIGA method is at almost 85%. Many evaluations are not directly comparable as the test sizes, language sets, and hyper-parameters differ. A particularly good example is the method of  . The original paper reports an accuracy of 99.8% over eight European languages (>300 bytes test size).  report an accuracy of 68.6% for the method over a dataset of 67 languages (500 byte test size), and  report an accuracy of over 90% for 285 languages (25 character test size). Separate to the question of the number and variety of languages included are issues regarding the quantity of training data used. A number of studies have examined the relationship between accuracy and quantity of training data through the use of learning curves.

used Jelinek-Mercer smoothing correction over the relative frequencies of words, calculated as follows: DISPLAYFORM0  where INLINEFORM0 is a smoothing parameter, which is usually some small value like 0.1.  used character 1–8 grams with Jelinek-Mercer smoothing. Their language identifier using character 5-grams achieved 3rd place (out of 12) in the TweetLID shared task constrained track.   and  used the Katz back-off smoothing  from the SRILM toolkit, with perplexity. Katz smoothing is an extension of Good-Turing discounting. The probability mass left over from the discounted is then distributed over unseen via a smoothing factor. In the smoothing evaluations by  , Katz smoothing performed almost as well as absolute discounting, which produced the best results.  evaluated Witten-Bell, Katz, and absolute discounting smoothing methods. Witten-Bell got 87.7%, Katz 87.5%, and absolute discounting 87.4% accuracy with character 4-grams.   used the PPM-C algorithm for . PPM-C is basically a product of Markovian probabilities with an escape scheme.

Feature Smoothing	Feature smoothing is required in order to handle the cases where not all features INLINEFORM0 in a test document have been attested in the training corpora. Thus, it is used especially when the count of features is high, or when the amount of training data is low. Smoothing is usually handled as part of the method, and not pre-calculated into the language models. Most of the smoothing methods evaluated by  have been used in , and we follow the order of methods in that article. In Laplace smoothing, an extra number of occurrences is added to every possible feature in the language model.  used Laplace's sample size correction (add-one smoothing) with the product of Markovian probabilities.  experimented with additive smoothing of 0.5, and noted that it was almost as good as Good-Turing smoothing.  calculate the values for each as: DISPLAYFORM0  where INLINEFORM0 is the probability estimate of INLINEFORM1 in the model and INLINEFORM2 its frequency in the training corpus. INLINEFORM3 is the total number of of length INLINEFORM4 and INLINEFORM5 the number of distinct in the training corpus. INLINEFORM6 is the Lidstone smoothing parameter.

Off-the-Shelf Language Identifiers	An “off-the-shelf” language identifier is software that is distributed with pre-trained models for a number of languages, so that a user is not required to provide training data before using the system. Such a setup is highly attractive to many end-users of automatic whose main interest is in utilizing the output of a language identifier rather than implementing and developing the technique. To this end, a number of off-the-shelf language identifiers have been released over time. Many authors have evaluated these off-the-shelf identifiers, including a recent evaluation involving 13 language identifiers which was carried out by  . In this section, we provide a brief summary of open-source or otherwise free systems that are available, as well as the key characteristics of each system. We have also included dates of when the software has been last updated as of October 2018. TextCat is the most well-known Perl implementation of the out-of-place method, it lists models for 76 languages in its off-the-shelf configuration; the program is not actively maintained.

Shared Tasks	To address specific sub-problems in , a number of shared tasks have been organized on problems such as in multilingual documents  , code-switched data  , discriminating between closely related languages  , and dialect and language variety identification in various languages  ,  ,  ,  . Shared tasks are important for because they provide datasets and standardized evaluation methods that serve as benchmarks for the community. We summarize all shared tasks organized to date in sharedtasks. Generally, datasets for shared tasks have been made publicly available after the conclusion of the task, and are a good source of standardized evaluation data. However, the shared tasks to date have tended to target specific sub-problems in , and no general, broad-coverage datasets have been compiled. Widespread interest in over closely-related languages has resulted in a number of shared tasks that specifically tackle the issue. Some tasks have focused on varieties of a specific language. For example, the DEFT2010 shared task  examined varieties of French, requiring participants to classify French documents with respect to their geographical source, in addition to the decade in which they were published.

Then in the identification phase, the mystery text was mapped to the closest cluster and identified with its language. A supervised centroid classifier based on cosine similarity obtained clearly better results in their experiments (93% vs. 77% accuracy).   and  evaluated the extreme gradient boosting (“XGBoost”) method  .  found that gradient boosting gave better results than RFs, while conversely,  found that LR gave better results than gradient boosting.   used compression methods for , whereby a single test document is added to the training text of each language in turn, and the language with the smallest difference (after compression) between the sizes of the original training text file and the combined training and test document files is selected as the prediction. This has obvious disadvantages in terms of real-time computational cost for prediction, but is closely related to language modeling approaches to (with the obvious difference that the language model doesn't need to be retrained multiply for each test document). In terms of compression methods,  experimented with Maximal Tree Machines (“MTMs”), and  used LZW-based compression.

They used same string kernels with Euclidean distance, which did not perform as well as SVM.  compared SVMs with linear and on-line passive–aggressive kernels for , and found passive–aggressive kernels to perform better, but both SVMs to be inferior to NB and Log-Likelihood Ratio (sum of log-probabilities).  experimented with the Sequential Minimal Optimization (“SMO”) algorithm, but found a simple linear kernel SVM to perform better.  achieved the best results using the SMO algorithm, whereas  found CRFs to work better than SMO.  found that SMO was better than linear, exponential and polynomial kernel SVMs for Arabic tweet gender and dialect prediction. MultipleKernelSVMarticlesTable lists articles where SVMs with different kernels have been compared.  evaluated three different SVM approaches using datasets from different DSL shared tasks. SVM-based approaches were the top performing systems in the 2014 and 2015 shared tasks.   used SVMs with the Margin Infused Relaxed Algorithm, which is an incremental version of SVM training. In their evaluation, this method achieved better results than off-the-shelf .

Sum or Average of Values	The sum of values can be expressed as: DISPLAYFORM0  where INLINEFORM0 is the INLINEFORM1 th feature found in the test document INLINEFORM2 , and INLINEFORM3 is the value for the feature in the language model of the language INLINEFORM4 . The language with the highest score is the winner. The simplest case of sumvalues1 is when the text to be identified contains only one feature. An example of this is  who used the frequencies of short words as values in word-level identification. For longer words, he summed up the frequencies of different-sized found in the word to be identified.  first calculated the language corresponding to each graphone. They then summed up the predicted languages, and the language scoring the highest was the winner. When a tie occurred, they used the product of the Markovian graphone . Their method managed to outperform SVMs in their tests.   used the average of all the relative frequencies of the in the text to be identified.  evaluated several variations of the LIGA algorithm introduced by  .  and  also used LIGA and logLIGA methods.

Orthography and Transliteration	is further complicated when we consider that some languages can be written in different orthographies (e.g. Bosnian and Serbian can be written in both Latin and Cyrillic script). Transliteration is another phenomenon that has a similar effect, whereby phonetic transcriptions in another script are produced for particular languages. These transcriptions can either be standardized and officially sanctioned, such as the use of Hanyu Pinyin for Chinese, or may also emerge irregularly and organically as in the case of arabizi for Arabic  .  identify variation in the encodings and scripts used by a given language as an open issue in , pointing out that early work tended to focus on languages written using a romanized script, and suggesting that dealing with issues of encoding and orthography adds substantial complexity to the task.  discuss the relative difficulties of discriminating between languages that vary in any combination of encoding, script and language family, and give examples of pairs of languages that fall into each category. across orthographies and transliteration is an area that has not received much attention in work to date, but presents unique and interesting challenges that are suitable targets for future research.

For both bible and UDHR data, we expect that the linguistic content is relatively grammatical and well-formed, an expectation that does not carry across to domains such as search engine queries and microblogs. Another “short text” domain where has been studied is of proper names.  identify this as an issue.  found that of names is more accurate than of generic words of equivalent length.   raise an important criticism of work on Twitter messages to date: only a small number of European languages has been considered.  expand the scope of for Twitter, covering nine languages across Cyrillic, Arabic and Devanagari scripts.  expand the evaluation further, introducing a dataset of language-labeled Twitter messages across 65 languages constructed using a semi-automatic method that leverages user identity to avoid inducing a bias in the evaluation set towards messages that existing systems are able to identify correctly.  also test a 1300-language model based on  , but find that it performs relatively poorly in the target domain due to a tendency to over-predict low-resource languages. Work has also been done on of single words in a document, where the task is to label each word in the document with a specific language.

In the negative Decision Rule, if a character or character combination that was found in INLINEFORM0 does not exist in a particular language, that language is omitted from further identification. The negative Decision Rule can be expressed as: DISPLAYFORM0  where INLINEFORM0 is the corpus for language INLINEFORM1 . The negative Decision Rule was first used by  in .   evaluated the JRIP classifier from the Waikato Environment for Knowledge Analysis (“WEKA”). JRIP is an implementation of the propositional rule learner. It was found to be inferior to the SVM, NB and DT algorithms. In isolation the desicion rules tend not to scale well to larger numbers of languages (or very short test documents), and are thus mostly used in combination with other methods or as a Decision Tree.

On Notation	This section introduces the notation used throughout this article to describe methods. We have translated the notation in the original papers to our notation, to make it easier to see the similarities and differences between the methods presented in the literature. The formulas presented could be used to implement language identifiers and re-evaluate the studies they were originally presented in. A corpus INLINEFORM0 consists of individual tokens INLINEFORM1 which may be bytes, characters or words. INLINEFORM2 is comprised of a finite sequence of individual tokens, INLINEFORM3 . The total count of individual tokens INLINEFORM4 in INLINEFORM5 is denoted by INLINEFORM6 . In a corpus INLINEFORM7 with non-overlapping segments INLINEFORM8 , each segment is referred to as INLINEFORM9 , which may be a short document or a word or some other way of segmenting the corpus. The number of segments is denoted as INLINEFORM10 . A feature INLINEFORM0 is some countable characteristic of the corpus INLINEFORM1 . When referring to the set of all features INLINEFORM2 in a corpus INLINEFORM3 , we use INLINEFORM4 , and the number of features is denoted by INLINEFORM5 .

used a Bayesian Net classifier in two-staged (group first) over the open track of the DSL 2015 shared task.  similarly evaluated Bayesian Nets, but found them to perform worse than the other 11 algorithms they tested.   used the product of the Markovian probabilities of character bigrams. The language identifier created by  ,  , “whatlang”, obtains 99.2% classification accuracy with smoothing for 65 character test strings, when distinguishing between 1,100 languages. The product of Markovian probabilities has recently also been used by  and  .   use a word-based backoff method called HeLI. Here, each language is represented by several different language models, only one of which is used for each word found in the test document. The language models for each language are: a word-level language model, and one or more models based on character of order 1– INLINEFORM0 . When a word that is not included in the word-level model is encountered in a test document, the method backs off to using character of the size INLINEFORM1 .

Other Methods	  used the decision table majority classifier algorithm from the WEKA toolkit in English variety detection. The bagging algorithm using DTs was the best method they tested (73.86% accuracy), followed closely by the decision table with 73.07% accuracy.   were the first to apply hidden Markov models (HMM) to . More recently HMMs have been used by  ,  , and  .  generated aggregate Markov models, which resulted in the best results when distinguishing between six languages, obtaining 74% accuracy with text length of ten characters.  used an extended Markov Model (“eMM”), which is essentially a standard HMM with modified emission probabilities. Their eMM used manually optimized weights to combine four scores (products of relative frequencies) into one score.  used Markov logic networks  to predict the language used in interlinear glossed text examples contained in linguistic papers.   evaluated the use of unsupervised Fuzzy C Means algorithm (“FCM”) in language identification. The unsupervised algorithm was used on the training data to create document clusters. Each cluster was tagged with the language having the most documents in the cluster.

Their probability-based ensemble obtained 89.2% accuracy, while the voting ensemble got 88.7%.  combined an SVM and a LR classifier. A mean probability ensemble can be used to combine classifiers that produce probabilities (or other mutually comparable values) for languages. The average of values for each language over the classifier results is used to determine the winner and the results are equal to the sum of values method (sumvalues1).  evaluated several ensemble methods and found that the mean probability ensemble attained better results than plurality voting, median probability, product, highest confidence, or Borda count ensembles. In a median probability ensemble, the medians over the probabilities given by the individual classifiers are calculated for each language.  and  used a median probability rule ensemble over SVM classifiers. Consistent with the results of  ,  found that a mean ensemble was better than a median ensemble, attaining 68% accuracy vs. 67% for the median ensemble. A product rule ensemble takes the probabilities for the base classifiers and calculates their product (or sum of the log probabilities), with the effect of penalising any language where there is a particularly low probability from any of the base classifiers.  used log probability voting with JSM classifiers.

Absolute discounting with Markovian probabilities from the VariKN toolkit was later also used by  ,  , and  . The original Kneser-Ney smoothing is based on absolute discounting with an added back-off function to lower-order models  .  introduced a modified version of the Kneser-Ney smoothing using interpolation instead of back-off.  used the Markovian probabilities with Witten-Bell and modified Kneser-Ney smoothing.  ,  , and  also recently used modified Kneser-Ney discounting.  used both original and modified Kneser-Ney smoothings. In the evaluations of  , Kneser-Ney smoothing fared better than additive, but somewhat worse than the Katz and absolute discounting smoothing. Lately  also used Kneser-Ney smoothing.   ,  evaluated several smoothing techniques with character and word : Laplace/Lidstone, Witten-Bell, Good-Turing, and Kneser-Ney. In their evaluations, additive smoothing with 0.1 provided the best results. Good-Turing was not as good as additive smoothing, but better than Witten-Bell and Kneser-Ney smoothing.

The probabilities of units that have been found in previous words, but not in the current one, are on the other hand decreased. After processing each word, the distance of the feature vector to the probability distribution of each language is calculated, and the best-matching language is chosen as the language of the current word. Their language identifier obtained 96.0% accuracy when classifying sentences with ten words between three languages. They used the Euclidean distance as the distance measure as follows: DISPLAYFORM0    compared the use of Euclidean distance with their own similarity functions.  calculated the cosine angle between the feature vector of the test document and the feature vectors acting as language models. This is also called the cosine similarity and is calculated as follows: DISPLAYFORM0  The method of  was evaluated by  in the context of over multilingual documents. The cosine similarity was used recently by  . One common trick with cosine similarity is to pre-normalise the feature vectors to unit length (e.g.

Kolmogorov-Smirnov proved to be the better of the two, obtaining 89% recall for 53 characters (one punch card) of text when choosing between two languages. In the goodness of fit test, the ranks of features in the models of the languages and the test document are compared.  experimented with Jiang and Conrath's (JC) distance  and Lin's similarity measure  , as well as the out-of-place method. They conclude that Lin's similarity measure was consistently the most accurate of the three. JC-distance measure was later evaluated by  ,  , and was outperformed by the RE measure.  and  calculated special ratios from the number of trigrams in the language models when compared with the text to be identified.  ,  ,  used the quadratic discrimination score to create the feature vectors representing the languages and the test document. They then calculated the Mahalanobis distance between the languages and the test document. Their language identifier obtained 98.9% precision when classifying texts of four “screen lines” between 19 languages.  used odds ratio to identify the language of parts of words when identifying between two languages.

Typically, data for a number of languages is collected from a single source, and the number of languages considered varies widely. Earlier work tended to focus on a smaller number of Western European languages. Later work has shifted focus to supporting larger numbers of languages simultaneously, with the work of  pushing the upper bound, reporting a language identifier that supports over 1300 languages. The increased size of the language set considered is partly due to the increased availability of language-labeled documents from novel sources such as Wikipedia and Twitter. This supplements existing data from translations of the Universal Declaration of Human Rights, bible translations, as well as parallel texts from MT datasets such as OPUS and SETimes, and European Government data such as JRC-Acquis. These factors have led to a shift away from proprietary datasets such as the ECI multilingual corpus that were commonly used in earlier research. As more languages are considered simultaneously, the accuracy of systems decreases. A particularly striking illustration of this is the evaluation results by  for the logLIGA method  .

evaluated methods for detecting foreign language inclusions and experimented with a Conditional Markov Model (“CMM”) tagger, which had performed well on Named Entity Recognition (“NER”).  was able to produce the best results by incorporating her own English inclusion classifier's decision as a feature for the tagger, and not using the taggers POS tags.  used syntactic parsers together with dictionaries and morpheme lexicons.  used composed of POS tags and function words.  used labels from a NER system, cluster prefixes, and Brown clusters  .  used POS tag from one to three and  from one to five, and  used POS tag trigrams with TF-IDF weighting.  ,  ,  , and  have also recently used POS tags.  used POS tags with emotion-labeled graphs in Spanish variety identification. In emotion-labeled graphs, each POS-tag was connected to one or more emotion nodes if a relationship between the original word and the emotion was found from the Spanish Emotion Lexicon. They also used POS-tags with IG-WP.  used the MADAMIRA tool for morphological analysis disambiguation.

“CBOW” in the methods column refer to Continuous Bag of Words neural network (neuralnetworks), and “MIRA” to Margin Infused Relaxed Algorithm (supportvectormachines).  evaluated different combinations of word and character with SVMs. The best combination for language variety identification was using all the features simultaneously.  used normal and gapped word and character simultaneously.   uses word embeddings consisting of Positive Pointwise Mutual Information (“PPMI”) counts to represent each word type. Then they use Truncated Singular Value Decomposition (“TSVD”) to reduce the dimension of the word vectors to 100.  used INLINEFORM0 -means clustering when building dialectal Arabic corpora.  used features provided by Latent Semantic Analysis (“LSA”) with SVMs and NB.   present two models, the CBOW model and the continuous skip-gram model. The CBOW model can be used to generate a word given it's context and the skip-gram model can generate the context given a word. The projection matrix, which is the weight matrix between the input layer and the hidden layer, can be divided into vectors, one vector for each word in the vocabulary.

observed a small increase in average accuracy using the product ensemble over a majority voting ensemble. In a INLINEFORM0 -best ensemble, several models are created for each language INLINEFORM1 by partitioning the corpus INLINEFORM2 into separate samples. The score INLINEFORM3 is calculated for each model. For each language, plurality voting is then applied to the INLINEFORM4 models with the best scores to predict the language of the test document INLINEFORM5 .  evaluated INLINEFORM6 -best with INLINEFORM7 based on several similarity measures.  compared INLINEFORM8 and INLINEFORM9 and concluded that there was no major difference in accuracy when distinguishing between six languages (100 character test set).  experimented with INLINEFORM10 -best classifiers, but they gave clearly worse results than the other classifiers they evaluated.  used INLINEFORM11 -best in two phases, first selecting INLINEFORM12 closest neighbors with simple similarity, and then using INLINEFORM13 with a more advanced similarity ranking. In bagging, independent samples of the training data are generated by random sampling with replacement, individual classifiers are trained over each such training data sample, and the final classification is determined by plurality voting.

Under the monolingual assumption (i.e. each document is written in exactly one language), this is not too much of a problem, as a false negative for one language must also be a false positive for another language, so precision and recall are closely linked. Nonetheless, authors have recently tended to explicitly provide both precision and recall for clarity. It is also common practice to report an F-score INLINEFORM0 , which is the harmonic mean of precision and recall. The F-score (also sometimes called F1-score or F-measure) was developed in IR to measure the effectiveness of retrieval with respect to a user who attaches different relative importance to precision and recall  . When used as an evaluation metric for classification tasks, it is common to place equal weight on precision and recall (hence “F1”-score, in reference to the INLINEFORM1 hyper-parameter, which equally weights precision and recall when INLINEFORM2 ). In addition to evaluating performance for each individual language, authors have also sought to convey the relationship between classification errors and specific sets of languages. Errors in systems are generally not random; rather, certain sets of languages are much more likely to be confused.

can be considered a milestone in automatic , as it popularized the use of automatic methods on character models for , and to date the method is still considered a benchmark for automatic .

If there is not even a partial coverage here, the method backs off to lower order and continues backing off until at least a partial coverage is obtained (potentially all the way to character unigrams). The system of  implementing the HeLI method attained shared first place in the closed track of the DSL 2016 shared task  , and was the best method tested by  for test documents longer than 30 characters.

Very popular in text categorization and topic modeling,  ,  , and  used Latent Dirichlet Allocation (“LDA”:  ) based features in classifying tweets between Arabic dialects, English, and French. Each tweet was assigned with an LDA topic, which was used as one of the features of an LR classifier.   used a Gaussian Process classifier with an RBF kernel in an ensemble with an LR classifier. Their ensemble achieved only ninth place in the “PAN” (Plagiarism Analysis, Authorship Identification, and Near-Duplicate Detection workshop) Author Profiling language variety shared task  and did not reach the results of the baseline for the task.   ,  used a Passive Aggressive classifier, which proved to be almost as good as the SVMs in their evaluations between five different machine learning algorithms from the same package.

Ensemble Methods	Ensemble methods are meta-classification methods capable of combining several base classifiers into a combined model via a “meta-classifier” over the outputs of the base classifiers, either explicitly trained or some heuristic. It is a simple and effective approach that is used widely in machine learning to boost results beyond those of the individual base classifiers, and particularly effective when applied to large numbers of individually uncorrelated base classifiers.   used simple majority voting to combine classifiers using different features and methods. In majority voting, the language of the test document is identified if a majority ( INLINEFORM0 ) of the classifiers in the ensemble vote for the same language. In plurality voting, the language with most votes is chosen as in the simple scoring method (simple1). Some authors also refer to plurality voting as majority voting.   used majority voting in tweet .  used majority voting with JSM classifiers.  and  used majority voting between SVM classifiers trained with different features.  used majority voting to combine four classifiers: RF, random tree, SVM, and DT.  and  used majority voting between three off-the-shelf language identifiers.  used majority voting between perplexity-based and other classifiers.

Discriminant Functions	The differences between languages can be stored in discriminant functions. The functions are then used to map the test document into an INLINEFORM0 -dimensional space. The distance of the test document to the languages known by the language identifier is calculated, and the nearest language is selected (in the manner of a nearest prototype classifier).   used multiple linear regression to calculate discriminant functions for two-way for Indian languages.  compared linear regression, NB, and LR. The precision for the three methods was very similar, with linear regression coming second in terms of precision after LR. Multiple discriminant analysis was used for by  . He used two functions, the first separated Finnish from English and Swedish, and the second separated English and Swedish from each other. He used Mahalanobis' INLINEFORM0 as a distance measure.  used Multivariate Analysis (“MVA”) with Principal Component Analysis (“PCA”) for dimensionality reduction and .  compared discriminant analysis with SVM and NN using characters as features, and concluded that the SVM was the best method.   experimented with the Winnow 2 algorithm  , but the method was outperformed by other methods they tested.

Number of Languages	Early research in tended to focus on a very limited number of languages (sometimes as few as 2). This situation has improved somewhat with many current off-the-shelf language identifiers supporting on the order of 50–100 languages (ots). The standout in this regard is  , supporting 1311 languages in its default configuration. However, evaluation of the identifier of  on a different domain found that the system suffered in terms of accuracy because it detected many languages that were not present in the test data  .   describe the construction of web crawlers specifically targeting IGT, as well as the identification of the languages represented in the IGT snippets. for thousands of languages from very small quantities of text is one of the issues that they have had to tackle. They list four specific challenges for in ODIN: (1) the large number of languages; (2) “unseen” languages that appear in the test data but not in training data; (3) short target sentences; and (4) (sometimes inconsistent) transliteration into Latin text.

Depending on the algorithm, it can be easier to view language models as vectors of weights over the target features. In the following methods, each language is represented by one or more feature vectors. Methods where each feature type is represented by only one feature vector are also sometimes referred to as centroid-based  or nearest prototype methods. Distance measures are generally applied to all features included in the feature vectors.   calculated the squared Euclidean distance between feature vectors. The Squared Euclidean distance can be calculated as: DISPLAYFORM0    used the simQ similarity measure, which is closely related to the Squared Euclidean distance.   investigated the of multilingual documents using a Stochastic Learning Weak Estimator (“SLWE”) method. In SLWE, the document is processed one word at a time and the language of each word is identified using a feature vector representing the current word as well as the words processed so far. This feature vector includes all possible units from the language models – in their case mixed-order character from one to four. The vector is updated using the SLWE updating scheme to increase the probabilities of units found in the current word.

INLINEFORM0 of the most frequent were extracted from training corpora for each language, and their relative frequencies were calculated. In the tests reported in  , INLINEFORM1 varied from 200 to 3,500 . Later  also evaluated different combinations of character as well as their combinations with words.   used mixed-order frequencies relative to the total number of in the language model.  used frequencies of from one to five and gapped 3- and 4-grams as features with an SVM. As an example, some gapped 4-grams from the word Sterneberg would be Senb, tree, enbr, and reeg.  used character as a backoff from Markovian word .  used the frequencies of word initial ranging from 3 to the length of the word minus 1.  used the most relevant selected using the absolute value of the Pearson correlation.  used only the first 10 characters from a longer word to generate the , while the rest were ignored.  used only those which had the highest TF-IDF scores.  used character weighted by means of the “BM25” (Best Match 25) weighting scheme.

Using words resulted in better identification results than using character bigrams (test document sizes of 20, 50, 100 or 200 characters), but always worse than character 3-, 4- or 5-grams. However, the combined use of words and character 4-grams gave the best results of all tested combinations, obtaining 95.6% accuracy for 50 character sequences when choosing between 13 languages.  used TF-IDF scores of words to distinguish between language groups. Recently, the frequency of words has also been used for by  ,  ,  , and  .   and  were the first to use relative frequencies of words in . As did  for word frequencies, also  found that combining the use of character with the use of words provided the best results. His language identifier obtained 99.8% average recall for 50 character sequences for the 10 evaluated languages (choosing between the 13 languages known by the language identifier) when using character from 1 to 6 combined with words.  calculated the relative frequency of words over all the languages.  calculated the IDF of words, following the approach outlined in artemenko1.

The issue of variation within a language complicates the development of standardized datasets, due to challenges in determining which variants of a language should be included. Since we have seen that the performance of systems can vary per-domain  , that research is often motivated by target applications (see applications), and that domain-specific information can be used to improve accuracy (see openissues:domainspecific), it is often unsound to use a generic dataset to develop a language identifier for a particular domain. A third challenge in standardizing datasets for is the cost of obtaining correctly-labeled data. Manual labeling of data is usually prohibitively expensive, as it requires access to native speakers of all languages that the dataset aims to include. Large quantities of raw text data are available from sources such as web crawls or Wikipedia, but this data is frequently mislabeled (e.g. most non-English Wikipedias still include some English-language documents). In constructing corpora from such resources, it is common to use some form of automatic , but this makes such corpora unsuitable for evaluation purposes as they are biased towards documents that can be correctly identified by automatic systems  .

Short Texts	of short strings is known to be challenging for existing techniques.  tested four different classification methods, and found that all have substantially lower accuracy when applied to texts of 25 characters compared with texts of 125 characters. These findings were later strengthened, for example, by  and  .   describes a method specifically targeted at short texts that augments a dictionary with an affix table, which was tested over synthetic data derived from a parallel bible corpus.  focus on messages of 5–21 characters, using language models over data drawn the from Universal Declaration of Human Rights (UDHR). We would expect that generic methods for of short texts should be effective in any domain where short texts are found, such as search engine queries or microblog messages. However,  and  both only test their systems in a single domain: bible texts in the former case, and texts from the UDHR in the latter case. Other research has shown that results do not trivially generalize across domains  , and found that in UDHR documents is relatively easy  .

“Unseen” Languages and Unsupervised 	“Unseen” languages are languages that we do not have training data for but may nonetheless be encountered by a system when applied to real-world data. Dealing with languages for which we do not have training data has been identified as an issue by  and has also been mentioned by  as a specific challenge in harvesting linguistic data from the web.  use an unlabeled training set with a labeled evaluation set for token-level code switching identification between Modern Standard Arabic (MSA) and dialectal Arabic. They utilize existing dictionaries and also a morphological analyzer for MSA, so the system is supported by extensive external knowledge sources. The possibility to use unannotated training material is nonetheless a very useful feature. Some authors have attempted to tackle the unseen language problem through attempts at unsupervised labeling of text by language.  uses an unsupervised clustering algorithm to separate a multilingual corpus into groups corresponding to languages. She uses singular value decomposition (SVD) to first identify the words that discriminate between documents and then to separate the terms into highly correlating groups.

all data from Twitter and Wikipedia is UTF-8 encoded). In practice, a disadvantage of this approach may be that some encodings are only applicable to certain languages (e.g. S-JIS for Japanese and Big5 for Chinese), so knowing that a document is in a particular encoding can provide information that would be lost if the document is transcoded to a universal encoding such as UTF-8.  used a parallel state machine to detect which encoding scheme a file could potentially have been encoded with. The knowledge of the encoding, if detected, is then used to narrow down the possible languages. Most features and methods do not make a distinction between bytes or characters, and because of this we will present feature and method descriptions in terms of characters, even if byte tokenization was actually used in the original research.

If, as a research community, we are interested in the social equitability of our systems, it is critical that we develop datasets that are truly representative of the global population, to better quantify and remove this effect. To this end,  detail a method for constructing a more representative dataset, and demonstrate the impact of training on such a dataset in terms of alleviating socio-economic bias.

identify language pairs from the same language family that also share a common script and the same encoding, as the most difficult to discriminate.  report that achieves only 45% accuracy when trained and tested on 3-way Bosnian/Serbian/Croatian dataset.  found that methods are not competitive with conventional word-based document categorization methods in distinguishing between national varieties of English.  reports that a character trigram model is able to distinguish Malay/Indonesian from English, French, German, and Dutch, but handcrafted rules are needed to distinguish between Malay and Indonesian. One kind of rule is the use of “exclusive words” that are known to occur in only one of the languages. A similar idea is used by  , in automatically learning a “blacklist” of words that have a strong negative correlation with a language – i.e. their presence implies that the text is not written in a particular language. In doing so, they achieve an overall accuracy of 98%, far surpassing the 45% of .  also adopts such “discriminative training” to make use of negative evidence in .

Let INLINEFORM1 be the word-tokenized segmentation of the corpus INLINEFORM2 of character tokens, then: DISPLAYFORM0  where INLINEFORM0 is the count of character trigrams INLINEFORM1 in INLINEFORM2 , and INLINEFORM3 is the total word count in the corpus. Later frequencies relative to the word count were used by  for character bigrams and trigrams.   divided characters into five phonetic groups and used a Markovian method to calculate the probability of each bigram consisting of these phonetic groups. In Markovian methods, the probability of a given character INLINEFORM0 is calculated relative to a fixed-size character context INLINEFORM1 in corpus INLINEFORM2 , as follows: DISPLAYFORM0  where INLINEFORM0 is an prefix of INLINEFORM1 of length INLINEFORM2 . In this case, the probability INLINEFORM3 is the value INLINEFORM4 , where INLINEFORM5 , in the model INLINEFORM6 .  used 4-grams with recognition weights which were derived from Markovian probabilities. MarkovianTable lists some of the more recent articles where Markovian character have been used.   was the first author to propose a full-fledged probabilistic language identifier.

These word-vectors are also referred to as word embeddings. The embeddings can be used as features in other tasks after the neural network has been trained.  ,  ,  ,  ,  ,  ,  , and  used word embeddings generated by the word2vec skip-gram model  as features in .  used word2vec word embeddings and INLINEFORM0 -means clustering.  ,  , and  also used word embeddings created with word2vec.   trained both character and word embeddings using FastText text classification method  on the Discriminating between Similar Languages (“DSL”) 2016 shared task, where it reached low accuracy when compared with the other methods.  used FastText to train word vectors including subword information. Then he used these word vectors together with some additional word features to train a CRF-model which was used for codeswitching detection.   extracted features from the hidden layer of a Recurrent Neural Network (“RNN”) that had been trained to predict the next character in a string. They used the features with a SVM classifier.

compared their own Inverse Class Frequency (“ICF”) method with the Arithmetic Average Centroid (“AAC”) and the Class Feature Centroid (“CFC”) feature vector updating methods. In ICF a character appearing frequently only in some language gets more positive weight for that language. The values differ from Inverse Document Frequency (“IDF”, artemenko1), as they are calculated using also the frequencies of characters in other languages. Their ICF-based vectors generally performed better than those based on AAC or CFC.  explored using the relative frequencies of characters with similar discriminating weights.  also used Mutual Information (“MI”) and chi-square weighting schemes with characters.   compared the identification results of single characters with the use of character bigrams and trigrams when classifying over 67 languages. Both bigrams and trigrams generally performed better than unigrams.  also found that the identification results from identifiers using just characters are generally worse than those using character sequences.

Work to date in this area has assumed that word tokenization can be carried out on the basis of whitespace.  explore word-level in the context of segmenting a multilingual document into monolingual segments. Other work has assumed that the languages present in the document are known in advance. Conditional random fields (“CRFs”:  ) are a sequence labeling method most often used in for labeling the language of individual words in a multilingual text. CRFs can be thought of as a finite state model with probabilistic transition probabilities optimised over pre-defined cliques. They can use any observations made from the test document as features, including language labels given by monolingual language identifiers for words.  used a CRF trained with generalized expectation criteria, and found it to be the most accurate of all methods tested (NB, LR, HMM, CRF) at word-level .  introduce a technique to estimate the parameters using only monolingual data, an important consideration as there is no readily-available collection of manually-labeled multilingual documents with word-level annotations.

), in which case the calculation takes the form of the simple dot product: DISPLAYFORM0    used chi-squared distance, calculated as follows: DISPLAYFORM0    compared Manhattan, Bhattacharyya, chi-squared, Canberra, Bray Curtis, histogram intersection, correlation distances, and out-of-place distances, and found the out-of-place method to be the most accurate.   ,  used cross-entropy and symmetric cross-entropy. Cross-entropy is calculated as follows, where INLINEFORM0 and INLINEFORM1 are the probabilities of the feature INLINEFORM2 in the the test document INLINEFORM3 and the corpus INLINEFORM4 : DISPLAYFORM0  Symmetric cross-entropy is calculated as: DISPLAYFORM0  For cross-entropy, distribution INLINEFORM0 must be smoothed, and for symmetric cross-entropy, both probability distributions must be smoothed. Cross-entropy was used recently by  .  used a cross-entropy estimating method they call the Mean of Matching Statistics (“MMS”). In MMS every possible suffix of the mystery text INLINEFORM1 is compared to the language model of each language and the average of the lengths of the longest possible units in the language model matching the beginning of each suffix is calculated.

Product of Values	The product of values can be expressed as follows: DISPLAYFORM0  where INLINEFORM0 is the INLINEFORM1 th feature found in test document INLINEFORM2 , and INLINEFORM3 is the value for the feature in the language model of language INLINEFORM4 . The language with the highest score is the winner. Some form of feature smoothing is usually required with the product of values method to avoid multiplying by zero.   was the first to use the product of relative frequencies and it has been widely used ever since; recent examples include  ,  ,  , and  . Some of the authors use a sum of log frequencies rather than a product of frequencies to avoid underflow issues over large numbers of features, but the two methods yield the same relative ordering, with the proviso that the maximum of multiplying numbers between 0 and 1 becomes the minimum of summing their negative logarithms, as can be inferred from: DISPLAYFORM0  When (multinomial) NB is used in , each feature used has a probability to indicate each language. The probabilities of all features found in the test document are multiplied for each language, and the language with the highest probability is selected, as in productvalues1.

The typical method of conveying this information is through the use of a confusion matrix, a tabulation of the distribution of (predicted language, actual language) pairs. Presenting full confusion matrices becomes problematic as the number of languages considered increases, and as a result has become relatively uncommon in work that covers a broader range of languages. Per-language results are also harder to interpret as the number of languages increases, and so it is common to present only collection-level summary statistics. There are two conventional methods for summarizing across a whole collection: (1) giving each document equal weight; and (2) giving each class (i.e. language) equal weight. (1) is referred to as a micro-average, and (2) as a macro-average. For under the monolingual assumption, micro-averaged precision and recall are the same, since each instance of a false positive for one language must also be a false negative for another language. In other words, micro-averaged precision and recall are both simply the collection-level accuracy. On the other hand, macro-averaged precision and recall give equal weight to each language.

used an unsupervised method to learn morphological features from training data. The method collects candidate affixes from a dictionary built using the training data. If the remaining part of a word is found from the dictionary after removing a candidate affix, the candidate affix is considered to be a morpheme.  used 5% of the most frequent affixes in language identification.  used character classified into different types, which included prefixes and suffixes. PrefixSuffixTable lists some of the more recent articles where prefixes and suffixes collected from a training corpus has been used for .   used trigrams composed of syllables.  used Markovian syllable bigrams for between Malay and English. Later  also experimented with syllable uni- and trigrams.  used the most frequent as well as the most discriminating Indian script syllables, called aksharas. They used single aksharas, akshara bigrams, and akshara trigrams. Syllables would seem to be especially apt in situations where distinction needs to be made between two closely-related languages.   used the trigrams of non-syllable chunks that were based on MI.

For example, the work of  is primarily focused on of spoken utterances, but makes a broader contribution in demonstrating the feasibility of on the basis of a statistical model of broad phonetic information. However, their experiments do not use actual speech data, but rather “synthetic” data in the form of phonetic transcriptions derived from written text. Another subfield of speech technology, speech synthesis, has also generated a considerable amount of research in the of text, starting from the 1980s. In speech synthesis, the need to know the source language of individual words is crucial in determining how they should be pronounced.  uses the relative frequencies of character trigrams as probabilities and determines the language of words using a Bayesian model. Church explains the method – that has since been widely used in LI – as a small part of an article concentrating on many aspects of letter stress assignment in speech synthesis, which is probably why  is usually attributed to being the one to have introduced the aforementioned method to of text. As Beesley's article concentrated solely on the problem of LI, this single focus probably enabled his research to have greater visibility.

A Brief History of 	as a task predates computational methods – the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional program for text is by  , a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible. In the early 1970s,  considered the problem of automatic . According to  and the available abstract of Nakamura's article, his language identifier was able to distinguish between 25 languages written with the Latin alphabet.

and  calculated the relative entropy between the language models and the test document, as follows: DISPLAYFORM0  This method is also commonly referred to as Kullback-Leibler (“KL”) distance or skew divergence.  compared relative entropy with the product of the relative frequencies for different-sized character , and found that relative entropy was only competitive when used with character bigrams. The product of relative frequencies gained clearly higher recall with higher-order when compared with relative entropy.   ,  also used the RE and MRE measures, which are based on relative entropy. The RE measure is calculated as follows: DISPLAYFORM0  MRE is the symmetric version of the same measure. In the tests performed by  ,  , the RE measure with character outperformed other tested methods obtaining 98.51% precision when classifying 100 character texts between 53 language-encoding pairs.   used a logistic regression (“LR”) model (also commonly referred to as “maximum entropy” within NLP), smoothed with a Gaussian prior.  defined LR for character-based features as follows: DISPLAYFORM0  where INLINEFORM0 is a normalization factor and INLINEFORM1 is the word count in the word-tokenized test document.

For example, presenting information in a user's native language has been found to be a critical factor in attracting website visitors  . Text processing techniques developed in natural language processing and Information Retrieval (“IR”) generally presuppose that the language of the input text is known, and many techniques assume that all documents are in the same language. In order to apply text processing techniques to real-world data, automatic is used to ensure that only documents in relevant languages are subjected to further processing. In information storage and retrieval, it is common to index documents in a multilingual collection by the language that they are written in, and is necessary for document collections where the languages of documents are not known a-priori, such as for data crawled from the World Wide Web. Another application of that predates computational methods is the detection of the language of a document for routing to a suitable translator. This application has become even more prominent due to the advent of Machine Translation (“MT”) methods: in order for MT to be applied to translate a document to a target language, it is generally necessary to determine the source language of the document, and this is the task of . also plays a part in providing support for the documentation and use of low-resource languages.

Detecting multilingual documents is also important for acquiring linguistic data from the web, and has applications in mining bilingual texts for statistical MT from online resources  , or to study code-switching phenomena in online communications. There has also been interest in extracting text resources for low-density languages from multilingual web pages containing both the low-density language and another language such as English. The need to handle multilingual documents has prompted researchers to revisit the granularity of . Many researchers consider document-level to be relatively easy, and that sentence-level and word-level are more suitable targets for further research. However, word-level and sentence-level tokenization are not language-independent tasks, and for some languages are substantially harder than others  .   is a language identifier that supports identification of multilingual documents. The system is based on a vector space model using cosine similarity. for multilingual documents is performed through the use of virtual mixed languages.  shows how to construct vectors representative of particular combinations of languages independent of the relative proportions, and proposes a method for choosing combinations of languages to consider for any given document. One weakness of this approach is that for exhaustive coverage, this method is factorial in the number of languages, and as such intractable for a large set of languages.

Equivalence between articles was determined using the cross-language links embedded within each Wikipedia article. The winning entry  first built monolingual models from multilingual training data, and then applied them to a chunked version of the test data, making the final prediction a function of the prediction over chunks. Another approach to handling multilingual documents is to attempt to segment them into contiguous monolingual segments. In addition to identifying the languages present, this requires identifying the locations of boundaries in the text which mark the transition from one language to another. Several methods for supervised language segmentation have been proposed.  generalized a algorithm for monolingual documents by adding a dynamic programming algorithm based on a simple Markov model of multilingual documents. More recently, multilingual algorithms have also been presented by  ,  ,  ,  , and  .

Decision Rules	  used a positive Decision Rules with unique characters and character , that is, if a unique character or character was found, the language was identified. The positive Decision Rule (unique features) for the test document INLINEFORM0 and the training corpus INLINEFORM1 can be formulated as follows: DISPLAYFORM0  where INLINEFORM0 is the set of unique features in INLINEFORM1 , INLINEFORM2 is the corpus for language INLINEFORM3 , and INLINEFORM4 is a corpus of any other language INLINEFORM5 . Positive decision rules can also be used with non-unique features when the decisions are made in a certain order. For example,  presents the pseudo code for her dictionary lookup tool, where these kind of decisions are part of an if-then-else statement block. Her (manual) rule-based dictionary lookup tool works better for Dutch–English code-switching detection than the SVM, DT, or CRF methods she experiments with. The positive Decision Rule has also been used recently by  ,  ,  ,  ,  ,  , and  .

Similarity Measures	The well-known method of  uses overlapping character of varying sizes based on words. The language models are created by tokenizing the training texts for each language INLINEFORM0 into words, and then padding each word with spaces, one before and four after. Each padded word is then divided into overlapping character of sizes 1–5, and the counts of every unique are calculated over the training corpus. The are ordered by frequency and INLINEFORM1 of the most frequent , INLINEFORM2 , are used as the domain of the language model INLINEFORM3 for the language INLINEFORM4 . The rank of an INLINEFORM5 in language INLINEFORM6 is determined by the frequency in the training corpus INLINEFORM7 and denoted INLINEFORM8 . During , the test document INLINEFORM0 is treated in a similar way and a corresponding model INLINEFORM1 of the K most frequent is created. Then a distance score is calculated between the model of the test document and each of the language models. The value INLINEFORM2 is calculated as the difference in ranks between INLINEFORM3 and INLINEFORM4 of the INLINEFORM5 in the domain INLINEFORM6 of the model of the test document.

One difference is that normalization is normally heuristic-driven, where a manually-specified set of rules is used to eliminate unwanted elements of the text, whereas domain-independent text representations are data-driven, where text from different sources is used to identify the characteristics that a language shares between different sources. Both approaches share conceptual similarities with problems such as content extraction for web pages. In essence, the aim is to isolate the components of the text that actually represent language, and suppress the components that carry other information. One application is the language-aware extraction of text strings embedded in binary files, which has been shown to perform better than conventional heuristic approaches  . Future work in this area could focus specifically on the application of language-aware techniques to content extraction, using models of language to segment documents into textual and non-textual components. Such methods could also be used to iteratively improve itself by improving the quality of training data.

decisiontrees), “LR” to Logistic Regression (discriminantfunctions), “KRR” to Kernel Ridge Regression (vectors), “KDA” to Kernel Discriminant Analysis (vectors), and “NN” to Neural Networks (neuralnetworks).   used the last two and three characters of open class words.  used an unordered list of distinct trigrams with the simple scoring method (Simplescoring).  used Fisher's discriminant function to choose the 1000 most discriminating trigrams.  used unique 4-grams of characters with positive Decision Rules (Decisionrule).  used the frequencies of bi- and trigrams in words unique to a language.  used lists of the most frequent trigrams.   divided possible character bigrams into those that are commonly used in a language and to those that are not. They used the ratio of the commonly used bigrams to all observed bigrams to give a confidence score for each language.  used the difference between the ISO Latin-1 code values of two consecutive characters as well as two characters separated by another character, also known as gapped character bigrams.   used the IDF and the transition probability of trigrams.

managed to improve the identification results by feeding the raw language distance calculations into an SVM. DifferingNgramTable3 lists recent articles where character of differing sizes have been used. “LR” in the methods column refer to Logistic Regression (maxent), “LSTM RNN” to Long Short-Term Memory Recurrent Neural Networks (neuralnetworks), and “DAN” to Deep Averaging Networks (neuralnetworks).  used up to the four last characters of words and calculated their relative frequencies.  used frequencies of 2–7-grams, normalized relative to the total number of in all the language models as well as the current language model.  compared the use of different sizes of in differing combinations, and found that combining of differing sizes resulted in better identification scores.  ,  ,  used mixed length domain-independent language models of byte from one to three or four. Mixed length language models were also generated by  and later by  ,  , who used the most frequent and discriminating longer than two bytes, up to a maximum of 12 bytes, based on their weighted relative frequencies.

Multilingual Documents	Multilingual documents are documents that contain text in more than one language. In constructing the hrWac corpus,  found that 4% of the documents they collected contained text in more than one language.  report that web pages in many languages contain formulaic strings in English that do not actually contribute to the content of the page, but may nonetheless confound attempts to identify multilingual documents. Recent research has investigated how to make use of multilingual documents from sources such as web crawls  , forum posts  , and microblog messages  . However, most methods assume that a document contains text from a single language, and so are not directly applicable to multilingual documents. Handling of multilingual documents has been named as an open research question  . Most NLP techniques presuppose monolingual input data, so inclusion of data in foreign languages introduces noise, and can degrade the performance of NLP systems. Automatic detection of multilingual documents can be used as a pre-filtering step to improve the quality of input data.

He defines the probability of a trigram INLINEFORM0 being written in the language INLINEFORM1 to be: DISPLAYFORM0  He considers the prior probabilities of each language INLINEFORM0 to be equal, which leads to: DISPLAYFORM0    used the probabilities INLINEFORM0 as the values INLINEFORM1 in the language models.   used a list of the most frequent bigrams and trigrams with logarithmic weighting.  was the first to use direct frequencies of character as feature vectors.  used Principal Component Analysis (“PCA”) to select only the most discriminating bigrams in the feature vectors representing languages.  used the most frequent and discriminating byte unigrams, bigrams, and trigrams among their feature functions. They define the most discriminating features as those which have the most differing relative frequencies between the models of the different languages.  tested from two to five using frequencies as feature vectors, frequency ordered lists, relative frequencies, and Markovian probabilities. FrequencyVectorTable lists the more recent articles where the frequency of character have been used as features. In the method column, “RF” refers to Random Forest (cf.

Neural Networks (“NN”)	  was the first to use Neural Networks (“NN”) for , in the form of a simple BackPropagation Neural Network (“BPNN”)  with a single layer of hidden units, which is also called a multi-layer perceptron (“MLP”) model. She used words as the input features for the neural network.  and  succesfully applied MLP to .   ,  and  used radial basis function (RBF) networks for .  were the first to use adaptive resonance learning (“ART”) neural networks for .  used Neural Text Categorizer (“NTC”:  ) as a baseline. NTC is an MLP-like NN using string vectors instead of number vectors.   were the first to use a RNN for . They concluded that RNNs are less accurate than the simple sum of logarithms of counts of character bi- or trigrams, possibly due to the relatively modestly-sized dataset they experimented with.  compared NNs with the out-of-place method (see sec. UID104 ). Their results show that the latter, used with bigrams and trigrams of characters, obtains clearly higher identification accuracy when dealing with test documents shorter than 400 characters.

As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual .   wrote his master's thesis “Language Identification by Statistical Analysis” for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use of text in military intelligence settings is evidenced by the recent articles of, for example,  ,  ,  , and  . As features for ,  used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic ( INLINEFORM0 ), he managed to achieve 89% accuracy over 53 characters when distinguishing between English and Spanish. His thesis actually includes the identifier program code (for the IBM System/360 Model 67 mainframe), and even the language models in printed form. Much of the earliest work on automatic was focused on identification of spoken language, or did not make a distinction between written and spoken language.

Other projects that exist with the aim of creating text corpora for under-resourced languages by crawling the web are the Crúbadán project  and SeedLing  . The Crúbadán crawler uses seed data in a target language to generate word lists that in turn are used as queries for a search engine. The returned documents are then compared with the seed resource via an automatic language identifier, which is used to eliminate false positives.  reports that corpora for over 400 languages have been built using this method. The SeedLing project crawls texts from several web sources which has resulted in a total of 1451 languages from 105 language families. According to the authors, this represents 19% of the world's languages. Much recent work on multilingual documents (openissues:multilingual) has been done with support for minority languages as a key goal. One of the common problems with gathering linguistic data from the web is that the data in the target language is often embedded in a document containing data in another language. This has spurred recent developments in text segmentation by language and word-level .

is a Java library that implements a language identifier based on a NB classifier trained over character . The software comes with pre-trained models for 53 languages, using data from Wikipedia. makes use of a range of normalization heuristics to improve the performance on particular languages, including: (1) clustering of Chinese/Japanese/Korean characters to reduce sparseness; (2) removal of “language-independent” characters, and other text normalization; and (3) normalization of Arabic characters. is a Python implementation of the method described by  , which exploits training data for the same language across multiple different sources of text to identify sequences of characters that are strongly predictive of a given language, regardless of the source of the text. This feature set is combined with a NB classifier, and is distributed with a pre-trained model for 97 languages prepared using data from 5 different text sources.  provide an empirical comparison of to , and and find that it compares favorably both in terms of accuracy and classification speed. There are also implementations of the classifier component (but not the training portion) of in Java, C, and JavaScript.   uses a vector-space model with per-feature weighting on character sequences.

TextCat is not the only example of an off-the-shelf implementation of the out-of-place method: other implementations include libtextcat with 76 language models, JTCL with 15 languages, and mguesser with 104 models for different language-encoding pairs. The main issue addressed by later implementations is classification speed: TextCat is implemented in Perl and is not optimized for speed, whereas implementations such as libtextcat and mguesser have been specifically written to be fast and efficient. whatlang-rs uses an algorithm based on character trigrams and refers the user to the  article. It comes pre-trained with 83 languages. is the language identifier embedded in the Google Chrome web browser. It uses a NB classifier, and script-specific classification strategies. assumes that all the input is in UTF-8, and assigns the responsibility of encoding detection and transcoding to the user. uses Unicode information to determine the script of the input. also implements a number of pre-processing heuristics to help boost performance on its target domain (web pages), such as stripping character sequences like .jpg. The standard implementation supports 83 languages, and an extended model is also available, that supports 160 languages.

The average or sum of relative frequencies was also used recently by  and  .   summed up LFDF values (see characters), obtaining 99.75% accuracy when classifying document sized texts between four languages using Arabic script.  calculates the score of the language for the test document INLINEFORM0 as the average of the probability estimates of the features, as follows: DISPLAYFORM0  where INLINEFORM0 is the number of features in the test document INLINEFORM1 .  summed weighted relative frequencies of character , and normalized the score by dividing by the length (in characters) of the test document. Taking the average of the terms in the sums does not change the order of the scored languages, but it gives comparable results between different lengths of test documents.   ,  summed up the feature weights and divided them by the number of words in the test document in order to set a threshold to detect unknown languages. Their language identifier obtained 89% precision and 94% recall when classifying documents between five languages.  used a weighting method combining alphabets, prefixes, suffixes and words.  summed up values from a word trigram ranking, basic dictionary and morphological analyzer lookup.

Word Combinations	  used the number of words in a sentence with NB.  and  used the sentence length calculated in both words and characters with several machine learning algorithms.   used the ratio to the total number of words of: once-occurring words, twice-occurring words, short words, long words, function words, adjectives and adverbs, personal pronouns, and question words. They also used the word-length distribution for words of 1–20 characters.   used at least the preceding and proceeding words with manual rules in word-level for text-to-speech synthesis.  used Markovian word with a Hidden Markov Model (“HMM”) tagger (othermethods). WordNgramTable lists more recent articles where word or similar constructs have been used. “PPM” in the methods column refers to Prediction by Partial Matching (smoothing), and “kNN” to INLINEFORM0 Nearest Neighbor classification (ensemble).   used word trigrams simultaneously with character 4-grams. He concluded that word-based models can be used to augment the results from character when they are not providing reliable identification results. WordCharacterNgramTable lists articles where both character and word have been used together.

present a two-pass approach to processing Turkish-Dutch bilingual documents, where the first pass labels each word independently and the second pass uses the local context of a word to further refine the predictions.  achieved 97,6% accuracy on distinguishing between the two languages using a linear-chain CRF.  are the only ones so far to use a CRF for of monolingual texts. With a CRF, they attained a higher F-score in German dialect identification than NB or an ensemble consisting of NB, CRF, and SVM. Lately CRFs were also used for by  and  .  investigate of individual words in the context of code switching. They find that smoothing of models substantially improves accuracy of a language identifier based on a NB classifier when applied to individual words.

They calculated the IDF values INLINEFORM0 of trigrams INLINEFORM1 for each language INLINEFORM2 , as in artemenko1, where INLINEFORM3 is the number of trigrams INLINEFORM4 in the corpus of the language INLINEFORM5 and INLINEFORM6 is the number of languages in which the trigram INLINEFORM7 is found, where INLINEFORM8 is the language-segmented training corpus with each language as a single segment. DISPLAYFORM0   INLINEFORM0 is defined as: DISPLAYFORM0    used from one to four, which were weighted with “TF-IDF” (Term Frequency–Inverse Document Frequency). TF-IDF was calculated as: DISPLAYFORM0  TF-IDF weighting or close variants have been widely used for .  used “CF-IOF” (Class Frequency-Inverse Overall Frequency) weighted 3- and 4-grams.   used the logarithm of the ratio of the counts of character bigrams and trigrams in the English and Hindi dictionaries.  used a feature weighting scheme based on mutual information (“MI”).

Character Combinations	In this section we consider the different combinations of characters used in the literature. Character mostly consist of all possible characters in a given encoding, but can also consist of only alphabetic or ideographic characters.   calculated the co-occurrence ratios of any two characters, as well as the ratio of consonant clusters of different sizes to the total number of consonants.  used the combination of every bigram and their counts in words.  used the proportions of question and exclamation marks to the total number of the end of sentence punctuation as features with several machine learning algorithms.   used FastText to generate character n-gram embeddings  . Neural network generated embeddings are explained in cooccurrencesofwords.   used the relative frequencies of vowels following vowels, consonants following vowels, vowels following consonants and consonants following consonants.  used vowel-consonant ratios as one of the features with Support Vector Machines (“SVMs”, supportvectormachines), Decision Trees (“DTs”, decisiontrees), and Conditional Random Fields (“CRFs”, openissues:short).   used the existence of word lengthening effects and repeated punctuation as features.

used the presence of characters repeating more than twice in a row as a feature with simple scoring (simple1).  used more complicated repetitions identified by regular expressions.  used letter and character bigram repetition with a CRF.  used the count of character sequences with three or more identical characters, using several machine learning algorithms. Character are continuous sequences of characters of length INLINEFORM0 . They can be either consecutive or overlapping. Consecutive character bigrams created from the four character sequence door are do and or, whereas the overlapping bigrams are do, oo, and or. Overlapping are most often used in the literature. Overlapping produces a greater number and variety of from the same amount of text.   was the first to use combinations of any two characters. He calculated the relative frequency of each bigram. RFTable2 lists more recent articles where relative frequencies of of characters have been used.  also used the relative frequencies of two character combinations which had one unknown character between them, also known as gapped bigrams.  used a modified relative frequency of character unigrams and bigrams. Character trigram frequencies relative to the word count were used by  , who calculated the values INLINEFORM0 as in vega1.

The general finding is that accuracy increases with more training data, though there are some authors that report an optimal amount of training data, where adding more training data decreases accuracy thereafter  . Overall, it is not clear whether there is a universal quantity of data that is “enough” for any language, rather this amount appears to be affected by the particular set of languages as well as the domain of the data. The breakdown presented by  shows that with less than 100KB per language, there are some languages where classification accuracy is near perfect, whereas there are others where it is very poor. Another aspect that is frequently reported on is how long a sample of text needs to be before its language can be correctly detected. Unsurprisingly, the general consensus is that longer samples are easier to classify correctly. There is a strong interest in classifying short segments of text, as certain applications naturally involve short text documents, such as of microblog messages or search engine queries. Another area where of texts as short as one word has been investigated is in the context of dealing with documents that contain text in more than one language, where word-level has been proposed as a possible solution (see openissues:multilingual).

was the first to use the length of words in .  used the length of full person names comprising several words. Lately, the number of characters in words has been used for by  ,  ,  , and  .  also used the length of the two preceding words.   used character frequencies as feature vectors. In a feature vector, each feature INLINEFORM0 has its own integer value. The raw frequency – also called term frequency (TF) – is calculated for each language INLINEFORM1 as: DISPLAYFORM0    was the first to use the probability of characters. He calculated the probabilities as relative frequencies, by dividing the frequency of a feature found in the corpus by the total count of features of the same type in the corpus. When the relative frequency of a feature INLINEFORM0 is used as a value, it is calculated for each language INLINEFORM1 as: DISPLAYFORM0    calculated the relative frequencies of one character prefixes, and  did the same for one character suffixes.   calculated character frequency document frequency (“LFDF”) values.

Characters	In this section, we review how individual character tokens have been used as features in .   used the formatting of numbers when distinguishing between Malay and Indonesian.  used the presence of non-alphabetic characters between the current word and the words before and after as features.  used emoticons (or emojis) in Arabic dialect identification with Naive Bayes (“NB”; see product). Non-alphabetic characters have also been used by  ,  ,  , and  .   used knowledge of alphabets to exclude languages where a language-unique character in a test document did not appear.  used alphabets collected from dictionaries to check if a word might belong to a language.  used the Unicode database to get the possible languages of individual Unicode characters. Lately, the knowledge of relevant alphabets has been used for also by  and  . Capitalization is mostly preserved when calculating character frequencies, but in contexts where it is possible to identify the orthography of a given document and where capitalization exists in the orthography, lowercasing can be used to reduce sparseness. In recent work, capitalization was used as a special feature by  ,  , and  .

Conclusions	This article has presented a comprehensive survey on language identification of digitally-encoded text. We have shown that is a rich, complex, and multi-faceted problem that has engaged a wide variety of research communities. accuracy is critical as it is often the first step in longer text processing pipelines, so errors made in will propagate and degrade the performance of later stages. Under controlled conditions, such as limiting the number of languages to a small set of Western European languages and using long, grammatical, and structured text such as government documents as training data, it is possible to achieve near-perfect accuracy. This led many researchers to consider a solved problem, as argued by  . However, becomes much harder when taking into account the peculiarities of real-world data, such as very short documents (e.g. search engine queries), non-linguistic “noise” (e.g. HTML markup), non-standard use of language (e.g. as seen in social media data), and mixed-language documents (e.g. forum posts in multilingual web forums). Modern approaches to are generally data-driven and are based on comparing new documents with models of each target language learned from data.

Domain-specific 	One approach to is to build a generic language identifier that aims to correctly identify the language of a text without any information about the source of the text. Some work has specifically targeted across multiple domains, learning characteristics of languages that are consistent between different sources of text  . However, there are often domain-specific features that are useful for identifying the language of a text. In this survey, our primary focus has been on of digitally-encoded text, using only the text itself as evidence on which to base the prediction of the language. Within a text, there can sometimes be domain-specific peculiarities that can be used for . For example,  investigates of user-to-user messages in the eBay e-commerce portal. He finds that using only the first two and last two words of a message is sufficient for identifying the language of a message.

One area where is frequently used in this regard is in linguistic corpus creation, where is used to process targeted web crawls to collect text resources for low-resource languages. A large part of the motivation for this article is the observation that lacks a “home discipline”, and as such, the literature is fragmented across a number of fields, including NLP, IR, machine learning, data mining, social medial analysis, computer science education, and systems science. This has hampered the field, in that there have been many instances of research being carried out with only partial knowledge of other work on the topic, and the myriad of published systems and datasets. Finally, it should be noted that this survey does not make a distinction between languages, language varieties, and dialects. Whatever demarcation is made between languages, varieties and dialects, a system is trained to identify the associated document classes. Of course, the more similar two classes are, the more challenging it is for a system to discriminate between them.

Training a system to discriminate between similar languages such as Croatian and Serbian  , language varieties like Brazilian and European Portuguese  , or a set of Arabic dialects  is more challenging than training systems to discriminate between, for example, Japanese and Finnish. Even so, as evidenced in this article, from a computational perspective, the algorithms and features used to discriminate between languages, language varieties, and dialects are identical.

Finally, in addition to the above-mentioned dataset of  , there are a number of notable multi-dialect corpora of Arabic: a multi-dialect corpus of broadcast speeches used in the ADI shared task  ; a multi-dialect corpus of (informal) written Arabic containing newspaper comments and Twitter data  ; a parallel corpus of 2,000 sentences in MSA, Egyptian Arabic, Tunisian Arabic, Jordanian Arabic, Palestinian Arabic, and Syrian Arabic, in addition to English  ; a corpus of sentences in 18 Arabic dialects (corresponding to 18 different Arabic-speaking countries) based on data manually sourced from web forums  ; and finally two recently compiled multi-dialect corpora containing microblog posts from Twitter  ,  . While not specifically targeted at identifying language varieties,  made the critical observation that when naively trained, systems tend to perform most poorly over language varieties from the lowest socio-economic demographics (focusing particularly on the case of English), as they tend to be most under-represented in training corpora.

compiled a dataset containing MSA, Egyptian Arabic, Gulf Arabic and Levantine Arabic, and used it to investigate three classification tasks: (1) MSA and dialectal Arabic; (2) four-way classification – MSA, Egyptian Arabic, Gulf Arabic, and Levantine Arabic; and (3) three-way classification – Egyptian Arabic, Gulf Arabic, and Levantine Arabic.   explores the use of sentence-level Arabic dialect identification as a pre-processor for MT, in customizing the selection of the MT model used to translate a given sentence to the dialect it uses. In performing dialect-specific MT, the authors achieve an improvement of 1.0% BLEU score compared with a baseline system which does not differentiate between Arabic dialects.

Finally, an interesting research direction could be to combine work on closely-related languages with the analysis of regional or dialectal differences in language use  ,  ,  ,  . In recent years, there has been a significant increase of interest in the computational processing of Arabic. This is evidenced by a number of research papers in several NLP tasks and applications including the identification/discrimination of Arabic dialects  ,  . Arabic is particularly interesting for researchers interested in language variation due to the fact that the language is often in a diaglossic situation, in which the standard form (Modern Standard Arabic or “MSA”) coexists with several regional dialects which are used in everyday communication. Among the studies published on the topic of Arabic ,  proposed a supervised approach to distinguish between MSA and Egyptian Arabic at the sentence level, and achieved up to 85.5% accuracy over an Arabic online commentary dataset  .  achieved higher results over the same dataset using a linear-kernel SVM classifier.

observed that general-purpose approaches to typically use a character representation of text, but successful approaches for closely-related languages, varieties, and dialects seem to favor a word-based representation or higher-order (e.g. 4-grams, 5-grams, and even 6-grams) that often cover whole words  ,  ,  ,  . The study compared character with word-based representations for over varieties of Spanish, Portuguese and French, and found that word-level models performed better for varieties of Spanish, but character models perform better in the case of Portuguese and French. To train accurate and robust systems that discriminate between language varieties or similar languages, models should ideally be able to capture not only lexical but more abstract systemic differences between languages. One way to achieve this, is by using features that use de-lexicalized text representations (e.g. by substituting named entities or content words by placeholders), or at a higher level of abstraction, using POS tags or other morphosyntactic information  ,  ,  , or even adversarial machine learning to modify the learned representations to remove such artefacts  .

A set of unique features in a corpus INLINEFORM6 is denoted by INLINEFORM7 . The number of unique features is referred to as INLINEFORM8 . The count of a feature INLINEFORM9 in the corpus INLINEFORM10 is referred to as INLINEFORM11 . If a corpus is divided into segments INLINEFORM12 , the count of a feature INLINEFORM13 in INLINEFORM14 is defined as the sum of counts over the segments of the corpus, i.e. INLINEFORM15 . Note that the segmentation may affect the count of a feature in INLINEFORM16 as features do not cross segment borders. A frequently-used feature is an , which consists of a contiguous sequence of INLINEFORM0 individual tokens. An starting at position INLINEFORM1 in a corpus segment is denoted INLINEFORM2 , where positions INLINEFORM3 remain within the same segment of the corpus as INLINEFORM4 . If INLINEFORM5 , INLINEFORM6 is an individual token. When referring to all of length INLINEFORM7 in a corpus INLINEFORM8 , we use INLINEFORM9 and the count of all such is denoted by INLINEFORM10 .

The count of an INLINEFORM11 in a corpus segment INLINEFORM12 is referred to as INLINEFORM13 and is defined by count: DISPLAYFORM0  The set of languages is INLINEFORM0 , and INLINEFORM1 denotes the number of languages. A corpus INLINEFORM2 in language INLINEFORM3 is denoted by INLINEFORM4 . A language model INLINEFORM5 based on INLINEFORM6 is denoted by INLINEFORM7 . The features given values by the model INLINEFORM8 are the domain INLINEFORM9 of the model. In a language model, a value INLINEFORM10 for the feature INLINEFORM11 is denoted by INLINEFORM12 . For each potential language INLINEFORM13 of a corpus INLINEFORM14 in an unknown language, a resulting score INLINEFORM15 is calculated. A corpus in an unknown language is also referred to as a test document.

An Archetypal Language Identifier	The design of a supervised language identifier can generally be deconstructed into four key steps: A representation of text is selected A model for each language is derived from a training corpus of labelled documents A function is defined that determines the similarity between a document and each language The language of a document is predicted based on the highest-scoring model

used byte up to length 25.   used consonant sequences generated from words.  used the presence of vowel sequences as a feature with a NB classifier (see naivebayes) when distinguishing between English and transliterated Indian languages.   used a basic dictionary (basicdictionary) composed of the 400 most common character 4-grams.   and  used character combinations (of different sizes) that either existed in only one language or did not exist in one or more languages.

Features	In this section, we present an extensive list of features used in , some of which are not self-evident. The equations written in the unified notation defined earlier show how the values INLINEFORM0 used in the language models are calculated from the tokens INLINEFORM1 . For each feature type, we generally introduce the first published article that used that feature type, as well as more recent articles where the feature type has been considered.

Bytes and Encodings	In , text is typically modeled as a stream of characters. However, there is a slight mismatch between this view and how text is actually stored: documents are digitized using a particular encoding, which is a mapping from characters (e.g. a character in an alphabet), onto the actual sequence of bytes that is stored and transmitted by computers. Encodings vary in how many bytes they use to represent each character. Some encodings use a fixed number of bytes for each character (e.g. ASCII), whereas others use a variable-length encoding (e.g. UTF-8). Some encodings are specific to a given language (e.g. GuoBiao 18030 or Big5 for Chinese), whereas others are specifically designed to represent as many languages as possible (e.g. the Unicode family of encodings). Languages can often be represented in a number of different encodings (e.g. UTF-8 and Shift-JIS for Japanese), and sometimes encodings are specifically designed to share certain codepoints (e.g. all single-byte UTF-8 codepoints are exactly the same as ASCII).

Most troubling for , isomorphic encodings can be used to encode different languages, meaning that the determination of the encoding often doesn't help in honing in on the language. Infamous examples of this are the ISO-8859 and EUC encoding families. Encodings pose unique challenges for practical applications: a given language can often be encoded in different forms, and a given encoding can often map onto multiple languages. Some research has included an explicit encoding detection step to resolve bytes to the characters they represent  , effectively transcoding the document into a standardized encoding before attempting to identify the language. However, transcoding is computationally expensive, and other research suggests that it may be possible to ignore encoding and build a single per-language model covering multiple encodings simultaneously  ,  . Another solution is to treat each language-encoding pair as a separate category  ,  ,  ,  . The disadvantage of this is that it increases the computational cost by modeling a larger number of classes. Most of the research has avoided issues of encoding entirely by assuming that all documents use the same encoding  . This may be a reasonable assumption in some settings, such as when processing data from a single source (e.g.

Similar Languages, Language Varieties, and Dialects	While one line of research into has focused on pushing the boundaries of how many languages are supported simultaneously by a single system  ,  ,  , another has taken a complementary path and focused on in groups of similar languages. Research in this area typically does not make a distinction between languages, varieties and dialects, because such terminological differences tend to be politically rather than linguistically motivated  ,  ,  , and from an NLP perspective the challenges faced are very similar.

The polySVOX text analysis module described by  uses two-level rules and morpheme lexicons on sub-word level and separate definite clause grammars (DCGs) on word, sentence, and paragraph levels. The language of sub-word units, words, sentences, and paragraphs in multilingual documents is identified at the same time as performing syntactic analysis for the document.  converted sentences into POS-tag patterns using a word-POS dictionary for Malay. The POS-tag patterns were then used by a neural network to indicate whether the sentences were written in Malay or not.  used Jspell to detect differences in the grammar of Portuguese variants.  used a syntactic grammar to recognize verb-da-verb constructions, which are characteristic of the Serbian language. The syntactic grammar was used together with several morphological grammars to distinguish between Croatian and Serbian.   used the weighted scores of the words to the left and right of the word to be classified.  used language labels within an HMM.  used the language labels of other words in the same sentence to determine the language of the ambiguous word. The languages of the other words had been determined by the positive Decision Rules (Decisionrule), using dictionaries of unique words when possible.

Morphemes, Syllables and Chunks	  used the suffixes of lexical words derived from untagged corpora.  used prefixes and suffixes determined using linguistic knowledge of the Arabic language.  used suffixes and prefixes in rule-based .  used morphemes and morpheme trigrams (morphotactics) constructed by Creutz's algorithm  .  used prefixes and suffixes constructed by his own algorithm, which was later also used by  .  used morpheme lexicons in .  compared the use of morphological features with the use of variable sized character . When choosing between ten European languages, the morphological features obtained only 26.0% accuracy while the reached 82.7%.  lemmatized Malay words in order to get the base forms.  used a morphological analyzer of Arabic.  used morphological information from a part-of-speech (POS) tagger.  and  used manually selected suffixes as features.  created morphological grammars to distinguish between Croatian and Serbian.  used morphemes created by Morfessor, but they also used manually created morphological rules.  used a suffix module containing the most frequent suffixes.  and  used word suffixes as features with CRFs.

experimented also with Markovian bigrams using both character and grapheme bigrams, but the syllable bigrams proved to work better. Graphemes in this case are the minimal units of the writing system, where a single character may be composed of several graphemes (e.g. in the case of the Hangul or Thai writing systems). Later,  also used grapheme uni- and trigrams.  achieved their best results combining word unigrams and syllable bigrams with a grapheme back-off.  used the MADAMIRA toolkit for D3 decliticization and then used D3-token 5-grams. D3 decliticization is a way to preprocess Arabic words presented by  . Graphones are sequences of characters linked to sequences of corresponding phonemes. They are automatically deduced from a bilingual corpus which consists of words and their correct pronunciations using Joint Sequence Models (“JSM”).  used language tags instead of phonemes when generating the graphones and then used Markovian graphone from 1 to 8 in .

Odds ratio for language INLINEFORM0 when compared with language INLINEFORM1 for morph INLINEFORM2 is calculated as in Equation EQREF127 . DISPLAYFORM0

Range compression is the grouping of a range of characters into a single logical set for counting purposes, and is a technique that is commonly used to deal with the sparsity that results from character sets for ideographic languages, such as Chinese, that may have thousands of unique “characters”, each of which is observed with relatively low frequency.  use such a technique where all characters in a given range are mapped into a single “bucket”, and the frequency of items in each bucket is used as a feature to represent the document. Byte-level representations of encodings that use multi-byte sequences to represent codepoints achieve a similar effect by “splitting” codepoints. In encodings such as UTF-8, the codepoints used by a single language are usually grouped together in “code planes”, where each codepoint in a given code plane shares the same upper byte. Thus, even though the distribution over codepoints may be quite sparse, when the byte-level representation uses byte sequences that are shorter than the multi-byte sequence of a codepoint, the shared upper byte will be predictive of specific languages. Cleaning may also be applied, where heuristic rules are used to remove some data that is perceived to hinder the accuracy of the language identifier.

Support Vector Machines (“SVMs”)	With support vector machines (“SVMs”), a binary classifier is learned by learning a separating hyperplane between the two classes of instances which maximizes the margin between them. The simplest way to extend the basic SVM model into a multiclass classifier is via a suite of one-vs-rest classifiers, where the classifier with the highest score determines the language of the test document. One feature of SVMs that has made them particularly popular is their compatibility with kernels, whereby the separating hyperplane can be calculated via a non-linear projection of the original instance space. In the following paragraphs, we list the different kernels that have been used with SVMs for . For with SVMs, the predominant approach has been a simple linear kernel SVM model. The linear kernel model has a weight vector INLINEFORM0 and the classification of a feature vector INLINEFORM1 , representing the test document INLINEFORM2 , is calculated as follows: DISPLAYFORM0  where INLINEFORM0 is a scalar bias term. If INLINEFORM1 is equal to or greater than zero, INLINEFORM2 is categorized as INLINEFORM3 .

The first to use a linear kernel SVM were  , and generally speaking, linear-kernel SVMs have been widely used for , with great success across a range of shared tasks.   were the first to apply polynomial kernel SVMs to . With a polynomial kernel INLINEFORM0 can be calculated as: DISPLAYFORM0  where INLINEFORM0 is the polynomial degree, and a hyperparameter of the model. Another popular kernel is the RBF function, also known as a Gaussian or squared exponential kernel. With an RBF kernel INLINEFORM0 is calculated as: DISPLAYFORM0  where INLINEFORM0 is a hyperparameter.  were the first to use an RBF kernel SVM for . With sigmoid kernel SVMs, also known as hyperbolic tangent SVMs, INLINEFORM0 can be calculated as: DISPLAYFORM0    were the first to use a sigmoid kernel SVM for , followed by  , who found the SVM to perform better than NB, Classification And Regression Tree (“CART”), or the sum of relative frequencies. Other kernels that have been used with SVMs for include exponential kernels  and rational kernels  .  were the first to use SVMs for , in the form of string kernels using Ukkonen's algorithm.

Text Preprocessing	Text preprocessing (also known as normalization) is an umbrella term for techniques where an automatic transformation is applied to text before it is presented to a classifier. The aim of such a process is to eliminate sources of variation that are expected to be confounding factors with respect to the target task. Text preprocessing is slightly different from data cleaning, as data cleaning is a transformation applied only to training data, whereas normalization is applied to both training and test data.  raise text preprocessing as an outstanding issue in , arguing that its effects on the task have not been sufficiently investigated. In this section, we summarize the normalization strategies that have been proposed in the literature. Case folding is the elimination of capitalization, replacing characters in a text with either their lower-case or upper-case forms. Basic approaches generally map between [a-z] and [A-Z] in the ASCII encoding, but this approach is insufficient for extended Latin encodings, where diacritics must also be appropriately handled. A resource that makes this possible is the Unicode Character Database (UCD) which defines uppercase, lowercase and titlecase properties for each character, enabling automatic case folding for documents in a Unicode encoding such as UTF-8.

RNNs were more successfully used later by  who also incorporated character n-gram features in to the network architecture.  were the first to use a Long Short-Term Memory (“LSTM”) for  , and  was the first to use Gated Recurrent Unit networks (“GRUs”), both of which are RNN variants.  used byte-level representations of sentences as input for the networks. Recently,  and  also used LSTMs. Later, GRUs were successfully used for by  and  . In addition to GRUs,  also experimented with deep residual networks (“ResNets”) at DSL 2016. During 2016 and 2017, there was a spike in the use of convolutional neural networks (CNNs) for , most successfully by  and  . Recently,  combined a CNN with adversarial learning to better generalize to unseen domains, surpassing the results of  based on the same training regime as .   used CBOW NN, achieving better results over the development set of DSL 2017 than RNN-based neural networks.  used deep averaging networks (DANs) based on word embeddings in language variety identification.

It uses a document representation based on tries  . The algorithm is a LR classifier using all possible substrings of the data, which is important to maximize the available information from the relatively short Twitter messages.   provides a comparison of 8 off-the-shelf language identifiers applied without re-training to Twitter messages. One issue they report is that comparing the accuracy of off-the-shelf systems is difficult because of the different subset of languages supported by each system, which may also not fully cover the languages present in the target data. The authors choose to compare accuracy over the full set of languages, arguing that this best reflects the likely use-case of applying an off-the-shelf system to new data. They find that the best individual systems are , and , but that slightly higher accuracy can be attained by a simple voting-based ensemble classifier involving these three systems. In addition to this, commercial or other closed-source language identifiers and language identifier services exist, of which we name a few. The Polyglot 3000 and Lextek Language Identifier are standalone language identifiers for Windows. Open Xerox Language Identifier is a web service with available REST and SOAP APIs.

One particular feature of is that it uses discriminative training in selecting features, i.e. it specifically makes use of features that are strong evidence against a particular language, which is generally not captured by NB models. Another feature of is that it uses inter-string smoothing to exploit sentence-level locality in making language predictions, under the assumption that adjacent sentences are likely to be in the same language.  reports that this substantially improves the accuracy of the identifier. Another distinguishing feature of is that it comes pre-trained with data for 1400 languages, which is the highest number by a large margin of any off-the-shelf system. whatthelang is a recent language identifier written in Python, which utilizes the FastText NN-based text classification algorithm. It supports 176 languages. implements an off-the-shelf classifier trained using Wikipedia data, covering 122 languages. Although not described as such, the actual classification algorithm used is a linear model, and is thus closely related to both NB and a cosine-based vector space model. In addition to the above-mentioned general-purpose language identifiers, there have also been efforts to produce pre-trained language identifiers targeted specifically at Twitter messages. is a Twitter-specific tool with built-in models for 19 languages.

used majority voting between three sum of relative frequencies-based classifiers where values were weighted with different weighting schemes.  ,  ,  ,  ,  , and  used plurality voting with SVMs.   used voting between several perplexity-based classifiers with different features at the 2017 DSL shared task. A voting ensemble gave better results on the closed track than a singular word-based perplexity classifier (0.9025 weighted F1-score over 0.9013), but worse results on the open track (0.9016 with ensemble and 0.9065 without). In a highest probability ensemble, the winner is simply the language which is given the highest probability by any of the individual classifiers in the ensemble.  used Gaussian Mixture Models (“GMM”) to give probabilities to the outputs of classifiers using different features.  used higher confidence between two off-the-shelf language identifiers.  used GMM to transform SVM prediction scores into probabilities.  ,  used highest confidence over a range of base SVMs.  used an ensemble composed of low-dimension hash-based classifiers. According to their experiments, hashing provided up to 86% dimensionality reduction without negatively affecting performance.

The 5th edition of the PAN at CLEF author profiling task included language variety identification for tweets  . There has also been research on identifying the language of private messages between eBay users  , presumably as a filtering step prior to more in-depth data analysis.

In such a setting, it is common to apply at two points: (1) to the documents being indexed; and (2) to the queries being executed on the collection. Simple keyword matching techniques can be problematic in text-based document retrieval, because the same word can be valid in multiple languages. A classic example of such words (known as “false friends”) includes gift, which in German means “poison”. Performing on both the document and the query helps to avoid confusion between such terms, by taking advantage of the context in which it appears in order to infer the language. This has resulted in specific work in of web pages, as well as search engine queries.  and  give overviews of shared tasks specifically concentrating on language labeling of individual search query words. Having said this, in many cases, the search query itself does a sufficiently good job of selecting documents in a particular language, and overt is often not performed in mixed multilingual search contexts. Automatic has also been used to facilitate linguistic and other text-based research.  report that their motivation for developing a language identifier was “to find out how many web pages are written in a particular language”. Automatic has been used in constructing web-based corpora.

German or Japanese) but is interspersed with words and phrases (often technical terms) from a language such as English. For example,  found that around 6% of word tokens in German text sourced from the Internet are English inclusions. In the context of POS tagging, one strategy for dealing with inclusions is to have a dedicated POS for all foreign words, and force the POS tagger to perform both foreign inclusion detection and POS tag these words in the target language; this is the approach taken in the Penn POS tagset, for example  . An alternative strategy is to have an explicit foreign inclusion detection pre-processor, and some special handling of foreign inclusions. For example, in the context of German parsing,  used foreign inclusion predictions to restrict the set of (German) POS tags used to form a parse tree, and found that this approach substantially improved parser accuracy. Another commonly-mentioned use case is for multilingual document storage and retrieval. A document retrieval system (such as, but not limited to, a web search engine) may be required to index documents in multiple languages.

In 2017 the competition introduced the inclusion of language varieties and dialects of Arabic, English, Spanish, and Portuguese, More ambitiously, the four editions of the Discriminating between Similar Languages (DSL)  ,  ,  ,  shared tasks required participants to discriminate between a set of languages in several language groups, each consisting of highly-similar languages or national varieties of that language. The dataset, entitled DSL Corpus Collection (“DSLCC”)  , and the languages included are summarized in dslcc. Historically the best-performing systems  ,  ,  have approached the task via hierarchical classification, first predicting the language group, then the language within that group.

Another example is the Arabic Dialect Identification (“ADI”) shared task at the VarDial workshop  ,  , and the Arabic Multi-Genre Broadcast (“MGB”) Challenge  . Two shared tasks focused on a narrow group of languages using Twitter data. The first was TweetLID, a shared task on of Twitter messages according to six languages in common use in Spain, namely: Spanish, Portuguese, Catalan, English, Galician, and Basque (in order of the number of documents in the dataset)  ,  . The organizers provided almost 35,000 Twitter messages, and in addition to the six monolingual tags, supported four additional categories: undetermined, multilingual (i.e. the message contains more than one language, without requiring the system to specify the component languages), ambiguous (i.e. the message is ambiguous between two or more of the six target languages), and other (i.e. the message is in a language other than the six target languages). The second shared task was the PAN lab on authorship profiling 2017  . The PAN lab on authorship profiling is held annually and historically has focused on age, gender, and personality traits prediction in social media.

Future work in this area could investigate other means of ensuring correct gold-standard labels while minimizing the annotation cost. Despite these challenges, standardized datasets are critical for replicable and comparable research in . Where a subset of data is used from a larger collection, researchers should include details of the specific subset, including any breakdown into training and test data, or partitions for cross-validation. Where data from a new source is used, justification should be given for its inclusion, as well as some means for other researchers to replicate experiments on the same dataset.

For example,  identify HTML entities as a candidate for removal in document cleaning, on the basis that classifiers trained on data which does not include such entities may drop in accuracy when applied to raw HTML documents. includes heuristics such as expanding HTML entities, deleting digits and punctuation, and removing SGML-like tags. Similarly, also removes “language-independent characters” such as numbers, symbols, URLs, and email addresses. It also removes words that are all-capitals and tries to remove other acronyms and proper names using heuristics. In the domain of Twitter messages,  remove links, usernames, smilies, and hashtags (a Twitter-specific “tagging” feature), arguing that these entities are language independent and thus should not feature in the model.  address of web pages, and report removing HTML formatting, and applying stopping using a small stopword list.  carry out experiments on the ECI multilingual corpus and report removing punctuation, space characters, and digits. The idea of preprocessing text to eliminate domain-specific “noise” is closely related to the idea of learning domain-independent characteristics of a language  .

For learning they experimented with both Kernel Discriminant Analysis (“KDA”) and Kernel Ridge Regression (“KRR”).  also used KDA.   calculated the Levenshtein distance between the language models and each word in the mystery text. The similary score for each language was the inverse of the sum of the Levenshtein distances. Their language identifier obtained 97.7% precision when classifying texts from two to four words between five languages. Later  used Levenshtein distance for Algerian dialect identification and  for query word identification.   ,  ,  , and  calculated the difference between probabilities as in Equation EQREF109 . DISPLAYFORM0   where INLINEFORM0 is the probability for the feature INLINEFORM1 in the mystery text and INLINEFORM2 the corresponding probability in the language model of the language INLINEFORM3 . The language with the lowest score INLINEFORM4 is selected as the most likely language for the mystery text.  ,  used the log probability difference and the absolute log probability difference. The log probability difference proved slightly better, obtaining a precision of 94.31% using both character and word when classifying 100 character texts between 53 language-encoding pairs.

Their solution to this task is to take advantage of a domain-specific feature: they assume that the name of the language that they are extracting must appear in the document containing the IGT, and hence treat this as a co-reference resolution problem. They report that this approach significantly outperforms the text-based approach in this particular problem setting. An interesting area to explore is the trade-off between the number of languages supported and the accuracy per-language. From existing results it is not clear if it is possible to continue increasing the number of languages supported without adversely affecting the average accuracy, but it would be useful to quantify if this is actually the case across a broad range of text sources. mostlanguages lists the articles where the with more than 30 languages has been investigated.

Future work could examine how useful different types of negative evidence are for filtering – if we want to detect English documents, e.g., are there empirical advantages in having distinct models of Italian and German (even if we don't care about the distinction between the two languages), or can we group them all together in a single “negative” class? Are we better off including as many languages as possible in the negative class, or can we safely exclude some?

Words	  used the position of the current word in word-level . The position of words in sentences has also been used as a feature in code-switching detection by  . It had predictive power greater than the language label or length of the previous word.   used the characteristics of words as parts of discriminating functions.  used the string edit distance and overlap between the word to be identified and words in dictionaries. Similarly  used a modified edit distance, which considers the common spelling substitutions when Hindi is written using latin characters.  used the Minimum Edit Distance (“MED”). Basic dictionaries are unordered lists of words belonging to a language. Basic dictionaries do not include information about word frequency, and are independent of the dictionaries of other languages.  used a dictionary for as a part of his speech synthesizer. Each word in a dictionary had only one possible “language”, or pronunciation category. More recently, a basic dictionary has been used for by  ,  , and  . Unique word dictionaries include only those words of the language, that do not belong to the other languages targeted by the language identifier.  used unique short words (from one to three characters) to differentiate between languages.

Recently, a dictionary of unique words was used for by  ,  , and  .   used exhaustive lists of function words collected from dictionaries.  used stop words – that is non-content or closed-class words – as a training corpus. Similarly,  used words from closed word classes, and  used lists of function words.  used a lexicon of Arabic words and phrases that convey modality. Common to these features is that they are determined based on linguistic knowledge.   used the most relevant words for each language.  used unique or nearly unique words.  used Information Gain Word-Patterns (“IG-WP”) to select the words with the highest information gain.   made an (unordered) list of the most common words for each language, as, more recently, did  ,  , and  .  encoded the most common words to root forms with the Soundex algorithm.   collected the frequencies of words into feature vectors.  compared the use of character from 2 to 5 with the use of words.

calculated the Pointwise Mutual Information (“PMI”) for words and used it to group words to Chinese dialects or dialect groups. Recently, the relative frequency of words has also been used for by  ,  and     used the relative frequency of words with less than six characters. Recently,  also used short words, as did  .   used the relative frequency calculated from Google searches. Google was later also used by  and  .   created probability maps for words for German dialect identification between six dialects. In a word probability map, each predetermined geographic point has a probability for each word form. Probabilities were derived using a linguistic atlas and automatically-induced dialect lexicons.   used commercial spelling checkers, which utilized lexicons and morphological analyzers. The language identifier of  obtained 97.9% accuracy when classifying one-line texts between 11 official South African languages.  used the ALMORGEANA analyzer to check if the word had an analysis in modern standard Arabic. They also used sound change rules to use possible phonological variants with the analyzer.  used spellchecking and morphological analyzers to detect English words from Hindi–English mixed search queries.

used spelling checkers to distinguish between 15 languages, extending the work of  with dynamic model selection in order to gain better performance.  used a similarity count to find if mystery words were misspelled versions of words in a dictionary.   used an “LBG-VQ” (Linde, Buzo & Gray algorithm for Vector Quantization) approach to design a codebook for each language  . The codebook contained a predetermined number of codevectors. Each codeword represented the word it was generated from as well as zero or more words close to it in the vector space.

,  used the language tags of the previous three words with an SVM.  used language labels of surrounding words with NB.  used the language probabilities of the previous word to determining weights for languages.  used unigram, bigram and trigram language label transition probabilities.  used the language labels for the two previous words as well as knowledge of whether code-switching had already been detected or not.  used the language label of the previous word to determine the language of an ambiguous word.  also used the language label of the previous word.  used the language identifications of 2–4 surrounding words for post-identification correction in word-level .  used language labels with a CRF.  used language labels of the current and two previous words in code-switching point prediction. Their predictive strength was lower than the count of code-switches, but better than the length or position of the word. All of the features were used together with NB, DT and SVM.  used language label bigrams with an HMM.  used the word-level language labels obtained with the approach of  on sentence-level dialect identification.

If an unseen context is encountered for the character being processed, the escape probability is used together with a lower-order model probability. In PPM-C, the escape probability is the sum of the seen contexts in the language model. PPM-C was lately used by  . The PPM-D+ algorithm was used by  .  and  used a PPM-A variant.  also used PPM. The language identifier of  obtained 91.4% accuracy when classifying 100 character texts between 277 languages.  used Witten-Bell smoothing with perplexity.   used a Chunk-Based Language Model (“CBLM”), which is similar to PPM models.   used several smoothing techniques with Markovian probabilities. Absolute discounting from the VariKN toolkit performed the best.  define the smoothing as follows: a constant INLINEFORM0 is subtracted from the counts INLINEFORM1 of all observed INLINEFORM2 and the held-out probability mass is distributed between the unseen in relation to the probabilities of lower order INLINEFORM3 , as follows: DISPLAYFORM0  where INLINEFORM0 is a scaling factor that makes the conditional distribution sum to one.

Witten-Bell proved to be clearly better than Kneser-Ney.

Theoretically the features are assumed to be independent of each other, but in practice using features that are functionally dependent can improve classification accuracy  . NB implementations have been widely used for , usually with a more varied set of features than simple character or word of the same type and length. The features are typically represented as feature vectors given to a NB classifier.  trained a NB classifier with language labels of surrounding words to help predict the language of ambiguous words first identified using an SVM. The language identifier used by  obtained 99.97% accuracy with 5-grams of characters when classifying sentence-sized texts between six language groups.  used a probabilistic model similar to NB.  used NB and naive Bayes EM, which uses the Expectation–Maximization (“EM”) algorithm in a semi-supervised setting to improve accuracy.  used Gaussian naive Bayes (“GNB”, i.e. NB with Gaussian estimation over continuous variables) from scikit-learn. In contrast to NB, in Bayesian networks the features are not assumed to be independent of each other. The network learns the dependencies between features in a training phase.

Such models have been successful in topic modeling, where an outstanding issue with the popular LDA model is the need to specify the number of topics in advance.  introduced an approach to detecting multilingual documents that uses a model very similar to LDA, where languages are analogous to topics in the LDA model. Using a similar analogy, an HDP-based model may be able to detect documents that are written in a language that is not currently modeled by the system.  used LDA to cluster unannotated tweets. Recently  used LDA in unsupervised sentence-level . They manually identified the languages of the topics created with LDA. If there were more topics than languages then the topics in the same language were merged. Filtering, a task that we mentioned earlier in this section, is a very common application of , and it is therefore surprising that there is little research on filtering for specific languages. Filtering is a limit case of with unseen languages, where all languages but one can be considered unknown.

They use a character representation for text, and a clustering algorithm that consists of an initial INLINEFORM0 -means phase, followed by particle-swarm optimization. This produces a large number of small clusters, which are then labeled by language through a separate step.  used co-occurrences of words with INLINEFORM1 -means clustering in word-level unsupervised . They used a Dirichlet process Gaussian mixture model (“DPGMM”), a non-parametric variant of a GMM, to automatically determine the number of clusters, and manually labeled the language of each cluster.  also used INLINEFORM2 -means clustering, and  used the INLINEFORM3 -means clustering algorithm in a custom framework.  utilized unlabeled data to improve their system by using a CRF autoencoder, unsupervised word embeddings, and word lists. A different partial solution to the issue of unseen languages is to design the classifier to be able to output “unknown” as a prediction for language. This helps to alleviate one of the problems commonly associated with the presence of unseen languages – classifiers without an “unknown” facility are forced to pick a language for each document, and in the case of unseen languages, the choice may be arbitrary and unpredictable  .

Decision Trees	  were the earliest users of Decision Trees (“DT”) in . They used DT based on characters and their context without any frequency information. In training the DT, each node is split into child nodes according to an information theoretic optimization criterion. For each node a feature is chosen, which maximizes the information gain at that node. The information gain is calculated for each feature and the feature with the highest gain is selected for the node. In the identification phase, the nodes are traversed until only one language is left (leaf node). Later,  ,  , and  have been especially successful in using DTs. Random Forest (RF) is an ensemble classifier generating many DTs. It has been succesfully used in by  ,  ,  , and  ,  .

Simple Scoring	In simple scoring, each feature in the test document is checked against the language model for each language, and languages which contain that feature are given a point, as follows: DISPLAYFORM0  where INLINEFORM0 is the INLINEFORM1 th feature found in the test document INLINEFORM2 . The language scoring the most points is the winner. Simple scoring is still a good alternative when facing an easy problem such as preliminary language group identification. It was recently used for this purpose by  with a basic dictionary. They achieved 99.8% accuracy when identifying between 6 language groups.  use a version of simple scoring as a distance measure, assigning a penalty value to features not found in a model. In this version, the language scoring the least amount of points is the winner. Their language identifier obtained 100% success rate with character 4-grams when classifying relatively large documents (from 1 to 3 kilobytes), between 10 languages. Simple scoring was also used lately by  ,  , and  .

The documents grouped together by these discriminating terms are merged and the process is repeated until the wanted number of groups (corresponding to languages) is reached.  also presents an approach to unseen language problem, building graphs of co-occurrences of words in sentences, and then partitioning the graph using a custom graph-clustering algorithm which labels each word in the cluster with a single label. The number of labels is initialized to be the same as the number of words, and decreases as the algorithm is recursively applied. After a small number of iterations (the authors report 20), the labels become relatively stable and can be interpreted as cluster labels. Smaller clusters are then discarded, and the remaining clusters are interpreted as groups of words for each language.  compared the Chinese Whispers algorithm of  and Graclus clustering on unsupervised Tweet . They conclude that Chinese Whispers is better suited to .  used Fuzzy ART NNs for unsupervised language clustering for documents in Arabic, Persian, and Urdu. In Fuzzy ART, the clusters are also dynamically updated during the identification process.   also tackle the unseen language problem through clustering.

summed up language labels of the surrounding words to identify the language of the current word.  summed up points awarded by the presence of morphological and syntactic features.  used inverse rank positions as values.  computed the sum of keywords weighted with TF-IDF.  summed up the TF-IDF derived probabilities of words.

Introduction	Language identification (“”) is the task of determining the natural language that a document or part thereof is written in. Recognizing text in a specific language comes naturally to a human reader familiar with the language. intro:langid presents excerpts from Wikipedia articles in different languages on the topic of Natural Language Processing (“NLP”), labeled according to the language they are written in. Without referring to the labels, readers of this article will certainly have recognized at least one language in intro:langid, and many are likely to be able to identify all the languages therein. Research into aims to mimic this human ability to recognize specific languages. Over the years, a number of computational approaches have been developed that, through the use of specially-designed algorithms and indexing structures, are able to infer the language being used without the need for human intervention. The capability of such systems could be described as super-human: an average person may be able to identify a handful of languages, and a trained linguist or translator may be familiar with many dozens, but most of us will have, at some point, encountered written texts in languages they cannot place.

Question: what evaluation methods are discussed?
Answer: document-level accuracy, precision, recall, F-score