However, this information has not yet been fully exploited in language ID systems. While the progress of language ID research in the last two years has been heartening, there is much to do. Moreover, the level of performance is nowhere near acceptability in a real-world environment. Written Language Written language identification has received less attention than spoken language recognition. Perfect discrimination of the eight languages was obtained.
Most methods rely on input in the form of character codes.
His experiments, however, did not use an independent test set. He compared the successive trigrams derived from a body of text with a database of trigram sets generated for each language. The language for which the greatest number of trigram matches were obtained, and for which the frequencies of occurrence of the trigrams exceeded a language-specific threshold, was chosen the winner. No results were specified. Their experiments revealed that the HMMs had better entropy than bigrams but were comparable to the computationally expensive trigrams.