Patent ID: 7865355

Claim:
A method for identifying a language corresponding to a string of data, comprising: receiving a data string from memory, the data string comprising a string of code points defining characters of a language where one or more of the code point values corresponds to more than one distinct character set, the data string having been electronically stored in the memory; dividing the data string into a plurality of coded character sequences for each of a plurality of languages to represent the same number of characters in each language of the plurality of languages, wherein a length of one or more coded character sequences varies among different languages; analyzing the coded character sequences to calculate, for each of the plurality of languages, an overall probability that the data string corresponds to a particular language, the analyzing coded character sequences for each language including: calculating a probability for each coded character sequence using statistical information, the probability being the probability that the particular coded character sequence corresponds to the language, determining that a probability for any of the coded character sequences is zero, comparing a code point of a coded character sequence having a probability of zero to respective code point definitions of each of the plurality of languages to affect a comparison, determining that the coded character sequence includes illegal code points for at least one language of the plurality of languages based on the comparison and eliminating the at least one language as a language to which the coded character sequence could correspond, removing the probability from an overall probability calculation when determining that the probability is zero and that the coded character sequence does not include an illegal code point, and using the probabilities to calculate the overall probability that the data string corresponds to the language; comparing the calculated overall probabilities among the languages; and identifying a language as corresponding to the data string based on comparing the calculated overall probabilities.