Patent ID: 7809553

Claim:
A system for creating compact linguistic data, comprising: a processor; and a memory coupled to the processor, the memory comprising: a corpus; and a linguistic data analyzer, wherein the linguistic data analyzer comprises instructions that, when executed by the processor, cause the processor to: calculate frequencies of words appearing in the corpus, map each unique character in the words to a character index, replace each character in the words with the character index to which the character is mapped, map sequences of characters that appear in the words to substitution indexes, replace each sequence of characters in each word with the substitution index to which the sequence of characters are mapped, arrange the words into groups where each group contains words that start with a common prefix, calculate frequencies of the groups of words appearing in the corpus, and map each prefix to location information for the group of words which start with the prefix, and wherein the processor uses a structure to calculate the frequencies of the words appearing in the corpus such that more frequently occurring words are located at inner nodes of the structure and less frequently occurring words are located at outer nodes of the structure; and wherein the compact linguistic data includes the unique characters, the character indexes, the substitution indexes, the location information, the groups of words, the frequencies of the words and the frequencies of the groups of words.