Patent ID: 9665560
Date: 2017-05-30
CPC Classifications: G06F

Claim:
1. A method for representing a plurality of languages in a lexicon data structure based on a unified language model, the method comprising: receiving, by a lexicon developer system comprising one or more microprocessors and storage devices, first text data representing content expressed in a first language, and second text data representing content expressed in a second language different from the first language; parsing, by the lexicon developer system, the first text data into a first set of content words and elements other than the content words, based on one or more of stop words, punctuation, or numbers associated with the first language; parsing, by the lexicon developer system, the second text data into a second set of content words and elements other than the content words, based on one or more of stop words, punctuation, or numbers associated with the second language; classifying by the lexicon developer system, the first set of content words and the second set of content words into a plurality of groups based on commonalities between the sounds of the words when spoken, wherein at least one of the groups includes at least one word from the first language and at least one word from the second language; performing, by the lexicon developer system, a statistical analysis on the classified groups of content words; defining, by the lexicon developer system, one or more entries in a lexicon data structure for the classified content words in each group based on the statistical analysis, receiving a data processing request comprising additional content expressed in either the first language or the second language, the data processing request comprising one or more of a data file compression request, a data file decompression request, a search query request, speech file processing request, or a text data file translation request; and using the lexicon data structure to perform the data processing request on the additional content.