Patent Document ID: 8301437
Application ID: 12178775

Base Claim:
1. A method for tokenizing a character string, comprising: (a) determining if there are any words or phrases in a dictionary that match a series of characters within the character string that begins at the first character of the character string, wherein the character string comprises a non-delimited character string; (b) for each matching word or phrase identified in step (a), assigning the matching word or phrase to a tokenization path, wherein the tokenization path comprises one or more contiguous words or phrases embedded within the character string, and removing a corresponding series of characters from the beginning of the character string, thereby generating a shortened character string associated with the tokenization path or terminating the tokenization path; (c) if no matching word or phrase is identified in step (a), then terminating any tokenization path with which the character string is associated; (d) recursively performing steps (a), (b) and (c) for any shortened character string generated in step (b) until all tokenization paths are terminated; (e) for any tokenization path formed through the performance of steps (a)-(d), calculating a score based on each word or phrase assigned to the tokenization path; and (f) selecting the word(s) and/or phrase(s) associated with a tokenization path having the highest score as tokens associated the character string.

---

Claim 8:
8. The method of claim 1 , further comprising isolating the character string by performing a tokenization process on an original character string that includes the character string, the tokenization processing comprising: (i) identifying one or more first tokens within the original character string based on any delimiters identified in the original character string; (ii) identifying one or more second tokens within each of the one or more first tokens based on any capital letters identified in each of the one or more first tokens; and (iii) selectively identifying one or more third tokens within each of the one or more second tokens based on alphabetic and numeric character combinations present in each of the one or more second tokens.