Patent ID: 7627567

Claim:
A process to evaluate an input string to segment said string into component parts comprising: providing a state transition model derived from training data from an existing collection of data records that includes probabilities to segment input strings into component parts, wherein the training data corresponding to database attributes in the existing collection of data records does not comprise manually segmented training data, and the state transition model categorizes tokens in database attribute values of the data records into positions based on a fixed beginning, middle, and trailing token topology that: categorizes each boundary token of a database attribute value that includes multiple tokens into corresponding beginning and trailing positions, categorizes each token that does not comprise a boundary token of a database attribute value into a middle position, defines beginning, middle, and trailing state categories, wherein each state category includes states that accept tokens only if appearing in a corresponding one of said beginning, middle, and trailing positions, and adjusts said states and probabilities associated with said states within said state categories in order to relax sequential specificity and account for erroneous token placement when evaluating tokens in the input string appearing in particular positions, wherein the state category corresponding to a particular position in which the token appears is adjusted to include states from another state category that accept tokens appearing in a different position; determining a most probable segmentation of the input string by comparing tokens that make up the input string with the state transition model derived from the existing collection of data records; segmenting the input string into one or more component parts according to the most probable segmentation; and storing the one or more component parts in a database on a computer system.