These scores are used to assist the entity assignment system in properly classifying entities. In addition to analyzing the phrase, words that precede and follow the phrase are analyzed. Lexicons can automatically be generated using a tagged corpus, but our testing has shown that it is more effective to augment these lexicons with existing information. This results in the system generating better results when training data is minimal or nonexistent. A training mode is built into the system in the event that no training information is available.
Human indexers can train the system by interactively assigning the correct entity types to phrases.
The scores generated by the finite state lexical lookup system are then passed to the entity assignment system. Tie scores can occur when it is acceptable to classify an entity as two different types. Estee Lauder is both a person and an organization. The entity assignment system is a neural network trained to do the final classification. Based on these inputs the neural net assigns the most likely entity value.