An Algorithm that Learns What ’ s in a

In this paper, we present IdentiFinderTM, a hidden Markov model that learns to recognize and classify names, dates, times, and numerical quantities. We have evaluated the model in English (based on data from the Sixth and Seventh Message Understanding Conferences [MUC-6, MUC-7] and broadcast news) and in Spanish (based on data distributed through the First Multilingual Entity Task [MET-1]), and on speech input (based on broadcast news). We report results here on standard materials only to quantify performance on data available to the community, namely, MUC-6 and MET-1. Results have been consistently better than reported by any other learning algorithm. IdentiFinder’s performance is competitive with approaches based on handcrafted rules on mixed case text and superior on text where case information is not available. We also present a controlled experiment showing the effect of training set size on performance, demonstrating that as little as 100,000 words of training data is adequate to get performance around 90% on newswire. Although we present our understanding of why this algorithm performs so well on this class of problems, we believe that significant improvement in performance may still be possible. 1. The Named Entity Problem and Evaluation 1.1. The Named Entity Task The named entity task is to identify all named locations, named persons, named organizations, dates, times, monetary amounts, and percentages in text (see Figure 1.1). Though this sounds clear, enough special cases arise to require lengthy guidelines, e.g., when is The Wall Street Journal n artifact, and when is it an organization? When is White House an organization, and when a location? Are branch offices of a bank an organization? Is a street name a location? Should yesterday and last Tuesday be labeled dates? Is mid-morning a time? In order to achieve human annotator consistency, guidelines with numerous special cases have been defined for the Seventh Message Understanding Conference, MUC-7 (Chinchor, 1998). † Daniel M. Bikel’s current address is Department of Computer & Information Science, University of Pennsylvania, 200 South 33 rd Street, Philadelphia, PA 19104. * Please address correspondence to this author. D. M. BIKEL, ET AL. 2 WHAT’S IN A NAME The delegation, which included the commander of the U . N. troops in Bosnia , Lt. Gen. Sir Michael Rose , went to the Serb stronghold of P ale , n ar S arajevo , for talks with Bosnian Serb leader Radovan Karadzic . Este ha sido el primer comentario publico del presidente Clinton respecto a la crisis de O riente Medio desde que el secretario de Estado, Warren Christopher , decidiera regresar precipitadamente a W ashington para impedir la ruptura del proceso de paz tras la violencia desatada en el sur de L ibano . 1. L ocations 2. Persons 3. O rganizations Figure 1.1 Examples. Examples of correct labels for English text and for Spanish text . Both the boundaries of an expression and its label must be marked. The Standard Generalized Markup Language, or SGML, is an abstract syntax for marking information and structure in text, and is therefore appropriate for named entity mark-up. Various GUIs to support manual preparation of answer keys are available. 1.2. Evaluation Metric A computer program is used to evaluate the performance of a name-finder, called a “scoring program”. The scoring program developed for the MUC and Multilingual Entity Task (MET) evaluations measures both precision (P) and recall (R), terms borrowed from the information-retrieval community, where P number of correct responses number of responses = and R number of correct responses number correct in key = . (1.1) (The term response is used to denote “answer delivered by a name-finder”; the term k y or key file is used to denote “an annotated file containing correct answers”.) Put informally, recall measures the number of “hits” vs. the number of possible correct answers as specified in the key, whereas precision measures how many answers were correct ones compared to the number of answers delivered. These two measures of performance combine to form one measure of performance, the F-measure, which is computed by the uniformly weighted harmonic mean of precision and recall: F RP R P = + 2 ( ) . (1.2) In MUC and MET, a correct answer from a name-finder is one where the label and both boundaries are correct. There are three types of labels, each of which use an attribute to specify a particular entity. Label types and the entities they denote are defined as follows: 1. entity (ENAMEX): person, organization, location 2. time expression ( TIMEX): date, time 3. numeric expression ( NUMEX): money, percent. A response is half correct if the label (both type and attribute) is correct but only one boundary is correct. Alternatively, a response is half-correct if only the type of the label (and D. M. BIKEL, ET AL. 3 WHAT’S IN A NAME not the attribute) and both boundaries are correct. Automatic scoring software is available, as detailed in Chinchor (1998).