Bayesian Information Extraction Network Content Areas : information extraction , probabilistic reasoning , information retrieval

Dynamic Bayesian networks (DBNs) offer an elegant way to integrate various aspects of language in one model. Many existing algorithms developed for learning and inference in DBNs are applicable to probabilistic language modeling. To demonstrate the potential of DBNs for natural language processing, we employ a DBN in an information extraction task. We show how to assemble wealth of emerging linguistic instruments for shallow parsing, syntactic and semantic tagging, morphological decomposition, named entity recognition etc. in order to incrementally build a robust information extraction system. Our method outperforms previously published results on an established benchmark domain. 1 Information Extraction Information extraction IE is the task of filling in template information from previously unseen text which belongs to a pre-defined domain. The resulting database is suited for formal queries and filtering. IE systems generally work by detecting patterns in the text that help to identify significant information. Researchers have shown [Freitag and McCallum, 1999; Ray and Craven, 2001] that a probabilistic approach allows the construction of robust and well-performing systems. However, the existing probabilistic systems are generally based on hidden Markov models (HMMs). Due to this relatively impoverished representation, they are unable to take advantage of the wide array of linguistic information used by many non-probabilistic IE systems. In addition, existing HMM-based systems model each target category separately, failing to capture relational information, such as typical elements order or the fact that each element only belongs to a single category. This paper shows how to incorporate a wide array of knowledge into a probabilistic IE system, based on dynamic Bayesian networks (DBN)—a rich probabilistic representation language that generalizes HMMs. Let us illustrate IE task by describing seminar announcements which got established as one of the most popular benchmark domains in the field [Califf and Mooney, 1999; Freitag and McCallum, 1999; Soderland, 1999; Roth and Yih, 2001; Ciravegna, 2001]. People receive dozens of seminar announcements weekly and need to manually extract information and paste it into personal organizers. The goal of an IE system is to automatically identify target fields such as location and topic of a seminar, date and starting time, ending time and speaker. Announcements come in many formats, but usually follow some pattern. We often find a header with a gist in the form “PostedBy: john@host.domain; Who: Dr. Steals; When: 1 am;” and so forth. Also in the body of the message, the speaker usually precedes both location and starting time, which in turn precedes ending time as in: ‘‘Dr. Steals presents in Dean Hall at one am. ’’ The task is complicated since some fields may be missing or may contain multiple values. This kind of data falls into so-called semi-structured text category. Instances obey certain structure and usually contain information for most of the expected fields in some order. There are two other categories: free text and structured text. In structured text, the positions of the information fields are fixed and values are limited to pre-defined set. Consequently, the IE systems focus on specifying the delimiters and order associated with each field. At the opposite end lies the task of extracting information from free text which, although unstructured, is assumed to be grammatical. Here IE systems rely more on syntactic, semantic and discourse knowledge in order to assemble relevant information potentially scattered all over a large document. IE algorithms face different challenges depending on the extraction targets and the kind of the text they are embedded in. In some cases, the target is uniquely identifiable (singleslot), while in others, the targets are linked together in multislot association frames. For example, a conference schedule has several slots for related speaker, topic and time of the presentation, while a seminar announcement usually refers to a unique event. Sometimes it is necessary to identify each word in a target slot, while some benefit may be reaped from partial identification of the target, such as labeling the beginning or end of the slot separately. Many applications involve processing of domain-specific jargon like Internetese—a style of writing prevalent in news groups, e-mail messages, bulletin boards and online chat rooms. Such documents do not follow a good grammar, spelling or literary style. Often these are more like a stream-of-consciousness ranting in which asciiart and pseudo-graphic sketches are used and emphasis is provided by writing in all-capitals, and using multiple exclamaPosition 1 2 3 4 5 6 7 8 9 10 Tag speaker location s-time Phrase Doctor Steals Presents in Dean Hall at 1 am . Lemma Dr. present in hall at am PoS NNP NNP(VB) VB(NNS) IN NNP NN(NNP) IN CD NN(RB) . Syn.segm. NP NP (VP) VP PP NP NP PP NP NP(VP) Semantic Title LstName LstName Location Time Length 3 6 8 2 4 4 2 2 2 1 Case Upper Upper Upper lower Upper Upper lower lower Table 1: Sample phrase and its representation in multiple feature values for ten tokens. tion signs. As we exemplify below, syntactic analysers easily fail on such corpora. Other examples of IE application domains include job advertisements [Califf and Mooney, 1999] (RAPIER), executive succession [Soderland, 1999] (WHISK), restaurant guides [Muslea et al., 2001](STALKER), biological publications [Ray and Craven, 2001] etc. Initial interest in the subject was stimulated by ARPA’s Message Understanding Conferences (MUC) which put forth challenges e.g. parsing newswire articles related to terrorism (see e.g. Mikheev [1998]). Below we briefly review various IE systems and approaches which mostly originated from MUC competitions. Successful IE involves identifying abstract patterns in the way information is presented in text. Consequently, all previous work necessarily relies on some set of textual features. The overwhelming majority of existing algorithms operate by building and pruning sets of induction rules defined on these features (SRV, RAPIER, WHISK, LP ). There are many features that are potentially helpful for extracting specific fields, e.g. there are tokens and delimiters that signal the beginning and end of particular types of information. Consider an example in table 1 which shows how the phrase “Doctor Steals presents in Dean Hall at one am. ” is represented through feature values. For example, the lemma “am” designates the end of a time field, while the semantic feature “Title” signals the speaker, and the syntactic category NNP (proper noun) often corresponds to speaker or location. Since many researchers use the seminar announcements domain as a testbed, we have chosen this domain in order to have a good basis of comparison. One of the systems we compare to (specifically designed for single-slot problems) is SRV [Freitag, 1998]. It is built on three classifiers of text fragments. The first classifier is a simple look-up table containing all correct slot-fillers encountered in the training set. The second one computes the estimated probability of finding the fragment tokens in a correct slot-filler. The last one uses constraints obtained by rule induction over predicates like token identity, word length and capitalization, and simple semantic features. RAPIER [Califf and Mooney, 1999] is fully based on bottom-up rule induction on the target fragment and a few tokens from its neighborhood. The rules are templates specifying a list of surrounding items to be matched and potentially, a maximal number of tokens for each slot. Rule generation begins with the most specific rules matching a slot. Then rules for identical slots are generalized via pair-wise merging, until no improvement can be made. Rules in RAPIER are formulated as lexical and semantic constraints and may include POS tags. WHISK [Soderland, 1999] uses constraints similar to RAPIER, but its rules are formulated as regular expressions with wild cards for intervening tokens. Thus, WHISK encodes a relative, rather than absolute position of tokens with respect to the target. This enables modeling long distance dependencies in the text. WHISK performs well on both single-slot and multi-slot extraction tasks. Ciravegna [2001] presents yet another rule induction method (LP) . He considers several candidate features such as lemma, lexical and semantic categories and capitalization to form a set of rules for inserting tags into text. Unlike other approaches, (LP) generates separate rules targeting the beginning and ending of each slot. This allows for more flexibility in subjecting partially correct extractions to several refinement stages, also relying on rule induction to introduce corrections. Emphasizing the relational aspect of the domain, Roth and Yoh [2001] developed a knowledge representation language that enables efficient feature generation. They used the features in a multi-class classifier SNOW-IE to obtain the desired set of tags. The resulting method (SNOW-IE) works in two stages: the first filters out the irrelevant parts of text, while the second identifies the relevant slots. Freitag & McCallum [1999] use hidden Markov models (HMM). A separate HMM is used for each target slot. No preprocessing or features is used except for the token identity. For each hidden state, there is a probability distribution over tokens encountered as slot-fillers in the training data. Weakly analogous to templates, hidden state transitions encode regularities in the slot context. In particular prefix and suffix states are used in addition to target and background slots to capture words frequently found in the neighborhood of targets. Ray&Craven [2001] make one step further by setting HMM hidden states in a product space of syntactic chunks and target tags to model the text structure. The success of the HMM-based approaches demonstrate the via