Building a Machine Learning Based Text Understanding System

Text understanding systems are approaching the point of being a practical technology as long as the system is trained for a narrowly defined domain. Machine learning and statistical approaches can minimize the effort involved in adapting a text understanding system to a new domain. This paper presents a system whose goal is deep understanding, limited only by the necessity of designing a formal representation of the target concepts relevant to the domain. This system is an advance over previous machine learning based systems because of its richer output representation, and an advance over equally expressive text understanding systems because of its more extensive use of machine learning. 1 Information Extraction from Free Text A variety of systems have been developed in recent years that extract information from text. None of them attempts general-purpose unders tanding, but instead focus on narrowly defined information needs. A domain is defined as a collection of documents and a clear definition of the target concepts for that domain. Some systems look for fairly simple types of information such as scanning seminar announcements for time, location, and speaker [Freitag, 1998], on-line job postings for position title, location, skills needed, and salary [Califf and Mooney, 1997], or rental ads for number of bedrooms, price, and neighborhood [Soderland, 1999]. Everything else in the document is ignored as irrelevant. Information systems that aim for a deeper understanding have come out of a series of DARPA sponsored Message Understanding Conferences. These systems must still be adapted to one domain at a time, but have a richer representation of the information relevant to the domain. One example is the terrorism domain from the MUC-4 conference [1992], which consisted of news stories about Latin American terrorism and represented the target concepts with an output template for each specific event with details about the perpetrators, victims, and physical targets. Another example is the joint ventures domain of the MUC-5 conference [1993] where the domain consisted of news stories about joint business ventures and the target concept was details about the companies involved, the business activity, and the ownership of the joint venture. Work on medical text understanding systems also aim at deep understanding. Examples of this are the work of Friedman et al. [1997, 1994] and of Taira and Soderland [2001], both groups initially developing their systems in the domain of radiology reports. The target concept is the findings mentioned in the report and details about their properties: such as existence, size, location, clinical trend, and interpretation. The assumption is that everything in a radiology report is relevant information, so the output representation must be deep enough to capture as much of the detail and nuances as possible. Figure 1 shows an example from Taira and Soderland’s system, a portion of a radiology report and output for one of the findings mentioned, a mass lesion in the right upper lobe of the lung. Properties extracted are the existence (present), quantity, location, size, and size trend (increasing). Note that some field values, shown in all capital letters, have been converted to a normalized form, and the anatomical locations that have been replaced with preferred terms from a medical controlled vocabulary. How close is all of this to what could be called deep understanding of the text? The limit of these systems is primarily the need to explicitly design an output representation in advance. The system can only map free text to a formal representation to the extent that the representation can express the concepts in the text. Anything else in the text must necessarily be ignored. Thus in the terrorism domain, if a story mentions the color of the get away car or the political affiliations of a kidnapping victim, these will be ignored. In the joint ventures domain, the name of the company spokesman is ignored. If these concepts were considered important, the system designers would need to include color-of-car or spokesman as part of the output representation. The two main challenges for automatic text understanding are creating a sufficiently expressive output representation for the target concepts of a domain, and then acquiring the domain-specific knowledge of how those concepts are represented in free text. This knowledge may include a Figure 1. A portion of a radiology report and one of the semantic frames extracted from it. glossary with word senses appropriate to the domain, patterns of syntactic word attachments, and semantic patterns of ways that target concepts are expressed. Efficient methods of knowledge acquisition are needed if text understanding systems are to be of practical use. Human effort can be minimized with a domainindependent system that uses machine learning and statistical techniques to acquire domain-specific knowledge. A system developer or end user creates sets of hand-tagged training examples from which the system automatically derives syntactic and semantic patterns. This is less labor intensive than creating complex rules by hand and can capture the subtlety and wide variation found in natural language. Some of the shallower extraction systems are entirely machine learning based. Freitag [1998] uses a combination of machine learning classifiers; Califf and Mooney [1997], Soderland [1999], and Freitag and Kushmerick [2000] use a variety of rule learning methods; Freitag and McCallum [2000] use hidden Markov models. Each of these systems finds extraction patterns in a single step with no explicit syntactic parsing, and operate best on semi-structured text rather than free narrative. Ciravegna [2000] obtains good results by learning a set of rules, then augmenting and correcting the rules in a multi-step system. The medical text understanding system of Friedman et al. [1997, 1994] uses a manually constructed semantic grammar specifically designed for the domain of radiology reports. Most systems developed for the MUC conferences use considerable hand coding of domain knowledge, such as syntactic/semantic extraction patterns. One MUC system that used machine learning was that of the University of Massachusetts [Lehnert et al., 1993] which derived extraction rules from training. BBN [Miller et al., 1998] went a step further and trained a statistical model to handle both full syntactic parsing and identifying semantic relations between constituents in the parse tree. Both systems used domain-specific code to filter, merge, and format output from the machine learning system. The work presented here combines extensive use of machine learning with an expressive output representation that moves information extraction in the direction of deep understanding. The remainder of this paper describes the system and discusses issues in designing an output representation and in learning semantic interpretation rules to map free text into that representation. 2 Steps in Text Analysis Our text understanding system has six main steps: 1. Structural analyzer 2. Lexical analyzer 3. Syntactic parser 4. Semantic interpreter 5. Frame builder 6. Coreference resolution The structural analyzer uses a maximum entropy classifier [Ratnaparkhi, 1998] to determine the sentence boundaries after dividing the document into sections based on page layout cues. The lexical analyzer looks up each word of the sentence in a domain-specific glossary. This assigns syntactic and semantic features to each word that are used later by the parser and the semantic interpreter. A glossary with about 5,000 entries was manually constructed for the domain of thoracic radiology reports. In addition to glossary look-up, the lexical analyzer uses rules to interpret dates and numerical expressions such as “1.2 x 1.7 cm”. The syntactic parser creates a dependency graph with an arc from each word to the word it modifies. We use a statistical parser somewhat similar to that of Collins [1996] and of Eisner [1996], in which the probability of an arc between each pair of words is estimated from hand-tagged training sentences. Dynamic programming is used to compute the parse graph with maximum joint probability, subject to some constraints on well formedness of the graph. The next step of processing, semantic interpretation, is described in detail later. It learns a set of semantic rules from hand-tagged training examples using a covering algorithm [Quinlan, 1990; Clark and Niblett, 1989]. Each rule traverses nodes in the parse graph, testing semantic and syntactic features. When a rule succeeds, it creates a semantic relation for a pair of phrases in the sentence. Figure 2 shows an example of a dependency graph for the sentence “An approximately 1.0 cm nodule is noted” and the semantic relations derived from the sentence. The arc from “noted” (word 5) to “nodule” (word 3) is interpreted as an Existence-value relation with the value of PRESENT . The arc from “1.0 cm” to “nodule” is a Size-value relation, while the arc from “approximately” to “1.0 cm” is a Size-precision relation with the value of APPROXIMATE. Figure 2. Semantic relations derived from a parse graph The following step is the frame builder that takes a set of semantic relations between pairs of words and assembles them into semantic frames. In the domain of thoracic radiology, the topic of each frame is a medical finding with a list of its properties. Figure 1 shows an example of a semantic frame with Existence, Quantity, Size, Location, and Size trend. The last step is coreference resolution that tracks references to a frame topic across sentences. For each finding, the probability is estimated that it co-refers to each previously mentioned finding based on semantic features and lexical cues (e.g. “the nodule” vs. “a nodule”). The current version of the coreference module uses hand-code rules to compute probabilities, although we intend to derive probabiliti