Corpus-Driven Knowledge Acquisition for Discourse Analysis

plg /9 40 60 16 7 Ju n 94 Proceedings of the Twelfth National Conference on Arti cial Intelligence, 1994 Corpus-Driven Knowledge Acquisition for Discourse Analysis Stephen Soderland and Wendy Lehnert Department of Computer Science University of Massachusetts Amherst, MA 01003-4610 soderlan@cs.umass.edu lehnert@cs.umass.edu Abstract The availability of large on-line text corpora provides a natural and promising bridge between the worlds of natural language processing (NLP) and machine learning (ML). In recent years, the NLP community has been aggressively investigating statistical techniques to drive part-of-speech taggers, but application-speci c text corpora can be used to drive knowledge acquisition at much higher levels as well. In this paper we will show how ML techniques can be used to support knowledge acquisition for information extraction systems. It is often very di cult to specify an explicit domain model for many information extraction applications, and it is always labor intensive to implement hand-coded heuristics for each new domain. We have discovered that it is nevertheless possible to use ML algorithms in order to capture knowledge that is only implicitly present in a representative text corpus. Our work addresses issues traditionally associated with discourse analysis and intersentential inference generation, and demonstrates the utility of ML algorithms at this higher level of language analysis. The bene ts of our work address the portability and scalability of information extraction (IE) technologies. When hand-coded heuristics are used to manage discourse analysis in an information extraction system, months of programming e ort are easily needed to port a successful IE system to a new domain. We will show how ML algorithms can reduce this development time to a few days of automated corpus analysis without any resulting degradation of overall system performance. 1. Information Extraction at the Discourse Level All IE systems must operate at both the sentence level and the discourse level. At the sentence level, relevant information is extracted by a sentence analyzer according to pre-de ned domain guidelines. Recent performance evaluations sponsored by ARPA have shown This research was supported by NSF Grant no. EEC9209623, State/Industry/University Cooperative Research on Intelligent Information Retrieval. that a number of di erent parsing strategies can handle sentence-level information extraction with varying degrees of success (Lehnert and Sundheim 1991, Sundheim 1991). This paper will concentrate on the discourse level, by which we mean all processing that takes place after sentence analysis. Once information has been extracted locally from various text segments, the IE system must make a series of higher-level decisions before producing its nal output. Multiple referents must be merged when they are coreferent, important relationships between distinct referents must be recognized, and referents that are spurious with respect to the IE application must be discarded. To get a sense of the decisions involved in discourse, consider the following fragment of a text from the MUC-5 micro-electronics domain.