Information Extraction from HTML: Application of a General Machine Learning Approach

Because the World Wide Web consists primarily of text, information extraction is central to any e ort that would use the Web as a resource for knowledge discovery. We show how information extraction can be cast as a standard machine learning problem, and argue for the suitability of relational learning in solving it. The implementation of a general-purpose relational learner for information extraction, SRV, is described. In contrast with earlier learning systems for information extraction, SRV makes no assumptions about document structure and the kinds of information available for use in learning extraction patterns. Instead, structural and other information is supplied as input in the form of an extensible token-oriented feature set. We demonstrate the e ectiveness of this approach by adapting SRV for use in learning extraction rules for a domain consisting of university course and research project pages sampled from the Web. Making SRV Web-ready only involves adding several simple HTML-speci c features to its basic feature set. The World Wide Web, with its explosive growth and ever-broadening reach, is swiftly becoming the default knowledge resource for many areas of endeavor. Unfortunately, although any one of over 200,000,000 Web pages is readily accessible to an Internet-connected workstation, the information content of these pages is, without human interpretation, largely inaccessible. Systems have been developed which can make sense of highly regularWeb pages, such as those generated automatically from internal databases in response to user queries (Doorenbos, Etzioni, & Weld 1997) (Kushmerick 1997). A surprising number of Web sites have pages amenable to the techniques used by these systems. Still, most Web pages do not exhibit the regularity required by they require. There is a larger class of pages, however, which are regular in a more abstract sense. ManyWeb pages come from collections in which each page describes a single entity or event (e.g., home pages in a CS department; each describes its owner). The purpose of such a page is often to convey essential facts about the entity it Copyright c 1998, American Association for Arti cial Intelligence (www.aaai.org). All rights reserved. describes. It is often reasonable to approach such a page with a set of standard questions, and to expect that the answers to these questions will be available as succinct text fragments in the page. A home page, for example, frequently lists the owner's name, a liations, email address, etc. The problem of identifying the text fragments that answer standard questions de ned for a document collection is called information extraction (IE) (Def 1995). Our interest in IE concerns the development of machine learning methods to solve it. We regard IE as a kind of text classi cation, which has strong a nities with the well-investigated problem of document classi cation, but also presents unique challenges. We share this focus with a number of other recent systems (Soderland 1996) (Cali & Mooney 1997), including a system designed to learn how to extract from HTML (Soderland 1997). In this paper we describe SRV, a top-down relational algorithm for information extraction. Central to the design of SRV is its reliance on a set of token-oriented features, which are easy to implement and add to the system. Since domain-speci c information is contained within this features, which are separate from the core algorithm, SRV is better poised than similar systems for targeting to new domains. We have used it to perform extraction from electronic seminar announcements, medical abstracts, and newswire articles on corporate acquisitions. The experiments reported here show that targeting the system to HTML involves nothing more than the addition of HTML-speci c features to its basic feature set. Learning for Information Extraction Consider a collection of Web pages describing university computer science courses. Given a page, a likely task for an information extraction system is to nd the title of the course the page describes. We call the title a eld and any literal title taken from an actual page, such as \Introduction to Arti cial Intelligence," an instantiation or instance of the title eld. Note that the typical information extraction problem involves multiple elds, some of which may have multiple instantiations in a given le. For example, a course page might