HTML Tags as Extraction Cues for Web Page Description Construction

Introduction The background to the research reported on in this paper is the author's ongoing programme aimed at developing a computerized abstractor's assistant (Craven, 1988 ; Craven, 1991; Craven, 1993 ; Craven, 1996 ; Craven, 1998a ). In addition to a simple word processor and other general writer's tools, the package integrates tools, such as an automatic extractor, related specifically to the task of summarizing. Apart from the author's own work, Paice ( 1994 ) has given a list of desirable features for such a package. A hybrid system, in which some tasks are performed by human abstractors and others by software, appears to be an appropriate short term goal, since purely automatic abstracting methods (Endres-Niggemeyer, 1998 ; Paice, 1990 ; Paice, 1994 ; Pinto & Galvez, 1999) do not show immediate promise of totally superseding human effort. With a view specifically to applying the computerized assistant to summarizing Web pages, this development work has lead to an investigation (Craven, 2000) into how people and organizations in fact summarize their own Web pages. An underlying assumption has been that author created descriptions will will reflect features that authors and other users may consider desirable. It is not assumed thereby that all descriptions reflect the same desirable features, nor that all authors consider the same features desirable, nor even that all descriptions contain only desirable features. The investigation has focused to date on descriptions stored on the pages themselves in meta tags. More exactly, the descriptions studied are those that appear as values of the "content" attribute of HTML "meta" tags the value of whose "name" attribute is "description". Variations on this pattern, such as making "description" the value of the "http-equiv" attribute or "dc.description" the value of the "content" attribute have been found to be very rare and have therefore been ignored. "Meta" tags are supposed to be found in the "head" element in HTML, but some ill-formed pages may include them without specifying a "head" tag. Other aspects of the content of Web pages have been studied by various researchers; for example, page layout of home pages (King, 1998); characteristics of anchors (Haas and Grams, 2000); informetric measures (Almind & Ingwersen, 1997); links to e-journals and their articles (Harter and Ford, 2000); the feasibility of using implicit structure in identifying differences automatically (Rahardjo and Yap, 2001); rates of change over time (Koehler, 2002). Little investigation has been done into descriptions in meta tags. Turner and Brackbill (1998) have, however, reported results of a small experiment that showed that addition of a description did not improve retrievability of Web pages on Infoseek and Altavista; similar results have been reported for these two search engines and five others by Hensha w and Valauskas (2001); Drott (2002) has noted the extent to which descriptions and keyword meta tags are used in the sites of 60 Fortune Global 500 companies. Another article by the author (Craven, 2001) has reviewed advice given in both printed and Web-based sources on the function, content, structure, and style of meta tag descriptions. Some of the main recommendations found in that review are that the description can be used for an abstract; that it should be no more than about 200 characters (about 30 words or one sentence, the typical maximum length for an abstract of a letter to the editor); that (like an abstract) it should be concise and not be the same as the title. The author's studies to date have revealed from 27% to 38% of Web pages containing meta tag descriptions, depending on the method of sampling. Some of the descriptions greatly exceed typical length guidelines. From 7% to 10% duplicate exactly phrasing found in the text of the body and most repeat some words and phrases. Meta tag keywords are slightly more likely to appear nearer the beginning of a description than nearer the end. …