Text Analysis and Automatic Indexing

Figure 3-8 Typical sentence extracting system. (Adapted from reference 39.) TEXT ANALYSIS AND AUTOMATIC INDEXING 89 lection or rejection. Contextual inferences may be based on sentence or word location or on the presence of so-called cue words. Thus, sentences occurring under certain headings may be particularly important; the same may be true of sentences occurring very early or very late in the paragraph structure. Similarly sentences ending with a question mark are normally rejected, as are certain sentence portions occurring between pairs of commas. The cue method is based on the presence of positive or negative indicators of sentence value. Thus, the presence of phrases such as “our work,” “this paper,” and “the present research” is assumed to introduce a statement that should be included in an abstract. Contrariwise, opinions and references to fig­ ures or tables that might be identified by “obvious,” “believe,” “fig. 1,” etc., lead to sentence rejection. To operate effectively, the cue method depends on the availability of a dictionary containing the cue words together with indica­ tions of their semantic and/or syntactic value. Such a cue word dictionary may be particularly valuable for sentence rejection, as opposed to selection, be­ cause rejection can often be based on the presence of a small number of fre­ quently occurring words, whereas selection may depend on longer lists of de­ sirable words. No matter how sophisticated the extraction process, a set of extracted sen­ tences is not likely to constitute a coherent whole. Even if the extract in fact consists of appropriate topic sentences, the flow of ideas from one sentence to the next is likely to be interrupted because the discourse and reasoning leading to, or following from, the selected sentences is probably absent. Coherence criteria based on syntactic or semantic considerations can be used to mitigate to some extent the shortcomings inherent in an automatic ex­ tracting process. Thus words or phrases indicating intersentence reference can be included in the cue dictionary (“these,” “they,” “it,” “above,” “pre­ sented earlier,” “ stated above,” etc.), and their presence in a given extract can be used as a clue for inclusion in the abstract of the earlier sentences being re­ ferred to. Similarly, if the same important terms occur in adjacent sentences, a presumption exists that the sentences are related, and both should probably be included or excluded [41]. No matter what is done, a stylistically beautiful abstract is not likely to be created automatically because the linguistic difficulties are simply too severe for an effective automatic treatment. For example, the clarification of ambigu­ ous antecedents such as those of certain pronouns is notoriously difficult. Some of the linguistic problems are further discussed in the next section together with other theoretical questions. The overall conclusion is that automatic abstracting is less developed than automatic indexing and less likely to be used on a production basis in the near future. Abstracts must be placed in a readable natural language context and must obey the normal stylistic constraints. Sets of