Automatic text indexing using complex identifiers

Abstract: Automatic text analysis methods are available that can assign identifying elements, or terms, to written documents, based on statistical and other formal criteria. Attempts to refine the established single-term indexing methods by using more complex syntactic and/or semantic methods have been unsuccessful. The problems relating to the identification and use of complex content identifiers are examined, and the available syntactic indexing methodologies are evaluated. Automatic text analysis methods are available that can assign identifying elements, or terms, to written documents, based on statistical and other formal criteria. Attempts to refine the established single-term indexing methods by using more complex syntactic and/or semantic methods have been unsuccessful. The problems relating to the identification and use of complex content identifiers are examined, and the available syntactic indexing methodologies are evaluated. Automatic Text Analysis During the past twenty years, a good deal has been learned about automatic text processing. From a practical point of view, the well-known word processing applications are probably the most influential, and the most widely used. This includes text editing and text correction, document preparation and formatting, spelling error detection and correction, and style and syntax monitoring systems. Less well known are the automatic text analysis systems designed to generate content descriptions that reflect document content. Such descriptions may serve as document surrogates in text storage and retrieval systems; in addition, they are useful for the generation of derivative products such as text indexes, and text abstracts and summaries. *This study was supported in pan by the National Science Foundation under grant IR187-02735. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. © 1988 ACM 0-89791-291-8/88/0012/0135 $1.50 Several approaches to text analysis may be distinguished, including the statistical analysis approaches in which the occurrence characteristics of the text words are used to determine word function and word importance indicators; the syntactic processing techniques used to detect text structure and syntactic constructions; and finally the semantic processing methods based mostly on information extracted from available dictionaries, thesauruses, and knowledge bases. Statistical methods are useful in distinguishing common function words occurring in the texts from other text elements that are more directly related to document content. In addition, statistical word occurrence characteristics are also used to generate term importance parameters and term weights. Syntactic analysis approaches, on the other hand, are used for the identification of related word groupings, including especially the nominal constructions that often help reflect document contenL Finally, semantic approaches are needed to eliminate potential ambiguities in the statistical and syntactic text analysis methods. Various simple text analysis approaches are introduced in this note, and illustrations are given of their use in automatic document processing. Text Indexing Theories The task of assigning identifying content indicators, or terms, to text items is known as indexing. The simplest approach to text indexing consists in using words occurring in the texts of individual documents for content representation. The following procedures may be used for this purpose: [1-3] 1. Take each document and identify the individual words occurring in the document. 136 ACM CONFERENCE ON DOCUMENT PROCESSING SYSTEMS 2. Use a dictionary of common function words, also known as a stop list, to delete from the texts the high-frequency words that are insufficiently specific for content representation (and, of, or, but, the, etc.). 3. Use a suffix stripping routine to reduce each remaining word to word stem form; this reduces to a common form all words exhibiting the same word stem (analysis, analyzer, analyzing, etc. would all be reduced to the common form ANALY). 4. For each remaining word stem compute a term weight reflecting term importance. [4,5] 5. Represent each document Di by the set of word stems together with the corresponding term weights. By following this process, a typical document Di will be represented as a term vector, consisting of sets of weighted word stems as follows: Di = (Tl ,Wil; T2,wi2; • "" ; Tt,wit) (I) where Wij represents the weight of term Tj in document Di. Various term weighting theories have been proposed. The term-discrimination approach postulates that useful terms for content representation are those distinguishing individual documents from the remaining documents in a collection. This implies that the best terms ate those which occur frequently in individual documents but rarely in the collection as a whole. Following this approach, the term weights might be computed as the product of the term frequency, ~fij (the frequency of occurrence of term Tj in document Di), and a document frequency factor, such as log N/nj, where N is the collection size, and nj is the number of documents in the collection to which term Tj is assigned. The factor log N/nj is inversely related to the postings frequency nj of term Tj and has previously been related to the notion of term specificity. [5] The corresponding term weighting system is known as ( tf ' idf) , that is, term frequency times inverse document frequency. [6] It is known that a document indexing system which assigns sets of properly weighted index terms to the documents of a collection can produce a high order of retrieval effectiveness. [7] Although the use of large sets of weighted single terms may in fact be adequate for automatic text processing, single terms are not ideal in environments where human beings take an active part in the processing sequence. In such cases, a richer vocabulary is needed, consisting of entries that reflect content in a more meaningful way. Thus, book index entries normally consist of term phrases, or terms in context, because complex entries are easier to assimilate than single-term entries used without context. Many approaches have been tried for the automatic generation of multi-term indexing units. Among these are so-called associative indexing methods where initially available terms lead to sets of related terms [8]; probabilistic indexing methods based on the occurrence probabilities of sets of terms in the relevant and nonrelevant document subsets of a collection [9,10]; and methods based on the use of vocabulary schedules, term association maps, and thesauruses to identify the term relationships. [11] Some of these multi-term indexing methods are briefly outlined in the next section. Associative Indexing Methods The usefulness of automatic text processing systems is often measured by computing the system recall and precision, where the recall reflects the ability of the system to find materials that are deemed relevant to the user needs, and the precision measures the capability of the system to reject the extraneous items. Useful text indexing and retrieval systems are expected to exhibit • both a high recall and a high precision. Associative indexing systems have been introduced in the hope of improving both the recall and the precision performance of text processing systems. Two kinds of term relationships are normally considered: word grouping techniques capable of assembling similar, or related, terms into affinity classes, making it possible to assign complete word classes instead. : single terms to the query and documents texts; and phrase generation methods capable of assigning structured identifiers to the texts instead of terms out of contexL A thesaurus provides a term schedule which can be used as an authority list to distinguish allowable indexing units from the remainder of the vocabulary. In addition, thesauruses are used as word grouping devices cap,able of displaying related, or synonymous terms (as in Roget's thesaurus). Thesauruses have been introduced in text processing to reduce the variability of the vocabulary used for content identification, and to supply related terms in addition to the originally available ones. There is some evidence that good thesauruses that fit the subject matter of particular document collections will improve the recall performance by 10 to 20 percent. [12] However, the construction of a thesaurus is onerous, and a consensus does not exist about the best way of constructing useful thesauruses that remain valid for subject areas of reasonable scope. A somewhat more modern interpretation of a thesaurus is as a term-association map which provides information about the concepts of interest in a given area, and about certain relationships between these concepts. [13] An excerpt from a typical term association map is shown in Fig. 1. [14] SANTA FE, NEW MEXICO. DECEMBER 5-9° 1988 137