Ranking Text Units According to Textual Saliency, Connectivity and Topic Aptness

An efficient use of lexical cohesion is described for ranking text units according to their contribution in defining the meaning of a text (textual saliency), their ability to form a cohesive subtext (textual connectivity) and the extent and effectiveness to which they address the different topics which characterize the subject mat ter of the text (topic aptness). A specific application is also discussed where the method described is employed to build the indexing component of a summarizat ion system to provide both generic and query-based indicative summaries. 1 I n t r o d u c t i o n As information systems become a more integral par t of personal computing, it appears clear that summarization technology must be able to address users' needs effectively if it is to meet the demands of a growing market in the area of document management . MinimaUy, the abridgement of a text according to a user's needs involves selecting the most salient portiolts of the text which are topically best suited to represent the user's interests. This selection must also take into consideration the degree of connectivity among the chosen text portions so as to minimize the danger of producing summaries which contain poorly linked sentences. In addition, the assessment of textual saliency, connectivity and topic aptness must be computed efficiently enough so that summa* This work was carried out within the Information Technology Group at SHARP Laboratories of Europe, Oxford, UK. I am indebted to Julian Asquith, Jan I Jdens, Ian Johnson and Victor Poznaliski for helpful comments on previous versions of this document. Many thanks also to Stephen Burns for internet programming support, tlalf St.einberger for assistance in dictionary conversion, and Charlotte Boynton for editorial help. rization can be conveniently performed on-line. The goal of this paper is to show how these objectives can be achieved through a conceptual indexing technique based on an efficient use of lexical cohesion. 2 B a c k g r o u n d Lexical cohesion has been widely used in text analysis for the cornparative assessment of saliency and connectivi ty of text fragments. Following Ho W (1991), a simple way of computing lexical cohesion ill a text is to segment tile text into units (e.g sentences) and to count non-stop words I which co-occur in each pair of distinct text units, as shown in Table 2 for the text in Table 1. Text units which contain a greater number of shared non-stop words are more likely to provide a bet ter abridgement of the original text for two reasons: • the more often a word with high informational content occurs in a text, the more topical and germane to the text the word is likely to be, and • the greater the number of t imes two text units share a word, the more connected they are likely to be. Text saliency and connectivi ty for each text unit is therefore established by summing tile number of shared words associated with the text unit. According to Itoey, the number of links (e.g. shared words) across two text units xnust be above a certain threshold for the two text units to achieve a lexical cohesion rank. For example, if only individual scores greater than 2 1Non-stop words can be intuitively thought of as words which have high inforinational content. They usually exclude words with a very high fequency of occurrence, especially closed class words such as determiners, prepositions and conjmmtions (Fox, 1992).