Holistic Information Retrieval Through Textual Data Mining

Information retrieval can be likened to a mining process. Searchers drill through a document space using keywords to extract document subsets. These subsets must be reviewed to extract the topically relevant documents from the irrelevant. Searchers interactively learn from the relevance of the document subsets and resubmit more search arguments to perhaps narrow the search to obtain more relevant documents or broaden the search to improve the likelihood of recalling the highest percentage of relevant documents from the document space. Searchers may be aided in the search by using a document organization scheme used by human categorizers to organize the document space. Such schemes, such as the Library of Congress Classification System, tend to be rigid and dated. Documents are greatly increasing and number and organizational schemes such as the LCCS are not adapting well to the varying content of books and documents being added to the document space. What is needed is an automatic mapping tool that 1) takes the document space as it is, 2) creates a conceptual map of the space, and 3) clusters like documents and places them together on the map. This research (in progress) is an attempt to determine the value of the Kohonen Self-Organizing Map (SOM) (Kohonen, 1995) for use as an interactive textual data mining tool for categorization of large sets of documents. The SOM algorithm analyzed 339 Management Information Systems Quarterly abstracts from 1985 to 1997. The first analysis resulted in a map of two major regions -Information and Systems. This demonstrated that the SOM was working correctly but produced a potentially uninteresting map. What may be more interesting is the next level of conceptual detail, i.e., the major conceptual areas of the MISQ document space below this high level of abstraction. To obtain this map, "management," "information," and "systems" was added to a stop-word list and the Kohonen algorithm was reapplied to obtain a mapping of the MISQ literature at this second level of detail below Management Information Systems. At both levels of abstraction, the 339 abstracts were partitioned among the conceptual regions. This suggests the possibility for an interactive tool that aids searchers in exploring large document spaces by using a "divide and conquer" approach of information retrieval whereby the tool clusters similar documents into topical regions of the map for exploratory browsing. Introduction Typical searches for information retrieval involve sending a search argument of key terms into a document space to retrieve a subset of those documents. The searcher judges the relevance of each returned document to the topic of the search. Even with more sophisticated Boolean searching, the process amounts to sending very specific terms into the document space in hopes of documents using exactly those terms in discussing relevant documents such that a match occurs and those documents are retrieved. The quality of the returned set of documents varies with respect to relevance of each of the documents to the topic of the search. The quality of the complete search session varies with respect to whether every relevant document in the document space is actually collected through multiple queries. Categorization schemes may be created to organize the document space. The most common is the Library of Congress Classification System used by many libraries to organize book collections by subject area. These schemes are difficult to create and maintain as new topical areas come into being. Further, more and more documents are created daily as new information sources such as the internet support them. The need is for a tool that automatically organizes a non-organized document space into conceptual (topical) regions as well as clusters like documents into their appropriate conceptual regions. This paper describes the application of the Kohonen Self-Organizing Map (SOM) to 339 Management Information Systems Quarterly abstracts from the years 1985 to 1997. The next section describes the SOM and how it was applied to the MISQ abstracts. The third section describes the results. Section Four contains conclusions and suggestions for further research work. 2. Kohonen Self-Organizing Map Neural network classifiers can be organized into two major types based upon the method with which they "learn:" supervised or unsupervised (Lippmann, 1987). Supervised neural networks use a set of data with known attributes and categories to train the network. A backpropagation neural network is an example that uses a supervised learning technique. Once the network is trained with known data new data can be presented to it for classification. An unsupervised learning technique lets the data itself determine the attributes upon which it will organize. The Kohonen Self-Organizing Map (SOM)