THE SIGNIFICANCE OF THE MISSING DATA PROBLEM IN KNOWLEDGE DISCOVERY

Knowledge discovery in databases (KDD) is a field that is enjoying much attention in the literature and rapid growth in algorithms, techniques, and available software. KDD is defined as a non-trivial process which gleans valid, previously unknown, and potentially useful information from stored data [9]. One key area of KDD that has not kept up with this phenomenal growth is the area of data preprocessing. There is a need for research that will l ead to the development of automated tools and techniques in this area. In general, data preprocessing is an unglamorous, tedious, and time-consuming task. Yet, without proper preprocessing, the results of knowledge discovery are suspect. Data preprocessing includes data cleaning, data attribute selection, and dealing with missing attribute values. The problem of dealing with missing values is of significant interest to the KDD research community. This paper will explore common approaches for handling missing attribute values and propose a new approach. 1. Background. Knowledge discovery is an emerging field which has gained in popularity with researchers and practitioners in academia, government, and industry. The need for automated data analysis techniques has grown as the sheer volume of data stored has increased. Knowledge discovery is considered by many as a natural extension of statistical data analysis and artificial intell igence (specificall y machine learning). Technological advances in computer hardware and database technology are also important contributing factors. However, there are many other areas that contribute to knowledge discovery in databases (KDD), including information theory, pattern recognition, neural networks, logic theory, high performance computing and others. The three primary fields that are commonly considered the foundation for KDD are statistics, database technology, and artificial intell igence [12]. Knowledge discovery is defined as a non-trivial process which gleans valid and potentially useful information that was previously unknown from stored data [9]. Knowledge discovery uses automated learning techniques in an efficient and accurate manner to deal with large amounts of data and provide interesting results to the user [8,9]. There are many current applications of KDD, including the following: · Telephone fraud detection [4] · Identification of sky objects [7] · Targeting high credit risks [10] · Identification of faulty network circuits [15] The knowledge discovery process consists of nine tasks as presented in Fayyad, Piatetsky-Shapiro, and Smyth [8]. These tasks consist of first developing an understanding of the data domain (1) and selecting appropriate data (2). Then the data must be preprocessed (3). Frequently it is necessary to perform data reduction and projection (4) to translate the data into an understandable form. The KDD approach (5), discovery algorithm and data parameters (6) must be selected before performing data mining (7). Finally, the discovery results must be transformed into knowledge (8) and documented in a usable manner (9). This research is focused in the area of data preprocessing and in particular on how to resolve the usefulness of records with missing values. Prior to the actual process of mining the data for knowledge, data preprocessing must occur to pare the data to a potentially useful and usable set of data. The nature of data preprocessing is critical, as paring too much data can restrict and hamper the discovery of useful knowledge. Conversely, mining with redundant, missing, or inaccurate data can result in false or misleading results. Quite frequently, database records have missing attribute values, which affect the potential usefulness of individual records. Accordingly, data preparation is a process that is best approached carefully and with reliable techniques. * Department of Computer Science, Mississippi State University, Starkville, MS 39762. pwright@cs.msstate. edu