ANNIE ’ 2000 November 5-8 , St . Louis , Misouri 1 / 6 EFFECT OF SAMPLING SIZE ON DATA MINING USING ARTIFICIAL NEURAL NETWORKS

Artificial Neural Networks (ANNs) represent a useful technique for data mining applications. They can be trained to properly represent various categories occurring in a data set. In large databases, and data warehousing techniques, the size of data sets can be huge which may result in inefficient ANNs learning. Thus, it is useful to find an efficient and practical training set size without compromising the results. This paper presents experimental results highlighting the effect of varying the sampling size used in training Artificial Neural Networks and demonstrates that the extra effort used in expanding the training set is not linearly proportional to the improved accuracy. These results are important, and they are currently validated on a variety of domains. INTRODUCTION Two important issues play a major role in the learning process of an ANN. The first one is the proper selection of the data set size, and the second is the sampling distribution of the data set (samples represent positive and negative examples in supervised learning). Many researchers have highlighted the importance of having adequate data to train, test and validate ANNs (Zurada 92). Data Mining applications have traditionally used inductive logic (Quinlan 86, Michalski et al 83, Freitas 99b) to extract rules describing various interesting conditions that occur in the data set. ANNs have been demonstrated to be an effective tool for mining large data sets and to extract relevant knowledge (Freitas and Lavington 99a). Although many researchers do not associate the use of ANNs with rule extraction, successful results in extracting logic-oriented rules from the weights of a trained ANN have been demonstrated for a specific type of domains (Kantardzic et al 97). ANNs are used for data mining with the perspective of training and testing using a limited data set. A trained ANN is then used in conjunction with additional or unseen data sets for classification based upon its previous training phase (Zurada 92). The ability of an ANN to generalize its learning is dependent upon the proper selection of its training and testing data sets. Generalization features of ANNs are not trivial since in some conditions Neural Network gaps can occur (Kantardzic et al 99) and in other cases, ANNS can be over trained and reach a saturation level that increases its memorization ability rather than its generalization ability. In this paper, we are presenting explorations of the effect of using different data set sizes and different sampling rates for positive examples, to train and test ANNs for data mining applications. The results reported in this paper are based upon using feed forward ANN that has been trained with Back-propagation learning algorithm (supervised learning). ANNIE’2000 November 5-8, St. Louis, Misouri 2/6 METHODOLOGY A large data set of over 48,000 records has been used in this research. Two sets of experiments have been implemented: 1 – Experiment set 1: Testing the effect of the data set size on the accuracy of the trained network. 2 – Experiment set 2: Testing the effect of varying the ratio of positive to negative samples on the achieved accuracy. The data set is a representative of complex data types and include 15 attributes of type numeric, symbolic and logic. The data represent the census domain, which has been used earlier to test the performance of other data mining techniques (Kohavi 96). It is desired to classify the examples into two classes based upon 14 attributes with minimum classification error. The two classes are: 1) earnings greater than or equal 50K dollars/year (considered as class1), and 2) earnings below 50K dollars/year (class2). The data set consists of a number of instances of each class. Typically, 37,155 for class1 and 11,687 for class2 totaling 48,842 samples. Examples supporting the first class are referred to as positive examples, while examples supporting the second class are referred to as negative (or counter) examples. It should be noted that, there is no one single rule that can divide this domain into two strict classes due to the complexity of the data and the presence of noise. An initial experiment was conducted to select the most relevant attributes (features) to the mining task in this domain. In general, feature selection reduces the domain dimensionality, ANN size, generalization errors, and the data mining time. Using this approach, only relevant attributes are used for training the ANN (Liu and Motoda 98). The experiment was carried out by testing the sensitivity of the ANN to each attribute in the data set. The ANN was trained with the full domain attributes until the highest predicted accuracy was achieved. Then, the sensitivity of the predicted accuracy to all attributes was computed and attributes that have a sensitivity value higher than a certain threshold were used for training the actual ANN. This method was applied to different ANN topologies and different threshold settings to obtain the best set of features that represent the domain efficiently. It was found out that the data set was sensitive to all the attributes, which means that we cannot consider any attribute as redundant and neglect it in the training phase. Following the initial examination of the data set, several systematic experiments have been conducted. Experiment set 1: In this set, the size of the training sets was set to a minimal size of 12 cases initially. Then it was incremented all the way up to the full size of 48,842 using a geometric sequence without changing the percentage of entries that describe class1 to those describing class2 (i.e. the ratio between positive and negative examples was kept constant). Then, trained ANNs for each data set is tested using the complete set of examples. The individual entries in the training sets were selected at random from the available examples. The training set sizes used in this experiment and the corresponding testing accuracy are shown in Table 1. It should be noted that three experiments were conducted for each data set size. The average testing accuracy was then computed. The results of the trained ANN for each data set are summarized in Fig. 1. Experiment set 2: In this set of experiments, the data set was formed by fixing the number of negative samples (examples) to a constant value (145) and then, varying the number of positive examples starting from 1 as shown in Table2. The number of negative examples has been chosen experimentally based upon the results of the first experiment. The results of experiment set 2 are shown in Fig. 2. ANNIE’2000 November 5-8, St. Louis, Misouri 3/6 Table 1. Number of positive and negative examples in each data set No. of positive examples No. of negative examples Training set size Average Testing Accuracy (%) 9 3 12 75.59 18 6 24 79.13 36 12 48 79.99 72 24 96 79.25