Algorithm for Outlier Detection Based on Utility and Clustering (ODUC)

Outlier analysis is one of the applied data mining technique. Outliers are data objects which do not comply with the general behavior or model of data. Statistical approach, distance-based approach, deviation-based approach are some of the outlier detection methods. Clustering data mining technique groups similar data objects into clusters, which indirectly eliminates outliers as noise. Not all the outliers are important for business improvement, this in turn gives birth to utility mining, which considers interestingness of the user while applying any data mining technique. The proposed system is to find outliers based on utilities and k-means clustering. i.e., first pruning the data objects whose utility value is lesser than user's minimum threshold value and the second step is to employ repeated k-means clustering. The repeated k-means clustering at each iteration prune's the data objects which lies near the centroid of the cluster. Outliers are deviated data objects which shows different characteristics and does not fits to any category of data objects. outliers could be caused by measurement or execution error or by inherent data variability. Outlier mining has wide applications. It can be used in fraud detection for finding unusual usage of credit cards or telecommunication services, in drug discovery, or in medical analysis for finding unexpected response to certain treatment. Outlier detection techniques can be classified as mainly 3 types, which are Distance-based, Deviation-based and Statistical approach. The statistical approach assumes the model underlying distribution that generates data set and then identifies outliers using a test called discordancy test. Distance-based approach is the most widely used technique which generalizes other notions provided by numerous discordancy tests and replaces many of these tests with a single algorithm detecting only outliers of the proposed types. Deviation-based techniques uses linear algorithm for deviation detection using implicit redundancy of the data. Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group or cluster are more similar to each other than to those in other groups or clusters. K-means clustering is the most widely used clustering algorithm. The data objects which does not fits to any clusters are termed as outliers. Clustering eliminates outliers as noisy data"s. Not all the outliers are important for business development ,and so we need to include interestingness of the user. For instance cyber fraud detection of detecting steal attacks is important than detecting virus attacks. and here we proposed the algorithm ODUC (Outlier detection based on Utility and Clustering) for outlier detection based on k-means clustering and consideration of utility of the user. The proposed algorithm consists of two phases. The first phase prunes the data objects whose utility value is lesser than user"s minimum threshold value. And the second phase consists of iterative k-means clustering. We apply K-means clustering to divide the data objects into clusters. The points or data objects which are lying near to centroid of the cluster are not probable candidate for outlier because they are clusters with strong similarity property and we can prune out such points from each cluster. Next, we calculate a distance-based measure for all remaining points, which is then used as a parameter to identify a point to be an outlier or not. We assume that there are n outliers in data set, and top n points will be reported as outliers by our method.