GA-MINER : Parallel Data Mining with Hierarchical Genetic Algorithms Final Report

Many organisations now routinely gather vast and ever-increasing amounts of data in the ordinary course of their business. While much of this information is collected for day-to-day operational reasons, many businesses are now realising that this data has much additional value for improving operational processes. Large databases can form the basis of decision support systems, often based around a data warehouse. Such systems may then be used for a variety of applications such as trend spotting, pattern recognition, behavioral modeling and customer worth assessment. Against this backdrop, the termdata mining is used to refer to the process of searching through a large volume of data to discover interesting and useful information. The authors have traditionally sought to divide data mining into three types or levels— undirected or pure data mining, where the system is left almost entirely unconstrained to discover patterns in the data free of prejudices from the user; directed data mining, where the user may specify some constraint to “steer” the system through its search; and hypothesis testing and refinement, where the user poses some hypothesis and the system first evaluates the hypothesis and then seeks to refine it. The system specified and built during this project—GA-MINER—is unique in being applicable to all three levels of data mining. Although undirected data mining has been the defining goal, the system has been deliberately built to enable also directed data mining and hypothesis refinement. The project has succeeded in its initial goal to implement a parallel genetic-algorithm based data mining tool for rule discovery and has maintained the principle that discovered knowledge must be scrutable and understandable. Performance results have shown that the system is scalable on both Symmetric Multi-processor (SMP) and Massively Parallel Multi-processor (MPP) systems. Fast and flexible development of GA-MINER was made possible by its implementation in the Reproductive Plan Language (RPL2), an extensible language, interpreter and run-time system for the implementation of stochastic search algorithms, with a special emphasis on evolutionary algorithms such as genetic algorithms. A central component of GA-MINER is the pattern template, which defines the general form of the patterns of “interest” to the user and restricts the search space to those patterns which are consistent with this form. A prototype graphical user interface has also been implemented to allow users to manipulate the pattern template and to visualise the results of the search. One of the biggest challenges in this project has been to produce a system that can differentiate between interesting and uninteresting patterns. The project considered several evaluation functions for estimating pattern interest, mainly based on statistical measures, and our experience has shown that many of these evaluation mechanisms give highly similar results. While falling short of providing fully satisfactory definitions of interesting patterns, the evaluation functions have been sufficiently successful for the system to discover several useful patterns in the databases provided by our industrial collaborators, GMAP Ltd. and Barclay Bank Plc. Many of the insights, ideas and results of GA-MINER are also being exploited by Quadstone Ltd in its Decisionhouse product. The commercial market for data warehousing and data mining tools is expanding rapidly and increasing competition is forcing all businesses to put their customers at the centre of their operation. They are therefore collecting and trying to exploit data that they can obtain on these customers. Although hardware and database software now supports very large stores of data, there is no software currently available to analyse large data volumes efficiently, or to perform effective data mining operations in acceptable timescales. GA-MINER has provided many of the underlying architectural and analytical insights that have enabled the imminent product release of Decisionhouse, which aims to fulfill this need for scalable data mining software. EPCC-AIKMS-GA-MINER-REPORT 1.0 3