SumatraTT: Towards a universal data preprocessor

In the practice of data mining (DM) and data warehousing (DWH), real-life data arrive in various different formats, and without putting them into an acceptable shape, even the most intelligent DM/DWH tool would be useless. SumatraTT (Transformation Tool) is an original universal data pre-processing tool allowing to access and transform data stored in various types of datasources (e.g. plain text, SQL etc.). We briefly review the concept of the system and summarize its recent developments. The paper briefly overviews the connectivity with inductive logic programming (ILP) systems and then informs on more recently added features consisting of new data interfaces, scripting features, and templates. The usage of Sumatra TT on an example application is shortly demonstratied. After a brief touch upon near-future plans, we finally discuss some questions typically arising at the first usage of SumatraTT. 1 OUR MOTIVATION AND GOALS DM algorithms [6] are being designed by researchers and SW houses all over the world. There are many of them and their offer is continuously growing. Their different available implementations differ in principles as well as in tiny details such as the format used for the input data. Moreover, most often the data subjected to DM have not been collected for DM purposes primarily; on the contrary they serve e.g. as a company archive. Consequently, the format of such data cannot meet the requirements of a specific DM algorithm most often. The challenge of a DM task is in finding the algorithm which will reveal interesting observations in the considered data. But to reach this goal many experiments have to be done. One cannot decide in advance which set of DM tools or which derived attributes will prove most useful for the given problem. Thus original data have to be processed or transformed in different ways to make them usable by the chosen DM algorithms. Format of data has to be changed, data has to be cleaned, filtered, aggregated, etc. This is the purpose of data transformation systems which have recently appeared as independent SW tools supporting DM process itself [14]. This is an important step simplifying data preparation processes and supporting experiments with real life data. The common pain of state-of-art data transformation systems is their insufficient generality. Our goal is to overcome this problem by designing a developing a system • that allows for virtually any customization with respect to different data standards and requirements on the transformation, • but at the same time provides ultimate ease-of-use in cases where only standard procedures are required. The former goal can be achieved by providing a dataprocessing oriented scripting language, and the latter goal by providing templates of the common procedures and standard interfaces to many kinds of datasources. These ideas form the design principles of the data preprocessing tool SumatraTT described briefly in the next section. Most industries benefit from appropriate standardization. Positive reaction of the research community to the activity of the PMML group proves this is the case of the research field concerned with decision support systems, too. Prediction Model Mark-up Language is being developed to simplify exchange or sharing the results „between compliant vendorś applications . so that proprietary issues and incompababilities are no longer a barrier“ ( http://www.dmg.org/pmmlspecs_v2/ ). Similar view can be taken towards data-transformations. We hope that our study of data transfomations using SumatraTT will complemet to development of a standard for data transfomation, namely to the Data Transformation Markup Language (DTML) supporting e.g. reuse of the same data by different algorithms through seemless import, rapid development of derived attributes, etc. 2 THE CONCEPT OF SUMATRA SumatraTT (Transformation Tool) is a metadata-driven, platform independent, extensible, and universal data processing tool [3]. The mentioned features have been achieved by building the tool as an interpreter of the transformation-oriented scripting language called Sumatra [2]. The Sumatra language is a fully interpreted Java-like language combining data access, metadata access, and common programming constructions. Furthermore, it supports the RAD (Rapid Application Development) technology by means of the library of reusable transformation templates. The principal scheme of SumatraTT is shown in Figure 1. As can be seen in the figure, the central part of SumatraTT is the Metadata repository module. Basically, the repository plays two roles. It is the central storage consisting of descriptions of all data sources and data transformations to be used. Moreover, the repository contains data objects interconnecting the abstract data access level in the Sumatra interpreter with real-life data sources. This intermediated connection helps to unify data access to very different data sources (e.g. SQL-based data sources, plain text files, etc). Such unification makes the process of transformation script development easier and data source independent. Moreover it separates the transformation "logic" from the data connection problems. In the case of very complicated data pre-processing task, the development of a data transformation script can be rather time consuming. SumatraTT allows to speed up this process by using reapplicable transformation templates. The idea of reusable templates is based on the library of solved types of tasks. E.g. there is a data set containing time series and we need to calculate a statistical characterization of the data. If this is carried out for the first time, a new template has to be developed. But the next time, the statistical transformation script can be developed via a parametric modification of the existing template within a fraction of the time required before. Every pre-processing task realized using SumatraTT consists of design and run-time phases. It corresponds to a client-server architecture where the design phase consists of the definition of all data sources and the development of transformation scripts on the client side. Regarding a typical user who is an expert in data mining or data warehousing but who is not a programmer, the design phase can be carried out using graphical user interface. The GUI allows to interactively realize both the data definition and script development by simple clicking on wizards. On the other hand, the run-time phase corresponds to a script execution on the server side. From the user's perspective, the execution can be invoked immediately or scheduled for a later run.