Using Local Intelligence to Actively Monitor Distributed Data Sources in Decision Making Architectures

In data warehouse environments operational processes are used to move data from a variety of sources to a central repository usually called the warehouse. These processes include data export, data preparation, and data loading which are usually performed by using Extraction, Transformation and Loading (ETL) tools. Most Business Intelligence (BI) applications have to consolidate data first before performing any sort of data analysis. Therefore, they use centralized decision support architectures in which commonly a data warehouse technology is employed as the way to consolidate, analyse and report data to support decision making based on all the information that has been collected from a variety of data sources. ETL functions have been described as a critical component of a BI solution as they are responsible for retrieving data from different sources (one or more), preparing it by normalizing and transforming it in some way to then be inserted into some other repository, usually called a data warehouse, for analysis and reporting [152][56]. The main characteristics of the ETL mechanisms used in BI include the use of incremental loading techniques. This movement of data, commonly called the data refreshment, is usually driven from the central repository, which is programmed to request “query” data from the sources in a timely manner. Consequently, data refreshments are time dependent. Research conducted in real time and active data warehousing technologies for BI have deal with near right time by mainly focusing on the response time from the “machine” point of view to capture information, rather than on improving the speed to analyse data and therefore, advise in the right time to decision makers that “something relevant” is going on. Even with faster refreshment, using streaming techniques, data analysis takes part of the last iii stage after the consolidation has been performed, consequently reporting and alerts are the last activities after some sort of ETL technique has been applied. Frameworks that support real time data analysis and reporting by detecting adverse events have not yet been proposed. Some of other research gaps identified in this area are related to how to distribute the Extraction, Transformation and Loading processes to best meet the needs of real time decision making. Traditionally the data warehouse is responsible for querying the sources in which a vast amount of data is pulled (extracted) from the sources of data to be analysed and included in reports. If included, data errors and unusual behaviours in the data extracted are discovered only after data is loaded in the data warehouse. Usually, data errors and inconsistencies depend on the mechanism programmed by the data base administrator who is mainly responsible for the performance of the architecture deployed. Thus, it might be possible to re-distribute/re-order the way traditional architectures perform data extraction/consolidation/analysis from a variety of data sources to advise decision makers in real time that unusual events have been detected. Moreover, traditional data warehouse approaches are mainly built on static architectures that are not capable of supporting changes in their content and structure. After the literature review was conducted it was also found that the idea of dynamic adaptation to business requirements, logical adaptation rather than structural adaptation, of a DW architecture has not yet been proposed. Consequently, the demand for new BI solutions and frameworks is continuously growing but information systems research in the field has been shown to be limited [111]. To address some of the above mentioned research gaps such as the need to react in real time to unusual behaviours, this research study focuses on the area of data management and creates a framework to monitor data from a variety of data sources in real time using data warehouses. By enabling the use of local intelligence at the sources level data is monitored and pre-analysed first, if unusual behaviour in the data captured are found actions are then taken. Actions might include alerting the decision makers as unusual behaviours have been detected, doing nothing or pushing data to the warehouse as more analysis is needed. Thus, as soon as valuable data arrives at a source is analysed and only if needed data is pushed to iv a central repository, a data warehouse, because more information is needed. Given this research proposes a new way to design a monitoring infrastructure that empowers the use of local knowledge and distributes the data analysis in architectures that uses a data warehouse, the development of a prototype was needed to demonstrate the validity of the design. The framework proposed has been empirically evaluated and validated by designing and testing an architecture to be used in the health care arena to monitor unusual symptoms in coronary heart disease patients. The validation included a review of the prototype findings with specialists (decision makers), and an evaluation in terms of computational efficiency in which a part of the architecture proposed was compared and contrasted with the deployment of a traditional loading mechanism to feed a data warehouse. The findings of this research show that by monitoring key behaviours in the data at the source level, evaluating them and then responding only to data abnormalities, the proposed architecture is able to respond and/or alert to unusual scenarios in a more effective and efficient way than traditional right time data warehousing strategies. Traditional ETL mechanisms were not used, the extraction, consolidation and analysis processes have been re-distributed instead. The process of data loading is driven by the sources who are responsible for capturing and pre-analysing data changes and sending the relevant findings to the data warehouse. Thus, data is captured and pre-analysed first and the number of steps to perform data analysis for reporting and error detection have been reduced and moved to an early stage. A form of intelligence and adaptation has been achieved by using software agents who autonomously perform the core activities of “Transformation, Transference and Loading ” of the framework discussed and presented in this dissertation.