# Development of Cost and Schedule Data Integration Algorithm Based on Big Data Technology
## 4 Big Data Algorithm for Integrated Cost-Schedule Data Management
### Big Data Technology-Driven Algorithm for Cost-Schedule Integration  
This research proposes a cost-schedule data integration algorithm employing big data technology, as shown in Figure 4. It consists of seven modules and provides four main functions:  
* building big data from various data sources based on the 5W1H data schema (Module 1-4),
* extracting data for users' requirements (Module 5),
* analyzing data using extracted data (Module 6),
* visualizing analysis results by creating graphs, charts, and diagrams (Module 7).  
Figure 3: Big Data Building Steps for Cost and Schedule Data Integration.  
The detailed definitions and functions of the seven modules are represented in Figure 5. Module 1 is an _Automatic Text Mining_ module, collects information units for cost and schedule data, which are at the lower level of the 5W1H data structure, from data sources (including quantity take-off, cost reports, and schedule reports). The big data scraping technology that extracts structured or unstructured data in natural language is applied to this module. Module 2, named _Data Parsing module_, incorporates crawling technology to transform and process the data collected by Module 1 into structured data stored in the database. This module first conducts mapping the collected data into the corresponding items in MasterFormat, UniFormat, and construction project management standards, and imports the items' codes. It also finds shared data among different sources and uses it as a medium to link all collected data; it integrates and stores all data into the 5W1H data structure in the MSSQL database.  
Figure 4: Conceptual Workflow of the Proposed Algorithm for Cost-Schedule Integration.  
Figure 5: Big Data System Architecture for Cost-Schedule Integration Consisting of Seven Modules.  
Module 3, _Metadata Mapping_ module, automatically assigns values for higher-levels, using lowest-level data. It is in line with the second principle in Section 4.2. The codes of lowest-level data identified in Module 2 are used as metadata for inferring higher-level data. Each inferring rule for each dimension that is related to the classification systems and standards has been developed in Python. Module 4 is called _Big Data Building_ module and establishes big data of all level information units in six dimensions, according to the 5W1H schema.  
Module 5, _Big Data Mining_ module, is designed to extract the required data from the big data. It adopts functions of online analytical processing (OLAP) (slice, dice, rotation, rollup, drill-down, query optimizer) to allow data extracting from multi-level of details and multi-perspectives. As Module 6, a _Big Data Analysis_ module provides various analyses regarding project status, project value, and project forecast, using the extracted data by Module 5. Lastly, Module 7 for _Data Reporting & Visualization_ visualizes analysis results with the help of R or Microsoft Power BI.  
The implementation of Module 1 uses Python; the scraped data from sources is translated into Open XML format and checked its validity. Module 2 has been developed by Python to (1) categorize the collected lowest-level data from different sources, (2) incorporate them using shared data, and (3) store it into the data structure in MSSQL database. Module 3 has also been constructed using Python to infer upper-level data according to the metadata--the codes of lower-level data. Three Python libraries are used in the implementation: openpyxl, pymssql, and time. Module 1 utilizes the _openpyxl_ to extract data from source files (Excel file), and Module 2 uses the _pymssql_ to transform and store the extracted data in the MSSQL database. The _time_ is adopted to Module 3 to update and store the inferred higher-level data in the MSSQL database. Interfaces of the modules are shown in Figure 6.