D7.4 – Extraction, Transformation & Loading Tools and Model Checking v2
Creators: name: Lilis, Georgios
name: Zigeng, Fang
name: Katsigarakis, Kyriakos
name: Rovas, Dimitrios
name: Gonzalez-Gerpe, Salvador
name: García-Castro, Raúl
Publisher: name: Zenodo
Year: 2022

description: The COGITO solution provides an integrated and extensible toolkit to support construction projects in the planning and implementation phases. Three broad application areas have already been identified as core application domains: health and safety, quality control, and workflow modelling and monitoring. Each domain presents its own challenges, distinct information requirements, and differing stakeholder needs and perspectives. The digital twin platform is a data integration middleware, providing a Master Data Management service, and supports data integration workflows, to expose application-specific data views. These views are in the form of the ontologies developed within WP3, and additional data, like the reality capture data or dynamic time-series data captured from sensors in the field. These data are stored, linked, and made available to applications through the DTP’s APIs.

The instantiation of these application-specific models is not trivial, as it relies on processing imperfect data that may come in various container formats. What is more egregious is that there are no safeguards that even when the information provided is syntactically correct, it will contain all information required or that the quality of this information will meet baseline quality expectations so that the information will be usable. Input data include structured information like, for example, BIM models or construction schedules in well-defined formats as exported from project management tools. Dynamic information includes time series data that might have quality issues like gaps or out-of-range values. 

The work in this task stems partly from the need to identify and, where possible, address data quality issues. Once data are deemed of acceptable quality, Extraction, Transformation, and Loading (ETL) tools transform data in a form usable by the upstream applications. Some of these are application-specific, and others are rather general. The actor-based architecture of the Digital Twin Platform (DTP) allows the deployment and management of multiple data quality checks and ETL components. The DTP supports complex data checking and integration workflows in an extensible and general manner.

A significant part of the information uploaded on the platform is in the form of BIM models. BIM models (e.g., geometry and schedules) are given as input files but can also be the output of applications (e.g., semantic enrichment with health and safety information). When BIM data exchanges take place (e.g., when a BIM model is uploaded to the platform or when a specific ETL tool is called), we can encode the particular information requirements as a specific Model View Definition (MVD). We have implemented a general-purpose completeness checker, the MVD checking tool, which applies data completeness checks on input IFC4x3 files. Given a specific exchange, the MVD checker can investigate whether particular information is available, as encoded in an exchange-specific mvdXML file. This allows performing Model Checking (MC) to ensure that a BIM model has the complete information required for a specific use. Once the model has been imported, further checks can occur; one example is clash detection, implemented as part of the Geometric Clash Checker (GCC) tool.

Data providers should correct errors detected by the MC tools before the DTP can use the BIM information. The B-rep generation tool (ETL) allows the visualisation of detected issues to DTP’s GUI so errors or missing data can be corrected. The process might require several iterations, but ETL tools can be applied when an error-free model has been received. One such tool is the IFC Optimizer, which involves a lossless compression algorithm to reduce BIM data sizes. Or the GCC tool enriches the BIM model with containment relationships among construction elements and their construction zones (when these are not present).

The Construction RDF ETL tool instantiates the RDF graph model from the optimized and enriched BIM data. Once the optimised and enriched BIM model has been loaded, along with supplementary information in non-BIM formats (e.g., construction schedules), and all input data quality checks and transformations have been performed, we can instantiate the COGITO ontologies. Other inputs might come in structured (XML/JSON) files and can populate additional information. In particular: the Project RDF ETL for project-related data, the Process RDF ETL for construction schedules, and the Resources RDF ETL for construction project resources. 

Once all ETL processes have been applied, the resulting RDF models should be fully instantiated and ready for querying. An additional model checking takes then place. The Shapes Constraint Language (SHACL) can capture different application requirements. Then the DTP checks the finalised RDF models to determine whether they meet application-specific requirements (as defined by additional SHACL files). If the SHACL tests pass, the queries performed by applications should be able to return results. This comprehensive bottom-up (input checking) and top-bottom (application checking) should help ensure that applications receive good-quality data.

Some dynamic data quality checks have been delegated to the IoT pre-processing module for dynamic data. Two tools for construction site data logging, the IoT data logger and serving these data to applications, the IoT Data Retriever, are part of the DTP and are described in detail.

This demonstrator deliverable describes the data quality checking and ETL components described above and developed to support the specific COGITO tasks and use cases. The deliverable is the output of Task 7.2. D7.4 is the second version expanding upon D7.3 to include additional work up to M24 of the project. 

In the following sections, we describe the input/output format, technology, requirements and dependencies of these tools in detail. The discussion focuses on describing the inputs and outputs of each tool developed, referencing the detailed data structures in other deliverables. Section 1 provides an introduction and details on updates from the previous version (D7.3) of this deliverable. Section 2 discusses the operational sequence of the tools presented. For static data, Section 3 discusses MC and ETL tools that work with BIM data, and Section 4 tools that take as input non-BIM data. For dynamic data, Section 5 discusses related ETL tools. We give examples of the application of these components on data from COGITO pre-validation sites.