D4.1 BIOVALUE Data Warehouse module
Creators: name: Theofilou, Asterios
name: Tsagris, Michail
name: Nastis, Stefanos
name: Mattas, Konstadinos
name: Tzouvelekas, Vangelis
name: Garcia Guirao, Daniel
name: Rodriguez, Santiago
name: Yercan, Murat
Publisher: name: Zenodo
Year: 2022

description: As stated in the Grant Agreement: “The approach of the BIOVALUE project is to set-up a holistic perspective, under the “multi-actor” approach, to analyze the link between biodiversity, the agro-food value chain agents, the environment, consumer food preferences and health. By employing a bottom-up vertical approach to develop the BIOVALUE TOOL, a dynamic and customizable agri-food value chain vis-a-vis biodiversity analysis tool, the proposal tries to introduce, model, evaluate, breed, produce and spread specifically selected genetically diverse underutilized crops (cereals, legumes, leafy and fruity vegetables) and develop final marketable, certified and labelled culinary products incorporating them (dish recipes and processed foodstuff), that enhance agro-biodiversity to the applied agro-ecosystems and appeal to the consumers, securing their future market performance and concurrently, their cultivation viability. Moreover, in a modelled user-friendly ready-to-work framework, the project will produce a set of key performance indicators destined to measure policy quality and impact, environmental evolution and compliance with regulations of introducing underutilized, genetically diverse crops to the value chain and are by design expandable to further enhance biodiversity in the value chain. Ultimately, this expanding nature, is highlighted by the complimentary effects of BIOVALUE processed and unprocessed final food products and dishes such as low energy consumption, environmental cultivation resiliency to marginal landscapes and future climate, as well as nutritional and health benefits. Incorporating the market power in the whole approach, the proposal can lead towards self-supported sustainability of biodiversity in the future.”  

This deliverable presents and explains the definition, the state-of-the-art tools and technologies that have been selected for the extraction, transformation and loading processes, the architectural design that will be followed, the requirements that have been defined, and the implementation phases of the BIOVALUE project’s DataWarehouse.  

Definition

 

What is a data warehouse A main component of the BIOVALUE tool is the Data Warehouse. A Data Warehouse, from a high-level point of view, is a collection of tools and methodologies, combined in such a way that supports datadriven decision making. It is a central repository that stores both historical and current data deriving from various sources.  In BIOVALUE, the DWH serves as the main repository for the data created and collected by the Project. It will also be used to ingest, transport, process and store data generated by the BIOVALUE tool, generating a cyclical process of data storage and supply. An example of this would be the simulations launched by the tool itself. The simulation processes need synthetic populations, among other components, to be executed. These synthetic populations need, in turn, to be fed with accurate data that would come from the DWH to store potentially useful datasets. Once the synthetic population is generated, it will be also stored in the DWH, as well as other elements that compose the simulation process and the results obtained from it.

ETL processes To transfer data to the DWH, the ETL process is used. ETL stands for Extract Transform and Load. ETL is the process of integrating data from multiple sources into a single data store, the DWH.  

Data warehouses, data lakes, and databases 

There are 3 commonly used combinations of tools to efficiently store, process, query and analyze data. Those are: Databases, Data lakes, and Data Warehouses, each with its own strengths and weaknesses. For the BIOVALUE project the optimal choice was to implement a Data Warehouse.

State-of-the-art-tools 

The end user of a Data Warehouse needs fast access to large sets of data, integrated from different sources. This means that the main task in data warehousing is to equip the analyst with a tool that provides fast access to aggregates of large sets of data, that are integrated from different sources, and to do so, a data warehouse system needs to decrease the query response time as much as possible. The tools described below are separated in 4 distinct categories, namely: data ingestion, data processing, data storage, and data visualization. The tools described are open-source software. Data ingestion The first activity of a Data Warehouse’s operations is data ingestion. In this step, data is extracted from various external sources to a single storage location for further processing. If this step was to be executed manually it would take a toll both on time and finances. Data ingestion software facilitates such transfers fast, automatically, and cost effectively. For this DWH task the selected tools are Apache Sqoop and Apache Flume. 

Data processing For the Data processing part of the DWH implementation, two software tools have been selected, Hadoop MapReduce and Apache Spark. 

Data storage For the data storage part of the DWH implementation, three software tools have been selected, Hadoop HDFS, YARN, and Apache HIVE. 

Data visualization For the data visualization part of the DWH implementation, two software tools have been selected, Grafana and Apache Superset. 

Architecture requirements Functional requirements

Taking into consideration the DWH’s intended use a list of requirements has been created that define the behavior and the functionalities that the system must perform. This is an exhaustive but editable list that will be extended throughout the project, as needed. 

Non-functional requirements An additional list of non-functional requirements has been created, to set the criteria that will be used to judge the systems operational performance in regard to processing, deployment, connectivity, storage, performance, veracity, variety, and accessibility. This is also an exhaustive but extensible list.

 

Architecture approach

Architecture scheme and DWH Architecture

After the analysis of the functional requirements and the state-of-the-art tools, the functional elements that comprise the BIOVALUE DWH architecture are: A distributed file system, to prevent data loss by replicating data, making the system fault tolerant. A resource management system, to manage efficiently the system resources. An extraction/ingestion part, to import data from various and disparate data sources. A batch processing component, to process the data from the distributed file system. A programming environment component, to enable users perform different operations related to the DWH. Those tools and processes will be created, automated, and maintained by a limited number of DWH administrator users, that will be able to access the tools needed with password verification for security and data protection compliance, additionally, data access control will be enforced to all users.

Development plan 

Being still early in the project for data gathering, a circular process will be followed throughout the project’s duration. Whenever deemed necessary, meetings with interested WP partners will be conducted, to set additional architectural requirements, information requirements, and processes to facilitate new data input (collected from other WPs) and processing. For the characterization of data sources, the project will use ARDIT (Agricultural Research Data Index Tool).  This tool allows users to search for different sources of publicly available data on the Web, categorized by the methodology implemented according to the ontology defined in the project. 

Architecture delivery strategy The development plan is divided into two phases. A first complete setup phase and a second continuous circular phase. Within the first phase, the BIOVALUE DWH has been deployed and all the software components have been setup within it. Within the second phase, additional data sources will be integrated, social network ontologies will be further elaborated (that are useful for the analysis of underutilized crops with the BIOVALUE tool), and the BIOVALUE tool will be integrated to the DWH.  

Governance implementation To align the project’s goals with the Data Warehouse development, a framework has been created and put into place, that will allow us to maintain compliance to those set goals, and to recover in case of errors and/or disasters. The governance framework is split in 5 categories: monitoring, architecture, development/deployment, maintenance, retrospective, and support. Within those categories, the responsible parties for each are defined.