Building a Data Lake with Open Source Components
Channel: British Computer Society Open Source Specialists
Date: 2021-12-19

Presented by Hendrik Nolte, GWDG

Across various domains, data lakes are successfully utilized to centrally store all data of an organization in their raw format. This promises a high reusability of the stored data since a schema is implied on read, which prevents an information loss due to ETL (Extract, Transform, Load) processes. Despite this schema-on–read approach, some modeling is mandatory to ensure proper data integration, comprehensibility, and quality. These data models are maintained within a central data catalog which can be queried. To further organize the data in the data lake, different architectures have been proposed, like the most widely known zone architecture. Here, data is assigned to different zones according to the processing they were subjected to. In this work, we present a novel data lake architecture based on FAIR (Findable, Accessible, Interoperable, Reusable) Digital Objects (FDO) with (high-performance) processing capabilities. The FAIR Digital Objects are connected by a provenance-centered graph. Users can define generic workflows, which are reproducible by design, making this data lake implementation ideally suited for science. During this talk, challenges and opportunities for open source software as elementary building blocks of a data lake will be discussed.

Hendrik Nolte ObtainedB.Sc & M.Sc. in Physics at the University of Göttingen. Since November 2019: At GWDG in the HPC-Team. Working on a third-party-funded project where a data lake is being built for MR images.