# goldMEDAL: A Data Lake Generic Metadata Model

Etienne Scholly, Pegdwendé Nicolas Sawadogo, Pengfei Liu, Javier A. Espinosa-Oviedo, Cécile Favre, Sabine Loudcher, Jérôme Darmont, Camille Noûs

► To cite this version:

Etienne Scholly, Pegdwendé Nicolas Sawadogo, Pengfei Liu, Javier A. Espinosa-Oviedo, Cécile Favre, et al.. goldMEDAL: A Data Lake Generic Metadata Model. 37e Conférence sur la Gestion de Données - Principes, Technologies et Applications (BDA 2021), Paris, 2021. hal-03911169

HAL Id: hal-03911169

https://hal.science/hal-03911169

Submitted on 22 Dec 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
goldMEDAL: A Data Lake Generic Metadata Model

Étienne Scholly

Université de Lyon, Lyon 2,
UR ERIC & BIAL-X
Lyon, France

etienne.scholly@bial-x.com

Pegdwendé N. Sawadogo

Université de Lyon, Lyon 2,
UR ERIC

Lyon, France
pegdwende.sawadogo@univ-lyon2.fr

Pengfei Liu

Université de Lyon, Lyon 2,
UR ERIC

Lyon, France
pengfei.liu@eric.univ-lyon2.fr

Javier A. Espinosa-Oviedo

Université de Lyon, Lyon 2,
UR ERIC-LAFMIA lab
Lyon, France

javier.espinosa@imag.fr

Cécile Favre

Université de Lyon, Lyon 2,
UR ERIC

Lyon, France
cecile.favre@univ-lyon2.fr

Sabine Loudcher

Université de Lyon, Lyon 2,
UR ERIC

Lyon, France
sabine.loudcher@univ-lyon2.fr

Jérôme Darmont

Université de Lyon, Lyon 2,
UR ERIC

Lyon, France
jerome.darmont@univ-lyon2.fr

Camille Noûs

Université de Lyon, Lyon 2,
Laboratoire Cogitamus
Lyon, France

camille.nous@cogitamus.fr

## ABSTRACT

We summarize here a paper published at the DOLAP 2021 international workshop, which was collocated with EDBT and ICDT. We introduce goldMEDAL, a generic metadata model for data lakes [5].

## 1 INTRODUCTION

The rise of big data has revolutionized data exploitation practices and led to the emergence of new concepts. Among them, data lakes are large heterogeneous data repositories that can be analyzed by various methods [1].

An efficient data lake requires a metadata system that addresses the many problems arising when dealing with big data. The study of data lake metadata models is currently an active research topic and many proposals have been made [2–4].

However, existing metadata models (including the most recent cited above) are either tailored for a specific use case or insufficiently generic to manage different types of data lakes. To address this issue, we propose goldMEDAL, a generalization of MEDAL, the *METadata model for DAta Lakes* [4]. This new metadata model is specified through a classical three-level modeling process, i.e., conceptual, logical and physical.

## 2 GOLDMEDAL CONCEPTUAL MODEL

goldMEDAL’s conceptual model features four main concepts: data entity, grouping, link and process (Figure 1).

* **Data entities** are the basic units of our metadata model. They are flexible in terms of data granularity. For example, a data entity can represent a spreadsheet file, a textual or semi-structured document, an image, a database table, a tuple or

<figure><img src="image_2.png" /><figcaption>Figure 1: goldMEDAL concepts (UML class diagram)</figcaption></figure>

an entire database. The introduction of any new element in the data lake leads to the creation of a new data entity.

* **A grouping** is a set of groups, with a **group** bringing together data entities based on common properties. For example, the raw and preprocessed data zones common in data lake architectures are the groups of a zone grouping. Another example is a grouping of textual documents according to the language of writing.

* **Links** are used to associate either data entities with each other or groups of data entities with each other. They can be oriented or not. They allow the expression of, e.g., simple similarity links between data entities or hierarchies between groups. For example, a temporal hierarchy month → quarter would have the months of January, February and March linked to the first quarter of a given year.

* **A process** refers to any transformation applied to a set of data entities that produces a new set of data entities.

<footnote>© 2021, Copyright is with the authors. Published in the Proceedings of the BDA 2021 Conference (October 25-28, 2021, En ligne, France). Distribution of this paper is permitted under the terms of the Creative Commons license CC-by-nc-nd 4.0.</footnote>

<footnote>© 2021, Droits restant aux auteurs. Publié dans les actes de la conférence BDA 2021 (25-28 octobre 2021, En ligne, France). Redistribution de cet article autorisée selon les termes de la licence Creative Commons CC-by-nc-nd 4.0.</footnote>
# 3 GOLDMEDAL LOGICAL MODEL

At the logical level, goldMEDAL concepts are represented by a graph. Data entities translate into nodes, links translate into edges and groups and processes translate into hyperedges.

For example, in Figure 2, four data entities (say, textual documents) are represented by nodes $n_1, n_2, n_3$ and $n_4$. The set of hyperedges $H_1 = \{\theta_{11}, \theta_{12}\}$ represents a zone grouping, where $\theta_{11}$ and $\theta_{12}$ are hyperedges representing the groups *Raw data zone* and *Processed data zone*, respectively. Similarly, $H_2 = \{\theta_{21}, \theta_{22}\}$ represents a language grouping, where $\theta_{21}$ and $\theta_{22}$ represent the groups *French* and *English*, respectively.

<figure><img src="image_3.png" /><figcaption>Figure 2: Sample grouping graph logical model</figcaption></figure>

Figure 3 represents the logical model of a process that merges two data entities (say, two relational tables) represented by nodes $n_7$ and $n_8$ into a new node $n_9$. Process $\Pi_1 = \{\gamma_1, \Omega_1\}$ is an oriented hyperedge, with $\gamma_1 = \{n_7, n_8\}$ and $\Omega_1 = \{n_9\}$ being the sets of input and output nodes of $\Pi_1$, respectively.

# 4 GOLDMEDAL PHYSICAL MODELS

At the physical level, we implemented goldMEDAL into three use-cases/data lakes dedicated to social housing, management sciences and archaeology, respectively. Metadata are managed with dedicated tools (the Neo4J<fnref n="1" /> graph database management system or the Apache Atlas<fnref n="2" /> data governance and metadata management) or a combination of such tools, i.e., Neo4J, the MongoDB<fnref n="3" /> document-oriented database management system (for handling non-atomic metadata) and Elasticsearch<fnref n="4" /> search engine (for indexing textual documents).

<figure><img src="image_4.png" /><figcaption>Figure 3: Sample process graph logical model</figcaption></figure>

# 5 CONCLUSION

Through the three physical models implemented with goldMEDAL, we demonstrated the feasibility and flexibility of our metadata model. Moreover, we demonstrate that goldMEDAL’s concepts generalize those of the most recent models from the literature [2–4], which makes of goldMEDAL the most generic metadata model for data lakes as of today.

Another particularity of goldMEDAL is the explicit possibility of data lineage tracing with the concept of process. Thus, goldMEDAL can manage the dynamics of data, while the most recent metadata model, HANDLE [2], does not natively support it.

Future research and open issues include the “industrialization” of data lakes, i.e., providing a software layer, connected to the metadata system, which allows non-data or non-computer scientists to transform and analyze their own data in autonomy.

Furthermore, exploiting a data lake and its metadata system may contribute to open data and open science. A well-designed data lake should indeed readily enforce the four FAIR principles<fnref n="5" />.

# ACKNOWLEDGEMENTS

Étienne Scholly (PhD), Pegdwendé Nicolas Sawadogo (PhD) and Pengfei Liu’s (postdoc) are funded by the BIAL-X company, the AURA Region and the IMU LabEx, respectively.

# REFERENCES

[1] Dixon, J. (2010). Pentaho, Hadoop, and Data Lakes. https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/.

[2] Eichler, R., C. Giebler, C. Gröger, H. Schwarz, and B. Mitschang (2020). HANDLE – A Generic Metadata Model for Data Lakes. In *DaWaK 2020, Volume 12393 of LNCS*, pp. 73–88.

[3] Ravat, F. and Y. Zhao (2019). Metadata management for data lakes. In *ADBIS 2019, Volume 1064 of CCIS*, pp. 37–44.

[4] Sawadogo, P. N., E. Scholly, C. Favre, E. Ferey, S. Loudcher, and J. Darmont (2019). Metadata systems for data lakes: models and features. In *BBIGAP@ADBIS 2019, Volume 1064 of CCIS*, pp. 440–451.

[5] Scholly, E., P. N. Sawadogo, P. Liu, J.-A. Espinosa-Oviedo, C. Favre, S. Loudcher, J. Darmont, and C. Noûs (2021). Coining goldMEDAL: A New Contribution to Data Lake Generic Metadata Modeling. In *DOLAP@EDBT/ICDT 2021, Volume 2840 of CEUR*, pp. 31–40.

<footnote n="1">¹https://neo4j.com/</footnote>

<footnote n="2">²https://atlas.apache.org/</footnote>

<footnote n="3">³https://www.mongodb.com/</footnote>

<footnote n="4">⁴https://www.elastic.co/</footnote>

<footnote n="5">⁵https://www.go-fair.org/fair-principles/</footnote>