# DAM2 -Data, Model and Monitoring A Scalable and Compliant Solution for Managing enriched Infrared images as FAIR Research Data

Jean Dumoulin, Thibaud Toullier, Nathanael Gey, Mathias Malandain

► To cite this version:

Jean Dumoulin, Thibaud Toullier, Nathanael Gey, Mathias Malandain. DAM2 -Data, Model and Monitoring A Scalable and Compliant Solution for Managing enriched Infrared images as FAIR Research Data. EGU 2025 - European Geosciences Union, Apr 2025, Vienna, Austria. 2025, 10.5281/zenodo.15182568. hal-05356991

HAL Id: hal-05356991

https://inria.hal.science/hal-05356991v1

Submitted on 10 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Jean Dumoulin<fnref n="1" />, Thibaud Toullier<fnref n="1" />, Nathanael Gey<fnref n="1" />, and Mathias Malandain<fnref n="2" />

## Summary

<figure><img src="image_2.png" /><figcaption>Overview of DAM2 in the overall context of a scientific instrumentation project</figcaption></figure>

A platform for managing native and enriched scientific datasets, generated by large scale ground based monitoring projects and opportunistic ones (implying multidimensional data types)

* including permissions and rights, visualization, etc.

* private and public dataset management through self-hosting possibility

* compliance to regulations (FAIR principles, GDPR)

* keep track of all data generated during the project

* iterative chunked upload (pause / resume upload feature)

## 1. Specifications

### File Formats

#### Requirements

* Open source

* Complete and documented API

* Wide variety of data types

* Performances

* Handle a large volume of data

* Support compression

* Store metadata

#### Key consideration

* Long-term sustainability

* What about data loss?

* Mechanisms for data integrity

* Compatible programming languages

* FAIR data compliant

* Cloud storage compatibility

#### State of the art comparison

Benchmarks are often specific and application related, often lacking generic use-cases

* In [1] HDF5 and Zarr offer better performances than NetCDF, Zarr performs better than Apache Parquet for satellite data [2]

* HumanCellAtlas project [3] decided to go for Zarr data, while some datasets are actually in HDF5

### Storage

#### Object Storage

* High scalability, ideal for cloud environments

* Unstructured and heterogeneous data

#### Public datasets

* hosted on Zenodo

* doi and metadata reference on the platform

* Web API (REST)

* Widely used

#### Private datasets

* self hosting

* right sharings customization

<figure><img src="image_3.png" /><figcaption>Zarr data format architecture</figcaption></figure>

Overall, Zarr seems to be well suited for cloud-based applications.

<figure><img src="image_4.png" /><figcaption>MinIO administrator interface</figcaption></figure>

Open-Source, AGPL-v3 licence and S3-compatible solution : MinIO [4]

<figure><img src="image_5.png" /></figure>

## 2. Architecture and deployment

### Overall architecture

* dj Backend: based on Django to provide REST API + handle the whole platform

* PostgreSQL: Relational database to store all management information

* Celery: use of distributed tasks

* Redis: broker to support Django / Celery communication

* MinIO: S3 compatible data storage

* Frontend: VueJS + Vuetify

* Docker: Deployment using docker-compose

<figure><img src="image_6.png" /><figcaption>DAM2 main flowchart</figcaption></figure>

## 3. Large files management

### Chunked upload feature

* Pause / Resume uploads

* Upload process status

* HDF5 to Zarr converter included

* manage history files

### Visualization

Raw files visualization:

<figure><img src="image_7.png" /></figure>

Use of the h5web widget [5]

* fork of the H5Grove to read both HDF5 and Zarr files

<figure><img src="image_8.png" /><figcaption>DAM2 raw data visualization using h5web</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>DAM2 upload history widget</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>DAM2 upload sequence diagram</figcaption></figure>

### Conclusions

Large data platform management that enables:

* project data tracking and management,

* files sharing,

* raw data visualization,

* FAIR and GDPR compliant

### Perspectives

Towards a full platform for handling monitoring (e.g. instrumentation systems) projects

* project management

* sensors & model configurations

* direct S3 data access

* integrated data & computed configuration

* dashboards creation & advanced visualization

* advanced data processing pipelines

Addressing a skeleton for automated digital twins building and deployment

### Bibliography

[1] S. Ambatipudi and S. Byna, "A Comparison of HDF5, Zarr, and netCDF4 in Performing Common I/O Operations." 2023.

[2] [Online]. Available: https://element84.com/software-engineering/benchmarking-zarr-and-parquet-data-retrieval-using-the-national-water-model-nwm-in-a-cloud-native-environment/

[3] A. Regev et al., "The Human Cell Atlas White Paper." [Online]. Available: https://arxiv.org/abs/1810.05192

[4] I. MinIO, "S3 compatible storage for AI." [Online]. Available: https://min.io/

[5] A. Bocciarelli, L. Huder, T. VINCENT, P. Chang, B. B. Maranville, and K. Bhasin, "silx-kit/h5web: v14.0.1." [Online]. Available: https://doi.org/10.5281/zenodo.15182568

<footnote n="1">¹Université Gustave Eiffel, Inria, COSYS Laboratory, I4S Team, F-44344, Bouguenais, France</footnote>

<footnote n="2">²Inria center at Rennes University, SED, F-35042 Rennes, France</footnote>