# HPC Software as a Service: A Data Logistics Approach with the DDF Pipeline as a Practical Case Study Deployment

Mathis Certenais, François Bodin

► To cite this version:

Mathis Certenais, François Bodin. HPC Software as a Service: A Data Logistics Approach with the DDF Pipeline as a Practical Case Study Deployment. SaLF 2025 - 11th Annual conference on Science At Low Frequencies, Dec 2025, Orleans (France), France. pp.1, 2025. hal-05424635

HAL Id: hal-05424635

https://hal.science/hal-05424635v1

Submitted on 18 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
## SKA SRCNet case study: DDF Pipeline radio astronomy data processing tool

<figure><img src="image_2.png" /></figure>

**DDF Pipeline total execution CPU usage in %.** dool monitoring tool. Eskemm single node, 128 CPU, 2 x AMD EPYC 7543 32-Core, 512 GB RAM. **75h total execution**, 6h for preparation, 69h for processing: 37h for calibration killMS and 23h for imaging DDFacet. 140 GB of input data for 8h integration, 600 GB of output data. https://arxiv.org/abs/2509.03075

## Cross-Facility Workflows: A High Performance Computing (HPC) deployment scenario

<figure><img src="image_3.png" /></figure>

**Cross-Facility Workflows:** Scientific processes that span heterogeneous, distributed, and federated infrastructures (e.g., the Digital Continuum) to automate data processing and analysis. The scenario spans multiple HPC systems with dynamic load allocation.

## Data Logistics support: Ephemeral Buffers

<figure><img src="image_4.png" /></figure>

**Data Logistics:** Data management processes for Cross-Facility Workflows within a federation. **Ephemeral Buffers:** A streamlined mechanism for operating temporary cross-facility data transfers. https://ebuffer.aqmo.org

## DDF Pipeline as a Service: Application Microservices

<figure><img src="image_5.png" /></figure>

**Application Microservices:** A streamlined mechanism for cross-facility service execution. https://ebservice.aqmo.org

## Acknowledgments

Funded by the Exa-AToW/NumPEx Program and supported by the ECLAT joint laboratory (CNRS, Inria, Observatoire de Paris, Observatoire de la Côte d'Azur, Eviden). HPC resources were provided by GENCI-IDRIS (Grant 2024-AD010615976).