# A description of the radio astronomy data processing tool DDF Pipeline

Mathis Certenais, Laurent Morin, François Bodin

► To cite this version:

Mathis Certenais, Laurent Morin, François Bodin. A description of the radio astronomy data processing tool DDF Pipeline. 2026. hal-05519244

HAL Id: hal-05519244

https://hal.science/hal-05519244v1

Preprint submitted on 19 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# A description of the radio astronomy data processing tool DDF Pipeline

Mathis Certenais

mathis.certenais@irisa.fr

Université de Rennes

Rennes, France

François Bodin

francois.bodin@irisa.fr

Université de Rennes

Rennes, France

Laurent Morin

laurent.morin@irisa.fr

Université de Rennes

Rennes, France

## ABSTRACT

This paper presents the DDF Pipeline, a radio astronomy data processing tool initially designed for the LOw-Frequency ARray (LOFAR) radio-telescope and a candidate for processing data from the Square Kilometre Array (SKA). This work describes the DDF Pipeline software and presents a coarse-grain profiling execution to characterize its performance.

## KEYWORDS

Profiling

## 1 INTRODUCTION

The next-generation Square Kilometre Array (SKA) [2] radio telescope is estimated to produce approximately 700 petabytes of archived data per year. To meet the challenges posed by scientific projects that generate such large volumes of data, advanced data processing tools are essential. The DDF Pipeline [7, 11] is a data processing tool initially designed for the LOw-Frequency ARray (LOFAR) [7] radio telescope and is a candidate for processing SKA data.

This work is divided into two parts: the first part describes the DDF Pipeline software, and the second part presents a coarse-grain profiling execution of the DDF Pipeline and characterizes its performance.

## 2 SOFTWARE DESCRIPTION

In this section, we present the DDF Pipeline [7, 11], a radio astronomy data processing software. Developed by Martin Hardcastle, Tim Shimwell, Cyril Tasse and Wendy Williams [4], this software was used in the production of data releases of the LOFAR Two-metre Sky Survey (LoTSS) and the Two-metre Sky Survey Deep Fields project (LoTSS Deep). The recent survey [6] generated images from 290 terabytes of data collected from 505 hours of on-source observations spanning 7.5 years. The DDF Pipeline is a composite application that integrates the imaging software DDFacet [10] and the calibration software killMS [8, 9]. Inputs to the pipeline are datasets known as MeasurementSets (denoted MS in the remainder of the paper) [5], and the outputs are data products, specifically sky images, in the Flexible Image Transport System (FITS) format. The DDF Pipeline operates in two main stages, each involving a sequence of calibration, imaging, and data product-generation steps. In the first stage, the pipeline processes a subset of the MS, while the second stage considers all the MS to produce the final data products. During execution with the default configuration file, the DDF Pipeline makes a total of 120 calls to killMS, processing each data subset independently, with 96 of these calls occurring during the second phase. Additionally, there are 19 calls to DDFacet: 12 calls process a subset of 6 out of the 24 data subsets, while the

<table><caption>Table 1: Duration of the main tasks executed during the DDF Pipeline, including the time taken in seconds and the percentage of total execution time for each process.</caption><thead><tr><td>Process name</td><td>Time (s)</td><td>Time (%)</td></tr></thead><tbody><tr><td>killMS</td><td>132 388</td><td>53.40%</td></tr><tr><td>DDFacet</td><td>83 969</td><td>33.87%</td></tr><tr><td>no profile</td><td>29 968</td><td>12.09%</td></tr><tr><td>killMS: smoothsol 1</td><td>825</td><td>0.33%</td></tr><tr><td>DDFacet: clusterGA</td><td>517</td><td>0.21%</td></tr><tr><td>DDFacet: mkmask</td><td>157</td><td>0.06%</td></tr><tr><td>DDFacet: maskdico</td><td>101</td><td>0.04%</td></tr><tr><td>killMS: smoothsol 2</td><td>4</td><td>0.00%</td></tr><tr><td><strong>total</strong></td><td><strong>247 929</strong></td><td><strong>100.00%</strong></td></tr></tbody></table>

remaining 7 calls process the entire MS during the second stage. The default settings for the final product sizes are 20,000 × 20,000 pixels at a resolution of 1.5 arcseconds.

## 3 PROFILING

In this section, we present a coarse-grain profiling execution of the DDF Pipeline and characterize its performance. To assess the profiling execution of the DDF Pipeline, we utilized the Dool monitoring tool [1]. The run was carried out at Eskemm Numérique [3], with resource usage monitored using Slurm [12]. The hardware and configuration were as follows: an HPE ProLiant DL385 Gen10 Plus (v2 model) single node equipped with 120 CPUs, 2 x AMD EPYC 7543 32-Core Processors, 32 cores each at 2.8GHz with hyper-threading, and 512 gigabytes of RAM configured with 50% of shared memory. We ran a Singularity container that includes the DDF Pipeline version 3.1, DDFacet version 0.7.2, and killMS version 3.1. This container version has been tested and approved by the developers. Although these application versions are several years old, their performance can be improved with the latest versions. We executed the DDF Pipeline with the default configuration file using the L234026 dataset, which covers a frequency range between 121 MHz and 164 MHz. The MS is split into two bands: 121–129 MHz and 130–164 MHz, both with a 2 MHz resolution. The input consisted of 24 tar-archive MeasurementSets (MS), totaling 38.72 gigabytes, which decompressed to 134.4 gigabytes. The profiling run of the DDF Pipeline was completed in 68.87 hours. The output totaled 594 gigabytes. The data products are composed of 15 images, with a total size of 17.92 gigabytes. This includes 11 full-resolution images, each of 1.575 gigabytes, and 4 low-resolution images, each of 0.148 gigabytes.
<figure><img src="image_2.png" /><figcaption>Figure 1: CPU user and system level occupancy, respectively operating system, inputs and outputs (I/Os), aggregated in average per minute, time in hours on the x-axis, number of CPUs used on the y-axis. The blue curve represents DDFacet, the yellow curve represents killMS, and the gray curve represents periods with no profile.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2: Memory used and cache system occupancy in average, aggregated in average per minute, time in hours on the x-axis, number of bytes on the y-axis. The blue curve represents DDFacet, the yellow curve represents killMS, and the gray curve represents periods with no profile.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3: Disk inputs and outputs (I/Os) system occupancy, aggregated in average per minute, time in hours on the x-axis, number of I/Os in logarithmic scaling on the y-axis. The blue curve represents DDFacet, the yellow curve represents killMS, and the gray curve represents periods with no profile.</figcaption></figure>

# 4 CONCLUSIONS AND FUTURE WORKS

In this work, we described and characterized the performance of the DDF Pipeline through a coarse-grain profiling execution. Our execution of the DDF Pipeline, utilizing the Dool monitoring tool

<table><caption>Table 2: Resource occupancy information of the CPU usage (User and System) and memory usage in gigabytes during the DDF Pipeline run</caption><thead><tr><th rowspan="2">Process name</th><th colspan="2">CPU</th><th colspan="2">Memory</th></tr><tr><th>Usr</th><th>Sys</th><th>Used (GB)</th><th>Cached (GB)</th></tr></thead><tbody><tr><td>DDFacet</td><td>25.1</td><td>1.5</td><td>45.7</td><td>428.5</td></tr><tr><td>no profile</td><td>12.3</td><td>1.9</td><td>37.4</td><td>425.2</td></tr><tr><td>killMS: smoothsol 1</td><td>10.9</td><td>12.0</td><td>13.6</td><td>382.6</td></tr><tr><td>killMS</td><td>9.2</td><td>2.5</td><td>33.6</td><td>414.4</td></tr><tr><td>DDFacet: clusterGA</td><td>5.5</td><td>0.7</td><td>14.1</td><td>384.4</td></tr><tr><td>DDFacet: maskdico</td><td>0.7</td><td>0.1</td><td>13.0</td><td>376.5</td></tr><tr><td>killMS: smoothsol 2</td><td>0.7</td><td>0.1</td><td>11.7</td><td>354.4</td></tr><tr><td>DDFacet: mkmask</td><td>0.5</td><td>0.7</td><td>14.8</td><td>388.3</td></tr></tbody></table>

<table><caption>Table 3: Resource occupancy information of the disk read and write operations during the DDF Pipeline run</caption><thead><tr><th rowspan="2">Process name</th><th colspan="2">Disk</th></tr><tr><th>#Total Read</th><th>#Total Write</th></tr></thead><tbody><tr><td>DDFacet</td><td>5488.3</td><td>13034.0</td></tr><tr><td>no profile</td><td>68923.8</td><td>6335.8</td></tr><tr><td>killMS: smoothsol 1</td><td>63353.5</td><td>618.8</td></tr><tr><td>killMS</td><td>11641.1</td><td>18397.8</td></tr><tr><td>DDFacet: clusterGA</td><td>125542.0</td><td>10711.4</td></tr><tr><td>DDFacet: maskdico</td><td>294100.9</td><td>15248.5</td></tr><tr><td>killMS: smoothsol 2</td><td>1241088.0</td><td>0.0</td></tr><tr><td>DDFacet: mkmask</td><td>647272.4</td><td>55726.5</td></tr></tbody></table>

and running on an HPE ProLiant DL385 Gen10 Plus single node at Eskemm Numérique, was completed in 68.87 hours. The pipeline processed 24 tar-archive MeasurementSets (MS) totaling 134.4 gigabytes and produced 594 gigabytes of output. Future work will focus on porting the DDF Pipeline to the Jean Zay HPC center.

# REFERENCES

[1] Scott Baker. 2024. scottchiefbaker/dool. https://github.com/scottchiefbaker/dool original-date: 2019-06-13T23:13:45Z.

[2] Peter E. Dewdney, Peter J. Hall, Richard T. Schilizzi, and T. Joseph L. Wazio. 2009. The Square Kilometre Array. Proc. IEEE 97, 8 (Aug. 2009), 1482–1496. https://doi.org/10.1109/JPROC.2009.2021005

[3] The Brittany Digital Public Interest Group. 2025. Eskemm Numérique. https://www.eskemm-numerique.fr/

[4] Martin Hardcastle, Tim Shimwell, Cyril Tasse, and Wendy Williams. 2025. DDF Pipeline source code. https://github.com/mhardcastle/ddf-pipeline

[5] A.P. Schoenmakers, G.A. Renting, and J.D.D. Mol. 2022. MeasurementSet description for LOFAR, Version 2.09.00. https://www.astron.nl/lofarwiki/lib/exe/fetch.php?media=public:documents:ms2_description_for_lofar_2.09.00.pdf

[6] T. W. Shimwell, C. L. Hale, P. N. Best, et al. 2025. The LOFAR Two-metre Sky Survey: Deep Fields Data Release 2. I. The ELAIS-N1 field. Astronomy & Astrophysics 695 (March 2025), A80. https://doi.org/10.1051/0004-6361/202452930 arXiv:2501.04093 [astro-ph].

[7] T. W. Shimwell, C. Tasse, M. J. Hardcastle, et al. 2019. The LOFAR Two-metre Sky Survey. II. First data release. Astronomy and Astrophysics 622 (Feb. 2019), A1. https://doi.org/10.1051/0004-6361/201833559 ADS Bibcode: 2019A&A...622A...1S.

[8] O. M. Smirnov and C. Tasse. 2015. Radio interferometric gain calibration as a complex optimization problem. Monthly Notices of the Royal Astronomical Society 449, 3 (May 2015), 2668–2684. https://doi.org/10.1093/mnras/stv418

[9] Cyril Tasse. 2014. Applying Wirtinger derivatives to the radio interferometry calibration problem. https://doi.org/10.48550/arXiv.1410.8706 arXiv:1410.8706 [astro-ph].
[10] C. Tasse, B. Hugo, M. Mirmont, O. Smirnov, M. Atemkeng, L. Bester, M. J. Hardcastle, R. Lakhoo, S. Perkins, and T. Shimwell. 2018. Faceting for direction-dependent spectral deconvolution. *Astronomy & Astrophysics* 611 (March 2018), A87. https://doi.org/10.1051/0004-6361/201731474 arXiv:1712.02078 [astro-ph].

[11] C. Tasse, T. Shimwell, M. J. Hardcastle, et al. 2021. The LOFAR Two-meter Sky Survey: Deep Fields Data Release 1. I. Direction-dependent calibration and imaging. *Astronomy and Astrophysics* 648 (April 2021), A1. https://doi.org/10.1051/0004-6361/202038804 ADS Bibcode: 2021A&A...648A...1T.

[12] Andy B. Yoo, Morris A. Jette, and Mark Grondona. 2003. SLURM: Simple Linux Utility for Resource Management. In *Job Scheduling Strategies for Parallel Processing*, Dror Feitelson, Larry Rudolph, and Uwe Schwiegelshohn (Eds.). Springer, Berlin, Heidelberg, 44–60. https://doi.org/10.1007/10968987_3