# The neXt Dirac incarnation

Federico Stagni, Alexandre Boyer, Andrei Tsaregorodtsev, Christophe Haen, Christopher Burr, Daniela Bauer, Simon Fayer, Janusz Martyniak

► To cite this version:

Federico Stagni, Alexandre Boyer, Andrei Tsaregorodtsev, Christophe Haen, Christopher Burr, et al. The neXt Dirac incarnation. 27th International Conference on Computing in High Energy and Nuclear Physics, Oct 2024, Krakow, Poland. pp.01336, 10.1051/epjconf/202533701336. hal-05307199

HAL Id: hal-05307199

https://hal.science/hal-05307199v1

Submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The neXt Dirac incarnation

Federico Stagni<fnref n="1" />,*, Alexandre Boyer<fnref n="1" />,**, Andrei Tsaregorodtsev<fnref n="2" />,***, Christophe Haen<fnref n="1" />,****, Christopher Burr<fnref n="1" />,†, Daniela Bauer<fnref n="3" />,‡, Simon Fayer<fnref n="3" />,§, and Janusz Martyniak<fnref n="3" />,¶

**Abstract.** The DIRAC interware has long served as a vital resource for user communities seeking access to distributed computing resources. Originating within the LHCb collaboration around 2000, DIRAC has undergone significant evolution. Despite its efficacy in meeting experiment-specific requirements, DIRAC has accrued technical debt over its 15-year history. Installation management remains intricate, with significant entry barriers and a reliance on bespoke infrastructure. The software development process lacks alignment with contemporary standards, impeding the onboarding process for new developers. Notably, integral components such as the network protocol and authentication mechanisms are proprietary and pose challenges for seamless integration with external applications. In response to these challenges, the DIRAC consortium has embarked on the development of DIRACX. DIRACX heralds a new era in distributed computing solutions. This contribution describes technical decisions, roadmap and timelines for the development of DIRACX, while recognizing the criticality of maintaining a continuously operational DIRAC system for numerous user communities.

## 1 Introduction

The DIRAC interware [1] boasts a rich history as a software product, having continuously evolved and modernized over the years. However, its developers and maintainers recently concluded that the time had come for a complete rewrite. This paper recounts the motivations and approach behind taking a well-established project and rebuilding it from the ground up.

The original project, DIRAC, has now been succeeded by DIRACX. While this is not the first rewrite in DIRAC's history, this iteration is more comprehensive and transformative.

<footnote n="1">¹CERN, EP Department, European Organization for Nuclear Research, Switzerland</footnote>

<footnote n="2">²Aix Marseille University, CNRS/IN2P3, CPPM, Marseille, France</footnote>

<footnote n="3">³Imperial College, London, UK</footnote>

<footnote>*e-mail: federico.stagni@cern.ch</footnote>

<footnote>**e-mail: alexandre.franck.boyer@cern.ch</footnote>

<footnote>***e-mail: atsareg@in2p3.fr</footnote>

<footnote>****e-mail: christophe.denis.haen@cern.ch</footnote>

<footnote>†e-mail: christopher.burr@cern.ch</footnote>

<footnote>‡e-mail: daniela.bauer@imperial.ac.uk</footnote>

<footnote>§e-mail: simon.fayer05@imperial.ac.uk</footnote>

<footnote>¶e-mail: janusz.martyniak@imperial.ac.uk</footnote>
## 2 What is DIRAC?

The DIRAC interware provides a comprehensive Grid solution for one or multiple user communities seeking to utilize distributed, heterogeneous computing resources efficiently. Acting as a middleware layer, DIRAC bridges the gap between user communities and various computational resources, enabling optimized, transparent, and reliable access.

DIRAC is widely adopted by diverse scientific communities, with its oldest and most experienced user being the LHCb collaboration. Other notable users include Belle II, ILC, and CTAO. DIRAC's rich history, illustrated in figure 1, traces its origins to the LHCb collaboration and its transition to open source in 2009. A significant milestone was the full rewrite undertaken in 2006–2007, which resulted in the DIRAC framework that continues to serve its users today.

<figure><img src="image_2.png" /><figcaption>Figure 1: DIRAC timeline, from the 2000 until the year 2024</figcaption></figure>

DIRAC source code is open source (GPLv3), written in python, and hosted on github.

### 2.1 What can DIRAC do?

DIRAC is a versatile software solution designed to manage distributed computing resources, including Computing Elements (e.g., HTCondorCE, ARC-CE) and Storage Elements (e.g., EOS, dCache). Its core functionalities encompass job submission, data administration, large-scale production management, and dataset handling. Acting as a comprehensive platform for Workload Management, Data Management, and Production Management, DIRAC is both flexible and extensible, making it widely adopted in various scientific domains such as physics, astrophysics, and life sciences.

Real-world use cases include:

* **LHCb:** Manages metadata and file provenance using a custom database with an Oracle backend [2].

* **Belle II:** A High Energy Physics (HEP) experiment that integrates DIRAC with Rucio [3] for data management.

* **CTAO:** Implements workflows tailored to its unique data processing requirements, which differ significantly from those of HEP experiments [4].

* **GridPP:** A multi-VO installation supporting neutrino (e.g. T2K and HyperK), dark matter (LZ) and other small to medium size VOs on grid and cloud resources.
* **HERD**: An astronomy and particle astrophysics experiment utilizing DIRAC's distributed High-Throughput Computing (dHTC) capabilities for data management.

* **EGI**: Employs DIRAC as a Workload Management System and integrates EGI-CheckIn [5] for identity management. EGI supports several communities not necessarily related to particle physics, like WeNMR (focused on structural biology and life sciences).

DIRAC is architected for flexibility and extensibility, with its functionalities organized into modular "Systems." In this contribution, we focus on key systems, including Workload Management, Data Management, and the Transformation System, highlighting their roles and capabilities.

## 2.2 Workload Management System

The role of the DIRAC Workload Management System is to run jobs. An overview of the DIRAC Workload Management System is shown in figure 2. DIRAC supports the "pull model" based on Pilot jobs [6], and a "push" solution for HPCs that do not support pilots (because of limited internet access) [7]. The recommended way of running jobs is by using the pull model. Pilots can be submitted to different computing element types, and make use, if needed, of the batch system.

<figure><img src="image_3.png" /><figcaption>Figure 2: DIRAC Workload Management System</figcaption></figure>
## 2.3 Data Management System

The DIRAC's Data Management System takes care of single files: placing, replicating, removing files. For dataset management the reader should refer to the next section about the *Transformation System*. A quick categorization about DIRAC concepts when it comes to data management:

* LFNs are "Logical File Names", and they are the only file identifiers that users work with.

* LFNs are registered in **catalog(s)**.

* LFNs may have PFNs (physical file names), stored in SEs (Storage Elements), that can be accessed with several protocols.

DIRAC supports various catalogs, which serve to address questions about the location of LFNs and/or their metadata. A catalog functions as an interface that requires implementation, such as the DIRAC Files Catalog (DFC), Rucio, or others, including those specific to extensions. For interacting with storage endpoints, DIRAC supports a limited number of protocols. Figure 3 pictures these concepts. The Data Management System also interfaces with FTS [8].

<figure><img src="image_4.png" /><figcaption>Figure 3: DIRAC Data Management System. The catalog implementations, as well as the storage protocols supported exceed what is pictured.</figcaption></figure>

## 2.4 The Transformation System (or "Workflow System")

"Transformation System" is a historic name for describing the DIRAC's system used for managing "Jobs Productions" (also known as "Data Processing transformations") and Dataset
management (e.g. replications). A Data Processing transformation (e.g. Simulation, Merge, Reconstruction...) creates jobs in the Workload Management System (and re-submits them if needed, eventually destroys them), while a Data Manipulation transformation replicates, or removes, datasets from storage elements.

<figure><img src="image_5.png" /><figcaption>Figure 4: DIRAC Transformation System</figcaption></figure>

Data processing and dataset management are operations having lots in common: for both of them it's about doing a certain *action* on a certain *data*. The *data* is effectively a list of LFNs, the action can be "run a payload with this (list of) LFN(s) as input", or "replicate this (list of) LFN(s) to Storage Element X". For both cases, re-try in case of failures. As seen in figure 4, the Transformation System is a client for the Workload Management System and the Data Management System. The actual definition of what is the production workflow, or the datasets description is left outside from this system (e.g. in a "Productions System", also part of DIRAC but not explored within this contribution).

For visualization of all the systems described above, DIRAC also provides a companion WebApp. Dashboards can be created within the DIRAC Web App and/or in Grafana.

## 2.5 Technicalities

DIRAC is written in python 3, with the Pilot code being the only element that still needs to support Python 2.7.

The DIRAC framework was developed initially in 2006 and 2007. It includes components named Services, Agents, Executors:
* **Services** are exposed at `urls like dips://box.some where:9132/WorkloadManagement/` where `dips` stands for "DIRAC protocol". The "DIRAC protocol", which is called **DISET** [9], is a thin layer on top of python sockets to allow GSI authentication. The services expose an RPC (Remote Procedure Call) implementation.

* **Agents** are like cron jobs. They are independent processes to fulfill one or several system functions.

* **Executors** are a simple implementation of task schedulers.

As back-ends `mysql` and `opensearch` are supported (for different purposes). The Web App is implemented using `ExtJS` [10], and fully custom Python bindings. For its internal authentication and authorization `DIRAC` understands X509 certificates and proxies [11]. `VOMS` [12] (Virtual Organization Membership Service) is effectively a hard dependency.

# 3 Motivations for DiracX

When designing a software product, we need to first of all consider its requirements. There are few stakeholders that can define requirements:

* **Communities/Users** want to interact with an easy to use system which is fast, responsive, scalable and flexible.

* **Administrators** care about ease of installation and update, up-to-date documentation, clear configuration, and possibly the availability of ready-to-use dashboards.

* **Developers** wants to develop code that is easy to test, but also modern, fun, and accessible to new developers.

All the above considered, we need to ensure business continuity for existing `DIRAC` users.

## 3.1 Technology trends, DIRAC, and the reasons for DiracX

From the pure technology point of view `DIRAC` is what nowadays would be called a "Web Application". As such, there are a few technology trends that `DIRAC` developers have been following closely: the emergence of ` OAuth2` [13], `REST` [14] APIs, and orchestration tools like `Kubernetes` [15]. ` OAuth2` has become the standard for secure authorization, enabling users to grant third-party applications access to their resources without sharing credentials. `REST` APIs have revolutionized communication between applications, offering a standardized and lightweight approach to enable seamless integration and data sharing across diverse platforms. Meanwhile, orchestration tools like `Kubernetes` have transformed how developers deploy and manage applications, providing automated scaling, fault tolerance, and efficient resource utilization in cloud-native environments. Together, these technologies underpin the modern landscape of microservices, cloud computing, and secure web development, fostering innovation and adaptability in rapidly evolving digital ecosystems. Despite efforts in introducing the aforementioned technologies within the current `DIRAC` framework, `DIRAC` at the moment does not provide comprehensive ` OAuth2` authorization capabilities, does not expose `REST` APIs, and efforts to deploy `DIRAC` on `Kubernetes` have proven difficult. While the use of these technologies is a desirable goal, the question that needs to be answered is how much effort would be needed to integrate the technologies above within the current implementation.

Another point to consider is that `DIRAC` has to interact with `Grid bodies` like `WLCG` [16] and `EGI` [17], which can dictate the use of certain technologies. For our case, we need to follow the changes in the security systems, with `WLCG` supporting the use of `IAM` [18] and
EGI the use of Check-IN [19]. For this case, DIRAC has no choice: these technologies are not only a desirable goal, but need to be used. Effectively, these are requirements. At the same time, the question is the same: how much effort would be needed to integrate the technologies above within the current framework?

DIRAC is a complex software, with a high entrance bar. Its deployment is somewhat cumbersome. DIRAC design has served us well for years, but it is not respecting today's trends. It is not very developer-friendly, in fact it is rather un-appealing and confusing, especially for new developers. All considered, we decided that, in order to satisfy the requirements listed above, it would be better to re-write DIRAC. The DIRACX project is a new incarnation of DIRAC that will ensure adherence to the cited requirements, and business continuity with the existing DIRAC user communities.

## 4 What is DIRACX

DIRACX is a cloud native application which, as much as possible, follows the existing state-of-the-art standards. It is multi-VO from the get-go, with single VO installations being special cases of multi-VO ones. In terms of functionalities DIRACX is the direct continuation of DIRAC, sharing all databases with it.

As the general architecture diagram provided in figure 5 shows, users can interface with DIRACX through a REST API developed using FastAPI [20]. The API is secured by OAuth2, with users authenticating through external identity providers like IAM. The DIRACX tasks will be implemented using Celery [21], backed up by RabbitMQ. MinIO [22] is added for storing the jobs input and output sandboxes, replacing DIRAC's disk server.

<figure><img src="image_6.png" /><figcaption>Figure 5: DIRAC architecture</figcaption></figure>
DIRACX also provides a new web interface, developed using NextJS [23], Material UI [24] and typescript.

Kubernetes will be the only supported technology for the deployment of DIRACX and its web interface. Comprehensive Helm [25] charts are provided, used for testing (in GitHub Actions), for local developments, running demos and productions.

## 4.1 Dealing with existing DIRAC instances

There are several communities using DIRAC right now, and they will all eventually need to migrate to DIRACX. The business continuity for all of them is one of our top priorities. Effectively, the continuity is assured by the fact that DIRAC shares its databases with DIRACX, so DIRAC services and DIRACX APIs will both have the ability to read and write them. A "legacy adaptor" will be used to move traffic from DIRAC to DIRACX services, as shown in figure 6. Once the transition is over, the old DIRAC services can be removed.

By now, we know that it is sometimes necessary to extend all Dirac(X) components. DiracX aims to provide an easy way to do so, and for DIRACX and DIRACX-Web the developers already provided reference extensions.

<figure><img src="image_7.png" /><figcaption>Figure 6: DIRAC legacy adaptors: transition from before to after using an adaptor.</figcaption></figure>

## 4.2 On Authentication and Authorization

The DIRACX security model<fnref n="1" /> is built around the OAuth 2.0 authorisation flow (see figure 7), with the exception that an identity must always be present alongside authorisation to enable user-specific behaviors such as limiting file access to a user's/group's files (POSIX style), enforcing user/group-specific quotas on compute and storage resources, allowing for easier traceability. For communication between users and DIRACX services JSON Web Tokens (JWTs [26]) are used as OAuth2 bearer tokens. The same mechanism is used whenever impersonation is required, such as running a user-provided payload in a job. The need to use encrypted JWT is not foreseen.

DIRACX needs to interact with DIRAC on one side, and with the Grid on the other. DIRAC does not understand OAuth2 and JWT, while uses X509 certificates and proxies. The Grid world is in a transition phase from VOMS (and certificates, and proxies) to OAuth2 based

<footnote n="1">¹https://github.com/DIRACGrid/diracx/blob/main/security_model.md</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 7: DIRAC authorization with standard Code Flow, redirecting to IdP</figcaption></figure>

tokens. Within the Grid, proxies and/or tokens are needed for defining identity (community membership), for submitting pilots, for data access, and for verifying a user's identity. In order to let DIRAC interacts with DIRACX services, we implemented a proxy exchange, depicted in figure 8.

# 5 Summary

DIRACX is "the neXt DIRAC incarnation", ensuring the future of the widely used DIRAC. We are rewriting the code, but it is still DIRAC that you love! DIRACX will ease the interoperability with Rucio and/or dask and/or any other tool out there. DIRACX will still have the Data Management part, but its Workload Management functionalities will come first. The first DIRACX release will be here in Q2 2025: it will live together with DIRAC v9 for a while, until it will replace it completely.

# References

[1] F. Stagni, A. Boyer, A. Tsaregorodtsev, A. Lytovchenko, A. Sailer, C. Haen, C. Burr, D. Bauer, S. Fayer, J. Martyniak et al., DIRAC current, upcoming and planned capabilities and technologies, EPJ Web of Conferences **295** (2024). 10.1051/epj-conf/202429504018

[2] Z. Mathe, P. Charpentier, Optimising query execution time in LHCb bookkeeping system using partition pruning and partition-wise joins, Journal of Physics: Conference Series **513**, 042032 (2014).

[3] M. Barisits, T. Beermann, F. Berghaus, B. Bockelman, J. Bogado, D. Cameron, D. Christidis, D. Ciangottini, G. Dimitrov, M. Elsing et al., Rucio: Scientific data management, Computing and Software for Big Science **3**, 11 (2019). 10.1007/s41781-019-0026-3
<figure><img src="image_9.png" /><figcaption>Figure 8: DIRAC exchanges a proxy for a DIRACX token</figcaption></figure>

[4] Arrabito, Luisa, Bregeon, Johan, Faure, Alice, Gueta, Orel, Pigoux, Natthan, Tsare-gorodtsev, Andrei, The Cherenkov Telescope Array Observatory workflow management system, EPJ Web of Conf. **295**, 04044 (2024). 10.1051/epjconf/202429504044

[5] G. La Rocca, EGI e-infrastructure - Advanced computing services for science (2023), https://doi.org/10.5281/zenodo.7925603

[6] F. Stagni, A. McNab, C. Luzzi, W. Krzemien, DIRAC universal pilots, Journal of Physics: Conference Series **898**, 092024 (2017). 10.1088/1742-6596/898/9/092024

[7] F. Stagni, A. Boyer, C. Haen, C. Burr, V. Romanovskiy, C. Bozzi, Integrating LHCb offline workflows on supercomputers state of practice, EPJ Web of Conferences **295** (2024). 10.1051/epjconf/202429510005

[8] A. Kiryanov, A.A. Ayllon, O. Keeble, FTS3 / WebFTS – A Powerful File Transfer Service for Scientific Communities, Procedia Comp. Sci. **66**, 670 (2015). 10.1016/j.procs.2015.11.076

[9] A. Casajús, R. Graciani Diaz, DIRAC distributed secure framework, Journal of Physics: Conference Series **219**, 042033 (2010). 10.1088/1742-6596/219/4/042033

[10] I. Sencha, Ext JS: High-Performance JavaScript Framework for Enterprise Applications, Sencha, Inc. (2024), available at https://www.sencha.com/products/extjs/, https://www.sencha.com/products/extjs/

[11] V. Welch, M. Thompson, D.E. Engert, S. Tuecke, L. Pearlman, Internet X.509 Public Key Infrastructure (PKI) Proxy Certificate Profile, RFC 3820 (2004), https://www.
rfc-editor.org/info/rfc3820

[12] R. Alfieri, R. Cecchini, V. Ciaschini, L. dell'Agnello, A. Frohner, A. Gianoli, K. Lörentey, F. Spataro, VOMS, an Authorization System for Virtual Organizations., in *European Across Grids Conference*, edited by F.F. Rivera, M. Bubak, A. Gómez-Tato, R. Doallo (Springer, 2003), Vol. 2970 of *Lecture Notes in Computer Science*, pp. 33–40, ISBN 3-540-21048-2

[13] D. Hardt, The OAuth 2.0 Authorization Framework (Internet Engineering Task Force (IETF), 2012), RFC 6749, https://datatracker.ietf.org/doc/html/rfc6749

[14] R.T. Fielding, Architectural Styles and the Design of Network-based Software Architectures (2000), doctoral dissertation, University of California, Irvine, https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf

[15] T.K. Authors, Kubernetes Documentation, The Cloud Native Computing Foundation (2024), version 1.28 (or replace with relevant version), https://kubernetes.io/docs/

[16] Worldwide LHC computing grid (WLCG), https://wlcg.web.cern.ch/ (2024), collaborative computing infrastructure for storing, distributing, and analyzing LHC data

[17] European Grid Infrastructure (EGI), https://www.egi.eu/ (2024), collaborative e-infrastructure providing integrated computing and storage resources

[18] W. Authentication, A.W. Group, Identity and Access Management in Worldwide LHC Computing Grid, https://wlcg.web.cern.ch/technology/authentication-and-authorization (2024), identity management infrastructure for WLCG computing resources

[19] Check-IN: EGI Federation Authentication and Authorization Service, https://www.egi.eu/services/check-in/ (2024), federated identity and access management service for EGI infrastructure

[20] S. Ramírez, FastAPI: Fast, modern, and easy to use web framework for building APIs with Python (2024), accessed: 2024-06-17, https://fastapi.tiangolo.com/

[21] Celery Contributors, Celery: Distributed Task Queue for Python (2024), accessed: 2024-06-17, https://docs.celeryq.dev/

[22] MinIO, Inc., MinIO: High performance object storage (2024), accessed: 2024-06-17, https://min.io

[23] Vercel, Next.js: The React Framework for Production (2024), accessed: 2024-06-17, https://nextjs.org

[24] MUI Team, Material UI: React components for faster and easier web development (2024), accessed: 2024-06-17, https://mui.com

[25] The Helm Authors, Helm: The Kubernetes Package Manager (2024), accessed: 2024-06-17, https://helm.sh

[26] M.B. Jones, J. Bradley, N. Sakimura, RFC 7519: JSON Web Token (JWT) (2015), accessed: 2024-06-17, https://datatracker.ietf.org/doc/html/rfc7519