bluuebunny/crossref_metadata_2025 huggingface dataset:
The CMS Inner Tracker DAQ system for the High Luminosity upgrade of LHC: From single-chip testing, to large-scale assembly qualification
Authors: Dinardo Mauro Emanuele
Published: May 2024

To cope with the challenging environment of the planned high luminosity upgrade of the Large Hadron Collider (HL-LHC), scheduled to start operation in the late 2020s, CMS will replace its entire tracking system. The requirements for the tracker are largely determined by the long operation time of 10 years with a peak instantaneous luminosity of up to 7.5 × 1034 cm−2 s−1 in the ultimate performance scenario. In particular, the Inner Tracker (IT) is being completely redesigned featuring a front-end chip capable to deal with hit rates of up to 3.38 GHz/cm2. The communication between the front-end and the back-end electronics occurs through an optical link based on a custom Low-power Gigabit Transceiver which sends data at 10 and 2.5 Gbps on the uplink and downlink, respectively. The number of pixels has been increased by a factor of 6 with respect to the present detector, resulting in an unprecedented number of channels of about two billion, and covering a pseudorapidity region up to 4. This represents a challenging requirement for the data acquisition system since it needs to efficiently configure, monitor, and calibrate them. A dedicated data acquisition system, written in C++ and based on a custom µTCA board to handle trigger, data, and detector control, equipped with an FPGA, was developed to fully test and characterize the IT modules on a bench and with beam tests. In this note, we will describe the system architecture and its scalability to the final system which will be based on custom back-end boards equipped with FPGAs and CPUs.
Track reconstruction for the ATLAS Phase-II Event Filter using GNNs on FPGAs
Authors: Dittmeier Sebastian
Published: May 2024

The High-Luminosity LHC (HL-LHC) will provide an order of magnitude increase in integrated luminosity and enhance the discovery reach for new phenomena. The increased pile-up necessitates major upgrades to the ATLAS detector and trigger. The Phase-II trigger will consist of two levels, a hardware-based Level-0 trigger and an Event Filter (EF) with tracking capabilities. Within the Trigger and Data Acquisition group, a heterogeneous computing farm consisting of CPUs and potentially GPUs and/or FPGAs is under study, together with the use of modern machine learning algorithms such as Graph Neural Networks (GNNs).

GNNs are a powerful class of geometric deep learning methods for modelling spatial dependencies via message passing over graphs. They are well-suited for track reconstruction tasks by learning on an expressive structured graph representation of hit data and considerable speedup over CPU-based execution is possible on FPGAs.

The focus of this publication is a study of track reconstruction for the Phase-II EF system using GNNs on FPGAs. We explore each of the steps in a GNN-based EF tracking pipeline: graph construction, edge classification using an interaction network, and track reconstruction. Several methods and hardware platforms are under evaluation, studying resource utilisation and minimization of model size using quantization aware training, while simultaneously retaining high track reconstruction efficiency and low fake rates required for the EF tracking system.
Triggerless data acquisition pipeline for Machine Learning based statistical anomaly detection
Authors: Grosso Gaia , Lai Nicolò , Migliorini Matteo , Pazzini Jacopo , Triossi Andrea , Zanetti Marco , Zucchetta Alberto
Published: May 2024

This work describes an online processing pipeline designed to identify anomalies in a continuous stream of data collected without external triggers from a particle detector. The processing pipeline begins with a local reconstruction algorithm, employing neural networks on an FPGA as its first stage. Subsequent data preparation and anomaly detection stages are accelerated using GPGPUs. As a practical demonstration of anomaly detection, we have developed a data quality monitoring application using a cosmic muon detector. Its primary objective is to detect deviations from the expected operational conditions of the detector. This serves as a proof-of-concept for a system that can be adapted for use in large particle physics experiments, enabling anomaly detection on datasets with reduced bias.
A parameter optimisation toolchain for Monte Carlo detector simulation
Authors: Volkel Benedikt , Concas Matteo , Wenzel Sandro , Morsch Andreas
Published: May 2024

Monte Carlo detector transport codes are one of the backbones in high-energy physics computing. They simulate the transport of a large variety of different particle types through complex detector geometries based on different physics models. Those simulations are usually configurable through a large set of parameters allowing for some tuning on the client side. Often, tuning the physics accuracy on the one hand and optimising the resource needs on the other hand are competing requirements. In this area, we are presenting a toolchain to tune Monte Carlo transport codes which is capable of automatically optimising large sets of parameters based on user-defined metrics. The toolchain consists of two central components. Firstly, the MCReplayEngine which is a quasi-Monte-Carlo transport engine able to fast replay pre-recorded MC steps. This engine for instance allows one to study the impact of parameter variations on quantities such as hits without the need to perform new full simulations. Secondly, it consists of an automatic and generic parameter optimisation framework called O2Tuner. The toolchain’s application in concrete use-cases will be presented. Its first application in ALICE led to a reduction of CPU time of Monte Carlo detector transport by 30%. In addition, further possible scenarios will be discussed.
A dimensions aware evaluator for High Energy Physics applications
Authors: Couturier Ben , Clemencic Marco
Published: May 2024

The LHCb software stack is developed in C++ and uses the Gaudi framework for event processing and DD4hep for the detector description. Numerical computations are done either directly in the C++ code or by an evaluator used to process the expressions embedded in the XML describing the detector geometry. The current system relies on conventions for the physical units used (identical as what is used in the Geant4 simulation framework) and it is up to the developers to ensure that the correct factors are applied to the values entered. Physical units are not primary entities in the framework, it is therefore not possible to check the dimensional consistency of the computation performed. In this paper we investigate the possibilities to add physical units and dimensions to the existing evaluator or to replace it by a more suitable system, and how this would integrate with the possible tools to express units in C++ code (such as boost::units).
CMS Tier-0 data processing during the detector commissioning in Run 3
Authors: Amado Jhonatan , Eysermans Jan , Giraldo German , Hufnagel Dirk , Kovalskyi Dmytro , Linares Sancho Antonio
Published: May 2024

The CMS Tier-0 system is responsible for the prompt processing and distribution of the data collected by the CMS Experiment. A number of upgrades were implemented during the long shutdown 2 of the Large Hadron Collider, which improved the performance and reliability of the system. In this report, these upgrades are discussed and a more detailed description of the Tier-0 system is given. The experience of the data taking during Run 3 detector commissioning as well as the performance of the system are highlighted.
Fast Integration of Poisson Distributions for Dead Sensor Marginalization
Authors: Bilodeau Zoë , Peters Christina , Tunnell Christopher
Published: May 2024

In dual-phase time-projection chambers there are photosensor arrays arranged to allow for inference of the positions of interactions within the detector. If there is a gap in data left by a broken or saturated photosensors, the inference of the position is less precise and less accurate. As we are unable to repair or replace photosensors once the experiment has begun, we develop methods to estimate the missing signals. Our group is developing a probabilistic graphical model of the correlations between the number of photons detected by adjacent photosensors that represents the probability distribution over photons detected as a Poisson distribution. Determining the posterior probability distribution over a number of photons detected by a sensor then requires integration over a multivariate Poisson distribution, which is computationally intractable for high-dimensions. In this work, we present an approach to quickly calculate and integrate over a multidimensional Poisson distribution. Our approach uses Zarr, a Python array compression package, to manage large multi-dimensional arrays and approximates the log factorial to quickly calculate the Poisson distribution without overflow.
Optimizing ATLAS data storage: The impact of compression algorithms on ATLAS physics analysis data formats
Authors: Marcon Caterina , Mete Alaettin Serhan , Van Gemmeren Peter , Carminati Leonardo
Published: May 2024

The increased footprint foreseen for Run-3 and HL-LHC data will soon expose the limits of currently available storage and CPU resources. Data formats are already optimized according to the processing chain for which they are designed. ATLAS events are stored in ROOT-based reconstruction output files called Analysis Object Data (AOD), which are then processed within the derivation framework to produce Derived AOD (DAOD) files. Numerous DAOD formats, tailored for specific physics and performance groups, have been in use throughout the ATLAS Run-2 phase. In view of Run-3, ATLAS has changed its analysis model, which entailed a significant reduction of the existing DAOD flavors. Two new formats, unfiltered, skimmable on read and designed to meet the requirements of the majority of the analysis workflows, have been proposed as replacements: DAOD_PHYS and DAOD_PHYSLITE, a smaller format containing already calibrated physics objects. As ROOT-based formats, they natively support four lossless compression algorithms: lzma, lz4, zlib and zstd. In this study, the effects of different compression settings on file size, compression time, compression factor and reading speed are investigated considering both DAOD_PHYS and DAOD_PHYSLITE formats. Moreover, the impact of the AutoFlush parameter controlling how in-memory data structures are serialized to ROOT files, has been evaluated. This study yields new quantitative results that can serve as a paradigm on how to make compression decisions for different ATLAS use cases. As an example, for both DAOD_PHYS and DAOD_PHYSLITE, the lz4 library exhibits the fastest reading speed, but results in the largest files, whereas the lzma algorithm provides larger compression factors at the cost of significantly slower reading speeds. In addition, guidelines for setting appropriate AutoFlush values are outlined.
Repurposing of the Run 2 CMS High Level Trigger Infrastructure as a Cloud Resource for Offline Computing
Authors: Mascheroni Marco , Pérez-Calero Yzquierdo Antonio , Kizinevic Edita , Khan Farrukh Aftab , Kim Hyunwoo , Acosta Flechas Maria , Tsipinakis Nikos , Haleem Saqib , Spiga Damiele , Wissing Christoph , Würthwein Frank
Published: May 2024

The former CMS Run 2 High Level Trigger (HLT) farm is one of the largest contributors to CMS compute resources, providing about 25k job slots for offline computing. This CPU farm was initially employed as an opportunistic resource, exploited during inter-fill periods, in the LHC Run 2. Since then, it has become a nearly transparent extension of the CMS capacity at CERN, being located on-site at the LHC interaction point 5 (P5), where the CMS detector is installed. This resource has been configured to support the execution of critical CMS tasks, such as prompt detector data reconstruction. It can therefore be used in combination with the dedicated Tier 0 capacity at CERN, in order to process and absorb peaks in the stream of data coming from the CMS detector. The initial configuration for this resource, based on statically configured VMs, provided the required level of functionality. However, regular operations of this cluster revealed certain limitations compared to the resource provisioning and use model employed in the case of WLCG sites. A new configuration, based on a vacuum-like model, has been implemented for this resource in order to solve the detected shortcomings. This paper reports about this redeployment work on the permanent cloud for an enhanced support to CMS offline computing, comparing the former and new models’ respective functionalities, along with the commissioning effort for the new setup.
The LHCb ultra-fast simulation option, Lamarr design and validation
Authors: Anderlini Lucio , Barbetti Matteo , Capelli Simone , Corti Gloria , Davis Adam , Derkach Denis , Kazeev Nikita , Maevskiy Artem , Martinelli Maurizio , Mokonenko Sergei , Siddi Benedetto G. , Xu Zehua
Published: May 2024

Detailed detector simulation is the major consumer of CPU resources at LHCb, having used more than 90% of the total computing budget during Run 2 of the Large Hadron Collider at CERN. As data is collected by the upgraded LHCb detector during Run 3 of the LHC, larger requests for simulated data samples are necessary, and will far exceed the pledged resources of the experiment, even with existing fast simulation options. The evolution of technologies and techniques for simulation production is then mandatory to meet the upcoming needs for the analysis of most of the data collected by the LHCb experiment. In this context, we propose Lamarr, a Gaudi-based framework designed to offer the fastest solution for the simulation of the LHCb detector. Lamarr consists of a pipeline of modules parameterizing both the detector response and the reconstruction algorithms of the LHCb experiment. Most of the parameterizations are made of Deep Generative Models and Gradient Boosted Decision Trees trained on simulated samples or alternatively, where possible, on real data. Embedding Lamarr in the general LHCb Gauss Simulation framework allows combining its execution with any of the available generators in a seamless way. Lamarr has been validated by comparing key reconstructed quantities with Detailed Simulation. Good agreement of the simulated distributions is obtained with two order of magnitude speed-up of the simulation phase.
All grown-up; 18 years of LHC@home
Authors: Cameron David , Field Laurence , Van der Veken Frederik , Høimyr Nils , Di Croce Davide , Gaillard Melissa , Masullo Germano , Noble Cath , Reguero Ignacio , Reid Ivan , Segal Ben
Published: May 2024

LHC@home was launched as a BOINC project in 2004 as an outreach project for CERN’s 50 years anniversary. Initially focused on the accelerator physics simulation code SixTrack, the project was expanded in 2011 to run other physics simulation codes on Linux thanks to virtualisation. Later on the experiment and theory applications running on the LHC@home platform have evolved to use containers and take advantage of the CVMFS file system as well as content delivery networks. Furthermore, a substantial part of the contributed computing capacity nowadays is provided as opportunistic back-fill from data centers with spare capacity, in addition to enthusiastic volunteers. The paper will address the challenges with this distributed computing model, new applications to exploit GPUs and the future outlook for volunteer computing.
Collaborative Operational Security: The future of Cybersecurity for Research and Education
Authors: Crooks David , Acris James , Atherton Liam , Clark Paul , Cutrina Pau , Jordan David , McKee Shawn , Vâlsan Liviu
Published: May 2024

No single organisation has the resources to defend its services alone against most modern malicious actors and so we must protect ourselves as a community. In the face of determined and well-resourced attackers, we must actively collaborate in this effort across HEP and more broadly across Research and Education (R&E).

Parallel efforts are necessary to respond appropriately to this requirement. We must share threat intelligence about ongoing cybersecurity incidents with our trusted partners and deploy the fine-grained security network monitoring necessary to make active use of this intelligence. We must also engage with senior management in our organizations to ensure that we work alongside any broader organisational cybersecurity development programs.

We report on progress of the Security Operations Center (SOC) Working Group, established by the WLCG but with membership encompassing the R&E sector. The goal of the Working Group is to develop reference designs for SOC deployments and empower R&E organisations to collect, leverage, and act upon targeted, contextualized, actionable threat intelligence. This report will include recent SOC deployment activities at sites with network connectivity in excess of 100Gb/s, as well as new technology designs. An important development, which is likely to form a key part of the WLCG security strategy, is the potential use of passive DNS logs to allow sites without fine-grained network monitoring to benefit from the threat intelligence available to our community.

We also report on higher-level progress in engaging with the broader community to establish common approaches to this vital area of cybersecurity.
Computing Activities at the Spanish Tier-1 and Tier-2s for the ATLAS experiment in the LHC Run 3 period and towards High Luminosity (HL-LHC)
Authors: González de la Hoz Santiago , Acin Vanesa , Accion Esther , Acosta-Silva Carles , Aparisi Javier , Collado Soto Pablo , del Peso Jose , Fernández Casani Álvaro , Flix Jose , García Montoro Carlos , Merino Gonzalo , Pacheco Pages Andreu , Planas Elena , Sánchez Javier , Salt Jose , Villaplana Perez Miguel
Published: May 2024

The ATLAS Spanish Tier-1 and Tier-2s have more than 18 years of experience in the deployment and development of LHC computing components and their successful operation. The sites are actively participating in, and in some cases coordinating, R&D computing activities in the LHC Run 3 and developing the computing models needed in the HL-LHC period. In this contribution, we present details on the integration of some components, such as HPC computing resources to execute ATLAS simulation workflows; the development of new techniques to improve efficiency in a cost-effective way; and improvements in Data Organization, Management and Access through storage consolidations, the use of data caches, and improving experiment data catalogues, through contributions such as Event Index. The design and deployment of novel analysis facilities using GPUs together with CPUs and techniques like Machine Learning are also presented. ATLAS Tier-1 and Tier-2 sites in Spain, are, and will be, contributing to significant R&D in computing and evaluating different models for improving performance of computing and data storage capacity in the LHC High Luminosity era.
Standardizing DIRAC’s Cloud Interfaces
Authors: Bauer Daniela , Fayer Simon
Published: May 2024

DIRAC is a widely used framework for distributed computing. It provides a layer between users and computing resources by offering a common interface to a number of heterogeneous resource providers. In these proceedings we describe a new implementation of the DIRAC to Cloud interface.
Microarchitecture: A useful tool to organize machines in heterogeneous shared computing environments
Authors: Thain Gregory , Sfiligoi Igor
Published: May 2024

The x86_64 instruction set architecture is not a single, consistent, compatible interface to execute computer programs. Since the initial release in 1999, every new generation has added new instructions, some of which were later removed. Most of these new instructions are intended to improve the performance of those programs which explicitly take advantage of them. However, running such a program on older CPUs without appropriate support, results in Linux SIGILL exception signal, which is difficult for end users to diagnose. On the other hand, compiling scientific code for the least common denominator ISA can leave significant performance on the table. High Throughput systems, containing very large number of machines, cannot require a single CPU version across hundreds of thousands of machines operating in dozens of sites. The OSG Open Science Pool alone consists of more than 20 different, subtly incompatible X86_64 implementations. In 2020, Intel, AMD and RedHat proposed new terminology and partitioned these dozens of microarchitectures into a strict hierarchy of four groups. The HTCondor Software Suite and the OSG now have first class support for these microarchitectures. This paper discusses the advantages for users and future work around microarchitecture support.
The ATLAS Workflow Management System Evolution in the LHC Run3 and towards the High-Luminosity LHC era
Authors: Pacheco Pages Andrés , Klimentov Alexei
Published: May 2024

The ATLAS experiment has 18+ years of experience using workload management systems to deploy and develop workflows to process and to simulate data on the distributed computing infrastructure. Simulation, processing and analysis of LHC experiment data require the coordinated work of heterogeneous computing resources. In particular, the ATLAS experiment utilizes the resources of 250 computing centers worldwide, the power of supercomputing centres, and national, academic and commercial cloud computing resources. In this contribution, we present new techniques for cost-effectively improving efficiency introduced in workflow management system software. The evolution from a mesh framework to new types of computing facilities such as cloud and HPCs is described, as well as new types of production and analysis workflows.
The Cherenkov Telescope Array Observatory workflow management system
Authors: Arrabito Luisa , Bregeon Johan , Faure Alice , Gueta Orel , Pigoux Natthan , Tsaregorodtsev Andrei
Published: May 2024

The Cherenkov Telescope Array Observatory (CTAO) is the next generation ground-based observatory for gamma-ray astronomy at very high energies. It is expected to produce about 2 PB of raw data each year and to manage a global data volume which will grow through the years to reach about 100 PB in 2030. In addition, CTAO will require a high computing capacity for data processing and Monte Carlo simulations, of the order of hundreds of millions of CPU HS06 hours per year. To meet these requirements, CTAO will adopt a distributed computing model using 4 academic data centers, and will use the DIRAC framework as its workload management system. In the past ten years, to optimize the instrument design and study its performances, CTAO has used the European Grid Infrastructure (EGI) to run massive Monte Carlo campaigns. In order to handle these campaigns and to automatize simulation and data processing workflows, we have developed a production system prototype based on DIRAC. Recently, we have also developed a user interface allowing for the configuration and submission of complex workflows. In this contribution we present the production system prototype, its user interface for workflow management as well as its application to CTAO workflows.
The Rubin Observatory’s Legacy Survey of Space and Time DP0.2 processing campaign at CC-IN2P3
Authors: Le Boulc’h Quentin , Hernandez Fabio , Mainetti Gabriele
Published: May 2024

The Vera C. Rubin Observatory, currently in construction in Chile, will start performing the Legacy Survey of Space and Time (LSST) in 2025 for 10 years. Its 8.4-meter telescope will survey the southern sky in less than 4 nights in six optical bands, and repeatedly generate about 2 000 exposures per night, corresponding to a data volume of about 20 TiB every night. Three data facilities are preparing to contribute to the production of the annual data releases: the US Data Facility will process 35% of the raw data, the UK data facility will process 25% of the raw data and the French data facility, operated by CC-IN2P3, will locally process the remaining 40% of the raw data.

In the context of the Data Preview 0.2 (DP0.2), the Data Release Production pipelines have been executed on the DC-2 simulated dataset (generated by the Dark Energy Science Collaboration, DESC). This dataset includes 20 000 simulated exposures, representing 300 square degrees of Rubin images with a typical depth of 5 years.

DP0.2 ran at the Interim Data Facility (based on Google cloud), and the full exercise was independently replicated at CC-IN2P3. During this exercise, 3 PiB of data and more than 200 million files were produced. In this contribution we will present a detailed description of the system that we set up to perform this processing campaign using CC-IN2P3’s computing and storage infrastructure. Several topics will be addressed: workflow generation and execution, batch job submission, memory and I/O requirements, etc. We will focus on the issues that arose during this campaign and how we addressed them and will present some perspectives after this exercise.
Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS
Authors: Maeno Tadashi , Alekseev Aleksandr , Barreiro Megino Fernando Harald , De Kaushik , Guan Wen , Karavakis Edward , Klimentov Alexei , Korchuganova Tatiana , Lin FaHui , Nilsson Paul , Wenaus Torre , Yang Zhaoyu , Zhao Xin
Published: May 2024

In recent years, advanced and complex analysis workflows have gained increasing importance in the ATLAS experiment at CERN, one of the large scientific experiments at LHC. Support for such workflows has allowed users to exploit remote computing resources and service providers distributed worldwide, overcoming limitations on local resources and services. The spectrum of computing options keeps increasing across the Worldwide LHC Computing Grid (WLCG), volunteer computing, high-performance computing, commercial clouds, and emerging service levels like Platform-as-a-Service (PaaS), Container-as-a-Service (CaaS) and Function-as-a-Service (FaaS), each one providing new advantages and constraints. Users can significantly benefit from these providers, but at the same time, it is cumbersome to deal with multiple providers, even in a single analysis workflow with fine-grained requirements coming from their applications’ nature and characteristics. In this paper, we will first highlight issues in geographically-distributed heterogeneous computing, such as the insulation of users from the complexities of dealing with remote providers, smart workload routing, complex resource provisioning, seamless execution of advanced workflows, workflow description, pseudointeractive analysis, and integration of PaaS, CaaS, and FaaS providers. We will also outline solutions developed in ATLAS with the Production and Distributed Analysis (PanDA) system and future challenges for LHC Run4.
Modern Software Development for JUNO offline software
Authors: Lin Tao
Published: May 2024

The Jiangmen Underground Neutrino Observatory (JUNO), under construction in South China, primarily aims to determine the neutrino mass hierarchy and to precise measure the neutrino oscillation parameters. The data-taking is expected to start in 2024 and the detector plans to run for more than 20 years. The development of the JUNO offline software (JUNOSW) started in 2012, and it is quite challenging to maintain the JUNOSW for such a long time. In the last ten years, tools such as Subversion, Trac, and CMT had been adopted for software development. However, new stringent requirements came out, such as how to reduce the building time for the whole project, how to deploy offline algorithms to an online environment, and how to improve the code quality with code review and continuous integration. To meet the further requirements of software development, modern development tools are evaluated for JUNOSW, such as Git, GitLab, CMake, Docker, and Kubernetes. This contribution will present the software development system based on these modern tools for JUNOSW and the functionalities achieved: CMake macros are developed to simplify the build instructions for users; CMake generator expressions are used to control the build flags for the online and offline environments; a tool named git-junoenv is developed to help users partially checkout and build the software; a script is used to build and deploy the software on the CVMFS server; a Docker image with CVMFS client installed is created for continuous integration; a GitLab agent is set up to manage GitLab runners in Kubernetes with all the configurations in a GitLab repository.
Multilanguage Frameworks Towards the Ideal Multilanguage Environment
Authors: Hřivnáč Julius
Published: May 2024

High Energy Physics software has been a victim of the necessity to choose one implementation language as no really usable multi-language environment existed. Even a co-existence of two languages in the same framework (typically C++ and Python) imposes a heavy burden on the system. The role of different languages was generally limited to well encapsulated domains (like Web applications, databases, graphics), with very limited connection to the central framework.

The new development in the domain of the compilers and run-time environments has enabled ways for creating really multilanguage frameworks, with seamless, user-friendly and high-performance inter-operation of many languages, which traditionally live in disconnected domains (like C-based languages vs JVM languages or Web languages).

Various possibilities and strategies for creation of the true multi-language frameworks are discussed, emphasizing their advantages and possible road blocks.
The ATLAS experiment software on ARM
Authors: Elmsheuser Johannes , Barreiro Megino Fernando , De Salvo Alessandro , De Silva Asoka , Hauser Reiner , Konstantinov Dmitri , Krasznahorkay Attila , Lassnig Mario , Sailer Andre , Snyder Scott
Published: May 2024

With an increased dataset obtained during the Run 3 of the LHC at CERN and the even larger expected increase of the dataset by more than one order of magnitude for the HL-LHC, the ATLAS experiment is reaching the limits of the current data processing model in terms of traditional CPU resources based on x86_64 architectures and an extensive program for software upgrades towards the HL-LHC has been set up. The ARM architecture is becoming a competitive and energy efficient alternative. Some surveys indicate its increased presence in HPCs and commercial clouds, and some WLCG sites have expressed their interest. Chip makers are also developing their next generation solutions on ARM architectures, sometimes combining ARM and GPU processors in the same chip. Consequently it is important that the ATLAS software embraces the change and is able to successfully exploit this architecture. We report on the successful porting to ARM of the Athena software framework, which is used by ATLAS for both online and offline computing operations. Furthermore we report on the successful validation of simulation workflows running on ARM resources. For this we have set up an ATLAS Grid site using ARM compatible middleware and containers on Amazon Web Services (AWS) ARM resources. The ARM version of Athena is fully integrated in the regular software build system and distributed in the same way as other software releases. In addition, the workflows have been integrated into the HEPscore benchmark suite which is the planned WLCG wide replacement of the HepSpec06 benchmark used for Grid site pledges. In the overall porting process we have used resources on AWS, Google Cloud Platform (GCP) and CERN. A performance comparison of different architectures and resources will be discussed.
Dark Matter Science Project
Authors: Little Jared , Doglioni Caterina , Heinrich Lukas
Published: May 2024

A Dark Matter Science Project is being developed in the context of the ESCAPE (European Science Cluster of Astronomy and Particle physics ESFRI research infrastructure) project as a collaboration between scientists in European Research Infrastructures and experiments seeking to explain the nature of dark matter (such as HL-LHC, KM3NeT, CTA, DarkSide). The goal of this ESCAPE Science Project is to highlight the synergies between different dark matter communities and experiments, by producing new scientific results as well as by making the necessary data and software tools fully available. As part of this Science Project, we use experimental data and software algorithms from selected direct detection, indirect detection, and particle collider experiments involved in ESCAPE as prototypes for end-to-end analysis pipelines on a Virtual Research Environment that is being prepared as one of the building blocks of the European Open Science Cloud (EOSC). This contribution focuses on the implementation of the workflows on the Virtual Research Environment using ESCAPE tools (such as the Data Lake and REANA), and on the prospects for data management, data analysis and computing in the EOSC-Future project.
Making Likelihood Calculations Fast: Automatic Differentiation Applied to RooFit
Authors: Singh Garima , Rembser Jonas , Moneta Lorenzo , Lange David , Vassilev Vassil
Published: May 2024

With the growing datasets of current and next-generation HighEnergy and Nuclear Physics (HEP/NP) experiments, statistical analysis has become more computationally demanding. These increasing demands elicit improvements and modernizations in existing statistical analysis software. One way to address these issues is to improve parameter estimation performance and numeric stability using Automatic Differentiation (AD). AD’s computational efficiency and accuracy are superior to the preexisting numerical differentiation techniques, and it offers significant performance gains when calculating the derivatives of functions with a large number of inputs, making it particularly appealing for statistical models with many parameters. For such models, many HEP/NP experiments use RooFit, a toolkit for statistical modeling and fitting that is part of ROOT.

In this paper, we report on the effort to support the AD of RooFit likelihood functions. Our approach is to extend RooFit with a tool that generates overheadfree C++ code for a full likelihood function built from RooFit functional models. Gradients are then generated using Clad, a compiler-based source-codetransformation AD tool, using this C++ code. We present our results from applying AD to the entire minimization pipeline and profile likelihood calculations of several RooFit and HistFactory models at the LHC-experiment scale. We show significant reductions in calculation time and memory usage for the minimization of such likelihood functions. We also elaborate on this approach’s current limitations and explain our plans for the future.
PHYSLITE - A new reduced common data format for ATLAS
Authors: Schaarschmidt Jana , Catmore James , Elmsheuser Johannes , Heinrich Lukas , Krumnack Nils , Mete Serhan , Ozturk Nurcan
Published: May 2024

The High Luminosity LHC (HL-LHC) era brings unprecedented computing challenges that call for novel approaches to reduce the amount of real and Monte Carlo-simulated data that is stored, while continuing to support the rich physics program of the ATLAS experiment. With the beginning of LHC Run 3, ATLAS introduced a new common data format, PHYS, that replaces most of the analysis-specific formats that were used in Run 2, and therefore reduces the disk storage significantly. ATLAS also launched the prototype of another common format, PHYSLITE, that is about a third of the size of PHYS. PHYSLITE will be the main format for ATLAS at the HL-LHC and aims to serve 80% of all physics analyses. To simplify analysis workloads and further reduce disk usage it is designed to largely replace user-defined analysis n-tuples and consequently contains pre-calibrated objects. Various forms of validations are in place to ensure correct functionality for users. Developments continue towards HL-LHC to improve the PHYSLITE format further.
Towards podio v1.0 - A first stable release of the EDM toolkit
Authors: Carceller Juan Miguel , Gaede Frank , Ganis Gerardo , Hegner Benedikt , Helsens Clement , Madlener Thomas , Sailer André , Stewart Graeme A. , Volkl Valentin
Published: May 2024

A performant and easy-to-use event data model (EDM) is a key component of any HEP software stack. The podio EDM toolkit provides a user friendly way of generating such a performant implementation in C++ from a high level description in yaml format. Finalizing a few important developments, we are in the ﬁnal stretches for release v1.0 of podio, a stable release with backward compatibility for dataﬁles written with podio from then on. We present an overview of the podio basics, and go into slighty more technical detail on the most important topics and developments. These include: schema evolution for generated EDMs, multithreading with podio generated EDMs, the implementation of them as well as the basics of I/O. Using EDM4hep, the common and shared EDM of the Key4hep project, we highlight a few of the smaller features in action as well as some lessons learned during the development of EDM4hep and podio. Finally, we show how podio has been integrated into the Gaudi based event processing framework that is used by Key4hep, before we conclude with a brief outlook on potential developments after v1.0.
RootInteractive tool for multidimensional statistical analysis, machine learning and analytical model validation
Authors: Ivanov Marian , Ivanov Marian , Eulisse Giulio
Published: May 2024

The ALICE experiment [1] at CERN’s LHC is specifically designed for investigating heavy ion collisions. The upgraded ALICE accommodates a tenfold increase in Pb–Pb luminosity and a two-order-of-magnitude surge in minimum bias events. To address the challenges of high detector occupancy and event pile-ups, advanced multidimensional data analysis techniques, including machine learning (ML), are indispensable. Despite ML’s popularity, the complexity of its models presents interpretation challenges, and oversimplification in analysis often leads to inaccuracies.

Our objective was to develop RootInteractive, a tool for multidimensional statistical analysis. This tool simplifies data analysis across dimensions, visualizes functions with uncertainties, and validates assumptions and approximations. In RootInteractive, it is crucial to easily define the functional composition of analytical parametric and non-parametric functions, exploit symmetries, and define multidimensional "invariant" functions and corresponding alarms.

RootInteractive [2] adopts a declarative programming paradigm, ensuring userfriendliness for experts, students, and educators. It facilitates interactive visualization, n-dimensional histogramming/projection, and information extraction on both Python/C++ server and Javascript client. The tool supports client/server applications in Jupyter or standalone client-side applications. Through data compression, datasets with O(107) entries and O(25) attributes can be interactively analyzed in a browser with O(0.500-1 GB) size. Representative downsampling and reweighting/pre-aggregation enable the effective analysis of one year of ALICE data for various purposes.
ROOT’s RNTuple I/O Subsystem: The Path to Production
Authors: Blomer Jakob , Canal Philippe , de Geus Florine , Hahnfeld Jonas , Naumann Axel , Lopez-Gomez Javier , Lazzari Miotto Giovanna , Padulano Vincenzo Eduardo
Published: May 2024

The RNTuple I/O subsystem is ROOT’s future event data file format and access API. It is driven by the expected data volume increase at upcoming HEP experiments, e.g. at the HL-LHC, and recent opportunities in the storage hardware and software landscape such as NVMe drives and distributed object stores. RNTuple is a redesign of the TTree binary format and API and has shown to deliver substantially faster data throughput and better data compression both compared to TTree and to industry standard formats. In order to let HENP computing workflows benefit from RNTuple’s superior performance, however, the I/O stack needs to connect efficiently to the rest of the ecosystem, from grid storage to (distributed) analysis frameworks to (multithreaded) experiment frameworks for reconstruction and ntuple derivation. With the RNTuple binary format soon arriving at its first production release, we present RNTuple’s feature set, integration efforts, and its performance impact on the time-to-solution. We show the latest performance figures of RDataFrame analysis code of realistic complexity, comparing RNTuple and TTree as data sources. We discuss RNTuple’s approach to functionality critical to the HENP I/O (such as multithreaded writes, fast data merging, schema evolution) and we provide an outlook on the road to its use in production.
Anomaly Detection in Data Center IT & Physical Infrastructure
Authors: Ronchieri Elisabetta , Giommi Luca , Scarponi Luigi Benedettto , Torzi Luca , Costantini Alessandro , Duma Doina Cristina , Salomoni Davide
Published: May 2024

Anomaly detection in data center IT and physical infrastructure is challenging due to the amount of heterogeneous data to be analyzed. Defining a solution that early identifies unexpected anomalies is particularly important to prevent data losses, breakdown of the system, and any other event considered to be critical for the activity of the data center.

In the context of the INFN CNAF data center, one of the WLCG Tier-1s, we have performed a study based on monitored cooling, electrical, and IT hardware and software metrics to identify anomalies. In the present work, we aim to explore statistical approaches and machine learning solutions in the anomaly detection field for time series numerical metrics related to IT and physical infrastructure sensors.

With the usage of statistical Z-score and percentile approaches and clustering DBSCAN technique, we have been able to group and identify anomalous events. Using the presented approach, different relevance can be attributed to the likely anomalies.
Architecting the OpenSearch service at CERN
Authors: Papadopoulos Sokratis , Saiz Pablo , Schwickerath Ulrich , Kleszcz Emil
Published: May 2024

The centralised Elasticsearch service has been running at CERN since 2016, providing the search and analytics engine for numerous CERN users. The service has been based on the open-source version of Elasticsearch, surrounded by a set of external open-source plugins offering security, multitenancy, extra visualization types, and more. In October 2020, CERN embarked on an evaluation of OpenDistro for Elasticsearch, an alternative solution that used a different set of modules while retaining the core of open-source Elasticsearch. Notably, OpenDistro offered the advantage of bundling all components together, simplifying the deployment of new versions. This evaluation gained increased significance following a license change imposed by Elastic, the original creators of Elasticsearch. Consequently, the OpenDistro project was rebranded, now based on a forked version of Elasticsearch, called OpenSearch. Motivated by the license change and the streamlined deployment of the featurerich OpenSearch project as a fully open-source environment, the decision was taken to migrate the service at CERN towards it. The migration required a complete architectural redesign to accommodate the new modules while upholding the established standards of resource efficiency. The new service not only introduced a wide range of additional capabilities but also resolved longstanding maintainability issues while meeting the growing demands of various use-cases. At the time of writing, CERN’s service comprises 42 OpenSearch and 41 OpenDistro clusters in active production, plus 28 OpenSearch development clusters. This article covers the motivation, design, and implementation of this transition, highlighting the challenges encountered throughout the process.
Automatic Monitoring of Large-Scale Computing Infrastructure
Authors: Kim Bockjoo , Bourilkov Dimitri
Published: May 2024

Modern distributed computing systems produce large amounts of monitoring data. For these systems to operate smoothly, underperforming or failing components must be identified quickly, and preferably automatically, enabling the system managers to react accordingly. In this contribution, we analyze jobs and transfer data collected in the running of the LHC computing infrastructure. The monitoring data is harvested from the Elasticsearch database and converted to formats suitable for further processing. Based on various machine and deep learning techniques, we develop automatic tools for continuous monitoring of the health of the underlying systems. Our initial implementation is based on publicly available deep learning tools, PyTorch or TensorFlow packages, running on state-of-the-art GPU systems.
Coffea-Casa: Building composable analysis facilities for the HL-LHC
Authors: Albin Sam , Attebury Garhan , Bloom Kenneth , Bockelman Brian , Lundstedt Carl , Shadura Oksana , Thiltges John
Published: May 2024

The large data volumes expected from the High Luminosity LHC (HL-LHC) present challenges to existing paradigms and facilities for end-user data analysis. Modern cyberinfrastructure tools provide a diverse set of services that can be composed into a system that provides physicists with powerful tools that give them straightforward access to large computing resources, with low barriers to entry. The Coffea-Casa analysis facility (AF) provides an environment for end users enabling the execution of increasingly complex analyses such as those demonstrated by the Analysis Grand Challenge (AGC) and capturing the features that physicists will need for the HL-LHC.

We describe the development progress of the Coffea-Casa facility featuring its modularity while demonstrating the ability to port and customize the facility software stack to other locations. The facility also facilitates the support of batch systems while staying Kubernetes-native. We present the evolved architecture of the facility, such as the integration of advanced data delivery services (e.g. ServiceX) and making data caching services (e.g. XCache) available to end users of the facility. We also highlight the composability of modern cyberinfrastructure tools. To enable machine learning pipelines at coffee-casa analysis facilities, a set of industry ML solutions adopted for HEP columnar analysis were integrated on top of existing facility services. These services also feature transparent access for user workflows to GPUs available at a facility via inference servers while using Kubernetes as enabling technology.