bluuebunny/crossref_metadata_2025 huggingface dataset:
Framework for custom event sample augmentations for ATLAS analysis data
Authors: van Gemmeren Peter , Mete Alaettin Serhan , Burzynski Jackson Carl , Catmore James , Heinrich Lukas , Nowak Marcin , Krumnack Nils
Published: May 2024

For HEP event processing, data is typically stored in column-wise synchronized containers, such as most prominently ROOT’s TTree, which have been used for several decades to store by now over 1 exabyte. These containers can combine row-wise association capabilities needed by most HEP event processing frameworks (e.g. Athena for ATLAS) with column-wise storage, which typically results in better compression and more efficient support for many analysis use-cases. One disadvantage is that these containers, TTree in the HEP use-case, require to contain the same attributes for each entry/row (representing events), which can make extending the list of attributes very costly in storage, even if those are only required for a small subsample of events. Since the initial design, the ATLAS software framework features powerful navigational infrastructure to allow storing custom data extensions for subsamples of events in separate, but synchronized containers. This allows adding event augmentations to ATLAS standard data products (such as DAOD-PHYS or PHYSLITE) avoiding duplication of those core data products, while limiting their size increase. For this functionality, the framework does not rely on any associations made by the I/O technology (i.e. ROOT), however it supports TTree friends and builds the associated index to allow for analysis outside of the ATLAS framework. A prototype based on the Long-Lived Particle search is implemented and preliminary results with this prototype will be presented. At this point, augmented data are stored within the same file as the core data. Storing them in separate files will be investigated in future, as this could provide more flexibility, e.g. certain sites may only want a subset of several augmentations or augmentations can be archived to tape once their analysis is complete.
Multithreading ATLAS offline software: A retrospective
Authors: Krasznahorkay Attila , Leggett Charles , Mete Serhan Alaettin , Snyder Scott , Tsulaia Vakho , Winklmeier Frank
Published: May 2024

For Run 3, ATLAS redesigned its offline software, Athena, so that the main workflows run completely multithreaded. The resulting substantial reduction in the overall memory requirements allows for better use of machines with many cores. This note will discuss the performance achieved by the multithreaded reconstruction, the process of migrating the large ATLAS code base, and tools and techniques that were useful in debugging threading-related problems.
Offline Data Processing Software for the Super Tau Charm Facility
Authors: Li Teng , Huang Wenhao , Huang Xingtao , Ai Xiaocong , Li He , Liu Dong
Published: May 2024

The Super Tau Charm Facility (STCF) proposed in China is a newgeneration electron–positron collider with center-of-mass energies covering 2-7 GeV and a peak luminosity of 0.5 1035 cm−2 s−1. The offline software of STCF (OSCAR) is developed to support the offline data processing, including detector simulation, reconstruction, calibration as well as physics analysis. To meet STCF’s specific requirements, OSCAR is designed and developed based on the SNiPER framework, a lightweight common software for HEP experiments. Besides the commonly used software such as Geant4 and ROOT, several state-ofthe-art software packages and tools in the HEP community are incorporated as well, such as the Detector Description Toolkit (DD4hep), the plain-old-data I/O (podio) and Intel Thread Building Blocks (TBB) etc. This paper will present the overall design as well as some implementation details of OSCAR, including the event data management, paralleled data processing based on SNiPER and TBB as well as the geometry management system based on DD4hep. Currently, OSCAR is fully functioning to facilitate the conceptual design of the STCF detector and the study of its physics potential.
Refined drift chamber simulation in the CEPC experiment
Authors: Fang Wenxing , Huang Xingtao , Li Weidong , Lin Tao , Liu Mengyao , Yuan Ye , Zhang Xueyao , Zhang Yao
Published: May 2024

The Circular Electron Positron Collider (CEPC) is a future experiment aimed at studying the properties of the Higgs boson with high precision. This requires excellent track reconstruction and particle identification (PID) performance, which is achieved in the 4th conceptual detector design of the CEPC experiments by combining a silicon tracker and a drift chamber. The drift chamber not only improves track reconstruction but also provides excellent PID with the cluster counting method. To evaluate the performance of this design accurately, a detailed simulation is necessary. In this paper, we present a refined drift chamber simulation by combining Geant4 and Garfield++. However, traditional waveform simulation using Garfield++ is extremely time-consuming, which motivates us to develop a fast waveform simulation method using a neural network. We validate the method using real data from the BESIII experiment. The results demonstrate the effectiveness of our approach and provide valuable insights for future experiments.
A holistic study of the WLCG energy needs for the LHC scientific program
Authors: Britton David , Campana Simone , Panzer-Stradel Bernd
Published: May 2024

The WLCG infrastructure provides the compute power and storage capacity needed by the Large Hadron Collider (LHC) experiments at CERN. The infrastructure is distributed across over 170 data centres in more than 40 countries. The amount of energy consumed by the WLCG to support the scientific program of the LHC experiments, and its evolution, depends on different factors: the luminosity of the LHC and its operating conditions; the data volume and the data complexity; the evolving computing models and the offline software of the experiments; the ongoing R&D program in preparation for the next LHC phase (HL-LHC); the evolution of computing hardware technology towards better energy efficiency; and the modernization of the facilities hosting the data centres to improve Power Usage Effectiveness. This contribution presents a study of the WLCG energy needs and their potential evolution during the future LHC program based on the factors mentioned above. Some of the information is obtained from the CERN experience but then extrapolated to the whole of WLCG. The study provides, therefore, a holistic view for the infrastructure rather than a detailed prediction at the level of the individual facilities. It presents a clear view of the trends and offers a model for more refined studies.
Adoption of a token-based authentication model for the CMS Submission Infrastructure
Authors: Pérez-Calero Yzquierdo Antonio , Mascheroni Marco , Kizinevic Edita , Khan Farrukh Aftab , Kim Hyunwoo , Acosta Flechas Maria , Tsipinakis Nikos , Haleem Saqib , Würthwein Frank
Published: May 2024

The CMS Submission Infrastructure (SI) is the main computing resource provisioning system for CMS workloads. A number of HTCondor pools are employed to manage this infrastructure, which aggregates geographically distributed resources from the WLCG and other providers. Historically, the model of authentication among the diverse components of this infrastructure has relied on the Grid Security Infrastructure (GSI), based on identities and X509 certificates. In contrast, commonly used modern authentication standards are based on capabilities and tokens. The WLCG has identified this trend and aims at a transparent replacement of GSI for all its workload management, data transfer and storage access operations, to be completed during the current LHC Run 3. As part of this effort, and within the context of CMS computing, the Submission Infrastructure group is in the process of phasing out the GSI part of its authentication layers, in favor of IDTokens and Scitokens. The use of tokens is already well integrated into the HTCondor Software Suite, which has allowed us to fully migrate the authentication between internal components of SI. Additionally, recent versions of the HTCondor-CE support tokens as well, enabling CMS resource requests to Grid sites employing this CE technology to be granted by means of token exchange. After a rollout campaign to sites, successfully completed by the third quarter of 2022, the totality of HTCondor CEs in use by CMS are already receiving Scitoken-based pilot jobs. On the ARC CE side, a parallel campaign was launched to foster the adoption of the REST interface at CMS sites (required to enable token-based job submission via HTCondor-G), which is nearing completion as well. In this contribution, the newly adopted authentication model will be described. We will then report on the migration status and final steps towards complete GSI phase out in the CMS SI.
Multicore workflow characterisation methodology for payloads running in the ALICE Grid
Authors: Bertran Ferrer Marta , Grigoras Costin , Badia Rosa M.
Published: May 2024

For LHC Run3 the ALICE experiment software stack has been completely refactored, incorporating support for multicore job execution. Whereas in both LHC Run 1 and 2 the Grid jobs were single-process and made use of a single CPU core, the new multicore jobs spawn multiple processes and threads within the payload. Some of these multicore jobs deploy a high amount of shortlived processes, in the order of more than a dozen per second. The overhead of starting so many processes impacts the overall CPU utilization of the payloads, in particular its System component. Furthermore, the short-lived processes were not correctly accounted for by the monitoring system of the experiment. This paper presents the developed new methodology for supervising the payload execution.

We also present a black box analysis of the new multicore experiment software framework tracing the used resources and system function calls issued by MonteCarlo simulation jobs. Multiple sources of overhead in the lifecycle of processes and threads have thus been identified. This paper describes how the source of each was traced and what solutions were implemented to address them. These improvements have impacted the resource consumption and the overall turnaround time of these payloads with a notable 35% reduction in execution time for a reference production job. We also introduce how this methodology will be used to further improve the efficiency of our experiment software and what other optimization venues are currently under research.
Analysis Grand Challenge benchmarking tests on selected sites
Authors: Koch David , Kuhr Thomas , Duckeck Günter , Hartmann Nikolai
Published: May 2024

A fast turn-around time and ease of use are important factors for systems supporting the analysis of large HEP data samples. We study and compare multiple technical approaches.

This article will be about setting up and benchmarking the Analysis Grand Challenge (AGC) [1] using CMS Open Data. The AGC is an effort to provide a realistic physics analysis with the intent of showcasing the functionality, scalability and feature-completeness of the Scikit-HEP Python ecosystem.

We will present the results of setting up the necessary software environment for the AGC and benchmarking the analysis’ run time on various computing clusters: the institute SLURM cluster at LMU Munich, a SLURM cluster at LRZ (WLCG Tier-2 site) and the analysis facility Vispa [2], operated by RWTH Aachen.

Each site provides slightly different software environments and modes of operation which poses interesting challenges on the flexibility of a setup like that intended for the AGC.

Comparing these benchmarks to each other also provides insights about different storage and caching systems. At LRZ and LMU we have regular Grid storage (HDD) as well as an SSD-based XCache server and on Vispa a sophisticated per-node caching system is used.
AUDITOR: Accounting for opportunistic resources
Authors: Boehler Michael , Gamel Anton J. , Kroboth Stefan , Rottler Benjamin , Sammel Dirk , Schumacher Markus
Published: May 2024

The increasing computational demand in High Energy Physics (HEP) as well as increasing concerns about energy efficiency in highperformance/high-throughput computing are driving forces in the search for more efficient ways to utilise available resources. Since avoiding idle resources is key in achieving high efficiency, an appropriate measure is sharing of idle resources of underutilised sites with fully occupied sites. The software COBalD/TARDIS can automatically, transparently, and dynamically (dis)integrate such resources in an opportunistic manner. Sharing resources however also requires accounting.

In this work we introduce AUDITOR (AccoUnting DatahandlIng Toolbox for Opportunistic Resources), a flexible and extensible accounting system that is able to cover a wide range of use cases and infrastructures. AUDITOR gathers accounting data via so-called collectors which are designed to monitor batch systems, COBalD/TARDIS, cloud schedulers, or other sources of information. The data is stored in a database and provided to so-called plugins, which act based on accounting records. An action could for instance be creating a bill of utilised resources, computing the CO2 footprint, adjusting parameters of a service, or forwarding accounting information to other accounting systems. Depending on the use case, a suitable collector and plugin can be chosen from a growing ecosystem of collectors and plugins. Libraries for interacting with AUDITOR are provided to facilitate the development of collectors and plugins by the community.
CernVM-FS at Extreme Scales
Authors: Promberger Laura , Blomer Jakob , Völkl Valentin , Harvey Matt
Published: May 2024

The CernVM File System (CVMFS) provides the software distribution backbone for High Energy and Nuclear Physics experiments and many other scientific communities in the form of a globally available shared software area. It has been designed for the software distribution problem of experiment software for LHC Runs 1 and 2. For LHC Run 3 and even more so for HL-LHC (Runs 4-6), the complexity of the experiment software stacks and their build pipelines is substantially larger. For instance, software is being distributed for several CPU architectures, often in the form of containers which includes base and operating system libraries, the number of external packages such as machine learning libraries has multiplied, and there is a shift from C++ to more Python-heavy software stacks that results in more and smaller files needing to be distributed. For CVMFS, the new software landscape means an order of magnitude increase of scale in several key metrics. This contribution reports on the performance and reliability engineering on the file system client to sustain current and expected future software access load. Concretely, the impact of the newly designed file system cache management is shown, including significant performance improvements for HEP-representative benchmark workloads, and an up to 25% performance increase in software built-time when the build tools reside on CVMFS. Operational improvements presented include better network failure handling, error reporting, and integration with container runtimes. And a pilot study using zstd as compression algorithm shows that it could bring significant improvements for remote data access times.
Distributed Machine Learning Workflow with PanDA and iDDS in LHC ATLAS
Authors: Guan Wen , Maeno Tadashi , Zhang Rui , Weber Christian , Wenaus Torre , Alekseev Aleksandr , Barreiro Megino Fernando Harald , De Kaushik , Karavakis Edward , Klimentov Alexei , Korchuganova Tatiana , Lin FaHui , Nilsson Paul , Yang Zhaoyu , Zhao Xin
Published: May 2024

Machine Learning (ML) has become one of the important tools for High Energy Physics analysis. As the size of the dataset increases at the Large Hadron Collider (LHC), and at the same time the search spaces become bigger and bigger in order to exploit the physics potentials, more and more computing resources are required for processing these ML tasks. In addition, complex advanced ML workflows are developed in which one task may depend on the results of previous tasks. How to make use of vast distributed CPUs/GPUs in WLCG for these big complex ML tasks has become a popular research area. In this paper, we present our efforts enabling the execution of distributed ML workflows on the Production and Distributed Analysis (PanDA) system and intelligent Data Delivery Service (iDDS). First, we describe how PanDA and iDDS deal with large-scale ML workflows, including the implementation to process workloads on diverse and geographically distributed computing resources. Next, we report real-world use cases, such as HyperParameter Optimization, Monte Carlo Toy confidence limits calculation, and Active Learning. Finally, we conclude with future plans.
Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory
Authors: Karavakis Edward , Guan Wen , Yang Zhaoyu , Maeno Tadashi , Wenaus Torre , Adelman-McCarthy Jennifer , Barreiro Megino Fernando , De Kaushik , Dubois Richard , Gower Michelle , Jenness Tim , Klimentov Alexei , Korchuganova Tatiana , Kowalik Mikolaj , Lin FaHui , Nilsson Paul , Padolski Sergey , Yang Wei , Ye Shuwei
Published: May 2024

The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment’s run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory’s needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.
The ALICE Grid Workflow for LHC Run 3
Authors: Storetvedt Maxim , Betev Latchezar , Helstrup Håvard , Hetland Kristin Fanebust , Kileng Bjarte
Published: May 2024

In preparation for LHC Run 3 and 4 the ALICE Collaboration has moved to a new Grid middleware, JAliEn, and workflow management system. The migration was dictated by the substantially higher requirements on the Grid infrastructure in terms of payload complexity, increased number of jobs and managed data volume, all of which required a complete rewrite of the middleware using modern software languages and technologies. Through containerisation, self-contained binaries, managed by the JAliEn middleware, we provide a uniform execution environment across sites and various architectures, including accelerators. The model and implementation have proven their scalability and can be easily deployed across sites with minimal intervention.

This contribution outlines the architecture of the new Grid workflow as deployed in production and the workflow process. Specifically shown is how core components are moved and bootstrapped through CVMFS, enabling the middleware to run anywhere fully independent of the host system. Furthermore, we examine how new middleware releases, containers and their runtimes are centrally maintained and easily deployed across the Grid, also by the means of a common build system.
Transitioning GlideinWMS, a multi domain distributed workload manager, from GSI proxies to tokens and other granular credentials
Authors: Mambelli Marco , Coimbra Bruno , Box Dennis
Published: May 2024

GlideinWMS is a distributed workload manager that has been used in production for many years to provision resources for experiments like CERN’s CMS, many Neutrino experiments, and the OSG. Its security model was based mainly on GSI (Grid Security Infrastructure), using X.509 certificate proxies and VOMS (Virtual Organization Membership Service) extensions. Even when other credentials, like SSH keys, were used to authenticate with resources, proxies were also added all the time, to establish the identity of the requestor and the associated memberships or privileges. This single credential was used for everything and was, often implicitly, forwarded wherever needed. The addition of identity and access tokens and the phase-out of GSI forced us to reconsider the security model of GlideinWMS, to handle multiple credentials which can differ in type, technology, and functionality. Both identity tokens and access tokens are supported. GSI proxies even if no more mandatory, are still used, together with various JWT (JSON Web Token) based tokens and other certificates. The functionality of the credentials, defined by issuer, audience, and scope, also differ: a credential can allow access to a computing resource, or can protect the GlideinWMS framework from tampering, or can grant read or write access to storage, can provide an identity for accounting or auditing, or can provide a combination of any the formers. Furthermore, the tools in use do not include automatic forwarding and renewal of the new credentials so credential lifetime and renewal requirements became part of the discussion as well. In this paper, we will present how GlideinWMS was able to change its design and code to respond to all these changes.
The migration to a standardized architecture for developing systems on the Glance project
Authors: Ferreira Brito Filho Carlos Henrique , Souza e Silva Gabriel Jose , Corti Gloria , Closier Joel
Published: May 2024

The Glance project is responsible for over 20 systems across three CERN experiments: ALICE [1], ATLAS [2] and LHCb [3]. Students, engineers, physicists and technicians have been using systems designed and managed by Glance on a daily basis for 20 years. In order to produce quality products continuously, considering internal stakeholder’s ever-evolving requests, there is a need for standardization. The adoption of such a standard had to take into account not only future developments but also legacy systems of the three experiments. These systems were created using an in-house built framework, and, as they scaled, became difficult to maintain due to the framework’s lack of documentation and use of technologies that were becoming obsolete. Migrating them to a new architecture would mean speeding up the development process, avoiding rework and integrating CERN systems widely. Since a lot of the core functionalities of the systems are shared between them, both on the frontend and on the backend, the architecture had to assure modularity and reusability. In this architecture, the principles behind Hexagonal Architecture are followed and the systems’ codebase is split into two applications: a JavaScript client and a REST backend server. The open-source framework Vue.js was chosen for the frontend. Its versatility, approachability and extended documentation made it the ideal tool for creating components that are reused throughout Glance applications. The backend uses PHP libraries created by the team to expose information through REST APIs both internally, allowing easier integration between the systems, and externally, introducing to users outside Glance information managed by the team.
The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3
Authors: Eulisse Giulio , Rohr David
Published: May 2024

ALICE has upgraded many of its detectors for LHC Run 3 to operate in continuous readout mode recording Pb–Pb collisions at 50 kHz interaction rate without trigger. This results in the need to process data in real time at rates 100 times higher than during Run 2. In order to tackle such a challenge we introduced O2, a new computing system and the associated infrastructure. Designed and implemented during the LHC long shutdown 2, O2 is now in production taking care of all the data processing needs of the experiment. O2 is designed around the message passing paradigm, enabling resilient, parallel data processing for both the synchronous (to LHC beam) and asynchronous data taking and processing phases. The main purpose of the synchronous online reconstruction is detector calibration and raw data compression. This synchronous processing is dominated by the TPC detector, which produces by far the largest data volume, and TPC reconstruction runs fully on GPUs. When there is no beam in the LHC, the powerful GPU-equipped online computing farm of ALICE is used for the asynchronous reconstruction, which creates the final reconstructed output for analysis from the compressed raw data. Since the majority of the compute performance of the online farm is in the GPUs, and since the asynchronous processing is not dominated by the TPC in the way the synchronous processing is, there is an ongoing effort to offload a significant amount of compute load from other detectors to the GPU as well.
Physics analysis for the HL-LHC: Concepts and pipelines in practice with the Analysis Grand Challenge
Authors: Held Alexander , Kauffman Elliott , Shadura Oksana , Wightman Andrew
Published: May 2024

Realistic environments for prototyping, studying and improving analysis workflows are a crucial element on the way towards user-friendly physics analysis at HL-LHC scale. The IRIS-HEP Analysis Grand Challenge (AGC) provides such an environment. It defines a scalable and modular analysis task that captures relevant workflow aspects, ranging from large-scale data processing and handling of systematic uncertainties to statistical inference and analysis preservation. By being based on publicly available Open Data, the AGC provides a point of contact for the broader community. Multiple different implementations of the analysis task that make use of various pipelines and software stacks already exist.

This contribution presents an updated AGC analysis task. It features a machine learning component and expanded analysis complexity, including the handling of an extended and more realistic set of systematic uncertainties. These changes both align the AGC further with analysis needs at the HL-LHC and allow for probing an increased set of functionality.

Another focus is the showcase of a reference AGC implementation, which is heavily based on the HEP Python ecosystem and uses modern analysis facilities. The integration of various data delivery strategies is described, resulting in multiple analysis pipelines that are compared to each other.
Apptainer Without Setuid
Authors: Dykstra Dave
Published: May 2024

Apptainer (formerly known as Singularity) since its beginning implemented many of its container features with the assistance of a setuidroot program. It still supports that mode, but as of version 1.1.0 it no longer uses setuid by default. This is feasible because it now can mount squashfs filesystems, ext3 filesystems, and overlay filesystems using unprivileged user namespaces and FUSE. It also now enables unprivileged users to build containers, even without requiring system administrators to configure /etc/subuid and /etc/subgid unlike other “rootless” container systems. As a result, all the unprivileged functions can be used nested inside of another container, even if the container runtime prevents any elevated privileges. As of version 1.2.0 Apptainer also supports completely unprivileged encryption of Singularity Image Format (SIF) container files. Performance with a particularly challenging HEP benchmark using the FUSE-based mounts both with and without encryption is essentially identical to the previous methods that required elevated privileges to use the Linux kernel-based counterparts.
Czech national e-infrastructure services for HEP
Authors: Chudoba Jiri , Kouril Daniel , Krenkova Ivana , Ruda Miroslav , Sitera Jiri , Stibor Tomas , Sustr Zdenek
Published: May 2024

Czech e-infrastructure project e-INFRA CZ is the only project for ICT services defined on the Road Map of the Czech Republic for Large Infrastructures for Research, Experimental Development and Innovations. It was created by 3 national e-infrastructures: CESNET, CERIT-SC and IT4Innovations. The project provides ICT services mostly for Czech universities and research institutions. High energy physics projects belong to the most demanding users in terms of network, computing and storage capacities. We will describe usage of these three main categories for several HEP and astroparticle physics projects in which Czech institutions are involved. But not only these most visible services are provided. Other services like security CSIRT team, backup solutions, virtualization platform, identity management, Jupyter notebooks, electronic logbook and user support system are extensively used. Many concrete examples will be given on e-INFRA services usage by the Czech WLCG Tier-2 center. We will also present the important role of the e-INFRA project as a national coordinator of international activities in the grid computing (EGI), open science solutions (EOSC) or high-performance computing (EuroHPC).
Finalizing Transition to the New Data Center at BNL
Authors: Latif Imran , Misawa Shigeki , Zaytsev Alexandr
Published: May 2024

Computational science, data management and analysis have been key factors in the success of Brookhaven National Laboratory's scientific programs at the Relativistic Heavy Ion Collider (RHIC), the National Synchrotron Light Source (NSLS-II), the Center for Functional Nanomaterials (CFN), and in biological, atmospheric, and energy systems science, Lattice Quantum Chromodynamics (LQCD) and Materials Science, as well as our participation in international research collaborations, such as the ATLAS Experiment at Europe's Large Hadron Collider (LHC) at CERN (Switzerland) and the Belle II Experiment at KEK (Japan). The construction of a new data center is an acknowledgement of the increasing demand for computing and storage services at BNL in the near term and enable the Lab to address the needs of the future experiments at the High-Luminosity LHC at CERN and the Electron-Ion Collider (EIC) at BNL in the long term. The Computing Facility Revitalization (CFR) project is aimed at repurposing the former National Synchrotron Light Source (NSLS-I) building as the new data center for BNL. The construction of the new data center was finished in 2021Q3, and it was delivered for production in early FY2022 for all collaborations supported by the Scientific Data and Computing Center (SDCC), including STAR, PHENIX and sPHENIX experiments at RHIC collider at BNL, the Belle II Experiment at KEK (Japan), and the Computational Science Initiative at BNL (CSI). This paper highlights the key mechanical, electrical, and networking components of the new data center in its final configuration as used in production since 2021Q4 and gives an overview for the extension of the central network systems into the new data center and the migration of a significant portion of IT load and services from the old data center to the new data center carried out in 20212023, with expected completion of the main phase of the gradual IT equipment replacement and migration from the old data center into the new one set to the end of FY2023 (Sep 30, 2023).
The Spanish CMS Analysis Facility at CIEMAT
Authors: Cárdenas-Montes M. , Delgado Peris A. , Flix J. , Hernández J.M. , León Holgado J. , Morcillo Pérez C. , Pérez-Calero Yzquierdo A. , Rodríguez Calonge F.J.
Published: May 2024

The increasingly larger data volumes that the LHC experiments will accumulate in the coming years, especially in the High-Luminosity LHC era, call for a paradigm shift in the way experimental datasets are accessed and analyzed. The current model, based on data reduction on the Grid infrastructure, followed by interactive data analysis of manageable size samples on the physicists’ individual computers, will be superseded by the adoption of Analysis Facilities. This rapidly evolving concept is converging to include dedicated hardware infrastructures and computing services optimized for the effective analysis of large HEP data samples. This paper describes the actual implementation of this new analysis facility model at the CIEMAT institute, in Spain, to support the local CMS experiment community. Our work details the deployment of dedicated highly performant hardware, the operation of data staging and caching services ensuring prompt and efficient access to CMS physics analysis datasets, and the integration and optimization of a custom analysis framework based on ROOT’s RDataFrame and CMS NanoAOD format. Finally, performance results obtained by benchmarking the deployed infrastructure and software against a CMS analysis workflow are summarized.
A return to in-person public engagement at STFC
Authors: Corbett Greg , Dack Tom , Collier Ian , Day Phill , Palmer Sophy
Published: May 2024

Since 2020, UKRI/STFC’s Scientific Computing Department (SCD) have developed several remote-first public engagement activities, drawing on its long and rich history of delivering face to face public engagement and outreach, as part of the wider STFC programme. With COVID19 restrictions lifted in the UK, STFC has been able to resume in-person public engagement, both on site and in public places. However, this has not meant a complete return to exclusively in-person engagement, but rather, recognising the clear benefits of remote engagement to meeting our strategic public engagement aims, STFC has produced a blended programme for 2022/23, with a mixture of in-person, remote and hybrid events. This paper presents how the remote activities have evolved since their initial creation, how the remote activities have become part of a blended programme and how the in-person activities in place since before the pandemic have been improved as a result of developing the remote activities.
Machine Learning for Columnar High Energy Physics Analysis
Authors: Kauffman Elliott , Held Alexander , Shadura Oksana
Published: May 2024

Machine learning (ML) has become an integral component of high energy physics data analyses and is likely to continue to grow in prevalence. Physicists are incorporating ML into many aspects of analysis, from using boosted decision trees to classify particle jets to using unsupervised learning to search for physics beyond the Standard Model. Since ML methods have become so widespread in analysis and these analyses need to be scaled up for HL-LHC data, neatly integrating ML training and inference into scalable analysis workflows will improve the user experience of analysis in the HL-LHC era.

We present the integration of ML training and inference into the IRISHEP Analysis Grand Challenge pipeline to provide an example of how this integration can look like in a realistic analysis environment. We also utilize Open Data to ensure the project’s reach to the broader community. Different approaches for performing ML inference at analysis facilities are investigated and compared, including performing inference through external servers. Since ML techniques are applied for many different types of tasks in physics analyses, we showcase options for ML integration that can be applied to various inference needs.
ErUM-Data-Hub - The Networking and Transfer Office serving Digital Transformation in Research on Universe and Matter in Germany
Authors: Erdmann Martin , Fackeldey Peter , Fischer Benjamin , Fröse Stefan , Nowack Andreas , Steinfeld Judith , Warkentin Angela
Published: May 2024

Research on Universe and Matter (ErUM), conducted at major infrastructures such as CERN and large observatories in collaboration with university groups, plays an important role in driving the digital transformation for the future. The German action plan "ErUM-Data" [2] promotes this transformation through the interdisciplinary networking and financial support of about 20.000 scientists. The ErUM-Data-Hub [4] serves as a central networking and transfer office to meet these ambitions. One central task is designing, organizing and performing schools and workshops for young and expert scientists in the areas of big data, deep learning, sustainable computing and many more. We present the achievements up to the first anniversary of the ErUM-Data-Hub in the German ErUM community.
An Object Condensation Pipeline for Charged Particle Tracking at the High Luminosity LHC
Authors: Lieret Kilian , DeZoort Gage
Published: May 2024

Recent work has demonstrated that graph neural networks (GNNs) trained for charged particle tracking can match the performance of traditional algorithms while improving scalability to prepare for the High Luminosity LHC experiment. Most approaches are based on the edge classification (EC) paradigm, wherein tracker hits are connected by edges, and a GNN is trained to prune edges, resulting in a collection of connected components representing tracks. These connected components are usually collected by a clustering algorithm and the resulting hit clusters are passed to downstream modules that may assess track quality or fit track parameters. In this work, we consider an alternative approach based on object condensation (OC), a multi-objective learning framework designed to cluster points belonging to an arbitrary number of objects, in this context tracks, and regress the properties of each object. We demonstrate that OC shows very promising results when applied to the pixel detector of the trackML dataset and can, in some cases, recover tracks that are not reconstructable when relying on the output of an EC alone. The results have been obtained with a modular and extensible open-source implementation that allows us to efficiently train and evaluate the performance of various OC architectures and related approaches.
Deep Learning Approaches for LHCb ECAL Reconstruction
Authors: Boldyrev Alexey , Derkach Denis , Ratnikov Fedor , Shevelev Andrey
Published: May 2024

Calorimeters are a crucial component for most detectors mounted on modern colliders. Their tasks include identifying and measuring the energy of photons and neutral hadrons, recording energetic hadronic jets, and contributing to the identification of electrons, muons, and charged hadrons. To fulfill these many tasks while keeping costs reasonable, the calorimeter construction requires good and thoughtful balancing with other components of the detector. Much harder operation conditions during LHC’s high luminosity Run 5 and beyond bring new technological and computational challenges. This requires optimization of technologies, layouts, readouts, reconstruction algorithms to achieve the best overall physics performance for the limited cost. In the traditional approach, the reconstruction of the physical objects in the calorimeter must be matched to the calorimetric showers simulation used. We present a deep learning-based approach to help utilize raw simulated calorimetric data of varying degrees of detail.
Embedded Continual Learning for High-Energy Physics
Authors: Barbone Marco , Brown Christopher , Gaydadjiev Georgi , Maguire Thomas , Mieskolainen Mikael , Radburn-Smith Benjamin , Luk Wayne , Tapper Alexander
Published: May 2024

Neural Networks (NN) are often trained offline on large datasets and deployed on specialised hardware for inference, with a strict separation between training and inference. However, in many realistic applications the training environment differs from the real world, or data arrives in a streaming fashion and is continuously changing. In these scenarios, the ability to continuously train and update NN models is desirable. Continual learning (CL) algorithms allow training of models on a stream of data. CL algorithms are often designed to work in constrained settings, such as limited memory and computational power, or limitations on the ability to store past data (e.g, due to privacy concerns or memory requirements). High-energy physics experiments are developing intelligent detectors, with algorithms running on computer systems located close to the detector to meet the challenges of increased data rates and occupancies. The use of NN algorithms in this context is limited by changing detector conditions, such as degradation over time or failure of an input signal which might cause the NNs to lose accuracy leading, in the worst case to the loss of interesting events. CL has the potential to solve this issue, using large amounts of continuously streaming data to allow the network to recognise changes, and to learn and adapt to detector conditions. It has the potential to outperform traditional NN training techniques as not all possible scenarios can be predicted and modelled in static training data samples. However, NN training is computationally expensive and when combined with the strict timing requirements of embedded processors deployed close to the detector, current state-of-the-art offline approaches cannot be directly applied to the real-time systems. Alternatives to typical backpropagation-based training that can be deployed on FPGAs for real-time data processing are presented, and their computational and accuracy characteristics are discussed in the context of High-Luminosity LHC.
FlashSim: Accelerating HEP simulation with an end-to-end Machine Learning framework
Authors: Vaselli Francesco , Rizzi Andrea , Cattafesta Filippo , Cicconofri Gloria
Published: May 2024

We developed a first prototype of an end-to-end machine learning based simulation framework for arbitrary analysis ntuples at the CMS experiment. Such a framework, called FlashSim, was capable of simulating a wide variety of physical objects with good performance on 1d distributions, correlations and desired physical content when compared to the current state-of-theart simulation. Current methods are based on MC techniques, computationally expensive and requiring a long time to compute. Our prototype was trained to replicate the samples from state-of-the-art methods through the use of the Normalizing Flows algorithm. It showed compatible results with a speedup of several orders of magnitude. This type of approach opens the way to general, analysis agnostic simulation frameworks which may be able to tackle the challenges of the simulation needs for HL-LHC and future collaborations.
Baler - Machine Learning Based Compression of Scientific Data
Authors: Bengtsson Folkesson Fritjof , Doglioni Caterina , Ekman Per Alexander , Gallén Axel , Jawahar Pratik , Camps Santasmasas Marta , Skidmore Nicola
Published: May 2024

A common and growing issue in scientific research and industry is that of storing and sharing ever-increasing datasets. In this paper we document the development and applications of Baler - a Machine Learning based tool for tailored compression of data across multiple disciplines.
Novel fully-heterogeneous GNN designs for track reconstruction at the HL-LHC
Authors: Caillou Sylvain , Collard Christophe , Rougier Charline , Stark Jan , Torres Heberth , Vallier Alexis
Published: May 2024

Data from the LHC detectors are not easily represented using regular data structures. These detectors are comprised of several species of subdetectors and therefore produce heterogeneous data. LHC detectors are highly granular by design so that nearby particles may be distinguished. As a consequence, LHC data are sparse, in that many detector channels are not active during a given collision event. Graphs offer a flexible and efficient alternative to rectilinear data structures for representing LHC data. Accordingly, graph-based machine learning algorithms are becoming increasingly popular for a large number of LHC physics tasks [1, 2]. This popularity, and the corresponding potential for substantial increase in physics output, are illustrated on the cover of a recent issue [3] of the CERN Courier magazine. The graphs used in almost all practical applications at the LHC so far are homogeneous, i.e. each node is assigned the same features, and each edge is assigned the same features. In other words, the power of graphs to represent sparse data has been exploited in applications for the LHC, but the potential of graphs to represent heterogeneous data has not. The pink graph on the cover of the CERN Courier [3] can be seen as an illustration of this limitation: all nodes are pink, regardless of their position in the detector. We present novel fully-heterogeneous GNN designs and apply them to simulated data from a tracking detector that resembles the trackers that will be used at the HL-LHC. It contains a pixel subsystem that provides 3D hits and a strip subsystem that provides 2D hits. We present a new design which solves the degraded performance observed in the strip detector in the first GNNbased tracking studies presented by the ATLAS Collaboration [4].
Refining fast simulation using machine learning
Authors: Bein Samuel , Connor Patrick , Pedro Kevin , Schleper Peter , Wolf Moritz
Published: May 2024

At the CMS experiment, a growing reliance on the fast Monte Carlo application (FastSim) will accompany the high luminosity and detector granularity expected in Phase 2. The FastSim chain is roughly 10 times faster than the application based on the Geant4 detector simulation and full reconstruction referred to as FullSim. However, this advantage comes at the price of decreased accuracy in some of the final analysis observables. In this contribution, a machine learning-based technique to refine those observables is presented. We employ a regression neural network trained with a sophisticated combination of multiple loss functions to provide post-hoc corrections to samples produced by the FastSim chain. The results show considerably improved agreement with the FullSim output and an improvement in correlations among output observables and external parameters. This technique is a promising replacement for existing correction factors, providing higher accuracy and thus contributing to the wider usage of FastSim.
Outlines in hardware and software for new generations of exascale interconnects
Authors: Ammendola Roberto , Biagioni Andrea , Chiarini Carlotta , Cretaro Paolo , Frezza Ottorino , Lo Cicero Francesca , Lonardo Alessandro , Martinelli Michele , Pastorelli Elena , Paolucci Pier Stanislao , Pontisso Luca , Rossi Cristian , Simula Francesco , Vicini Piero
Published: May 2024

RED-SEA (https://redsea-project.eu/) is a European project funded in the framework of the H2020-JTI-EuroHPC-2019-1 call that started in April 2021. The goal of the project is to evaluate the architectural design of the main elements of the interconnection networks for the next generation of HPC systems supporting hundreds of thousands of computing nodes enabling the Exascale for HPC, HPDA and AI applications while providing preliminary prototypes.

The main technological feature is the BXI network, originally designed and produced by ATOS (France). The plan is to integrate in the next release of the network – BXI3 – the architectural solutions and novel IPs developed within the framework of the RED-SEA project.

The consortium is composed of 11 well-established research teams across Europe, with extensive experience in interconnects, including network design, deployment and evaluation.

Within RED-SEA, INFN is adopting a hardware/software co-design approach to design APEnetX, a scalable interconnect prototyped on latest generation Xilinx FPGAs, adding innovative components for the improvement of the performance and resiliency of the interconnect. APEnetX is an FPGA-based, PCIe Gen3/4 network interface card equipped with RDMA capabilities being the endpoint of a direct multidimensional toroidal network and suitable for integration in the BXI environment. APEnetX design will be benchmarked on project testbeds using real scientific applications like NEST, a spiking neural network simulator.