bluuebunny/crossref_metadata_2025 huggingface dataset:
Optimizing access to conditions data in ATLAS event data processing
Authors: Rinaldi Lorenzo , Gallas Elizabeth J , Formica Andrea
Published: September 2019

The processing of ATLAS event data requires access to conditions
data which are stored in database systems. This data includes, for example
alignment, calibration, and configuration information which may be characterized
by large volumes, diverse content, and/or information which evolves over
time as refinements are made in those conditions. Additional layers of complexity
are added by the need to provide this information across the worldwide
ATLAS computing grid and the sheer number of simultaneously executing processes
on the grid, each demanding a unique set of conditions to proceed. Distributing
this data to all the processes that require it in an efficient manner has
proven to be an increasing challenge with the growing needs and numbers of
event-wise tasks. In this presentation, we briefly describe the systems in which
we have collected information about the database content and the use of conditions
in event data processing. We then proceed to explain how this information
has been used not only to refine reconstruction software and job configuration
but also to guide modifications of underlying conditions data configuration and
in some cases, rewrites of the data in the database into a more harmonious form
for offline usage in the processing of both real and simulated data..
Conditions evolution of an experiment in mid-life, without the crisis (in ATLAS)
Authors: Rinaldi Lorenzo , Formica Andrea , Gallas Elizabeth J , Ozturk Nurcan , Roe Shaun
Published: September 2019

The ATLAS experiment is approaching mid-life: the long shutdown
period (LS2) between LHC Runs 1 and 2 (ending in 2018) and the future collision
data-taking of Runs 3 and 4 (starting in 2021). In advance of LS2, we have
been assessing the future viability of existing computing infrastructure systems.
This will permit changes to be implemented in time for Run 3. In systems with
broad impact such as the conditions database, making assessments now is critical
as the full chain of operations from online data-taking to offline processing
can be considered: evaluating capacity at peak times, looking for bottlenecks,
identifying areas of high maintenance, and considering where new technology
may serve to do more with less.

We have been considering changes to the ATLAS Conditions Database related
storage and distribution infrastructure based on similar systems of other experiments.
We have also examined how new technologies may help and how we
might provide more RESTful services to clients. In this presentation, we give
an overview of the identified constraints and considerations, and our conclusions
for the best way forward: balancing preservation of critical elements of
the existing system with the deployment of the new technology in areas where
the existing system falls short.
Distributed caching system for multi-site DPM storage
Authors: Doria Alessandra , Carlino Gianpaolo , De Salvo Alessandro , Spisso Bernardino , Vilucchi Elisabetta
Published: September 2019

The main goal of this work, in the context of WLCG, is to test a storage setup where the storage areas are geographically distributed and the system provides some pools behaving as data caches. Users can find data needed for their analysis in a local cache and process them locally. We first demonstrate that the distributed setup for a DPM storage is almost transparent to the users, in terms of performance and functionalities. Then, we implement a mechanism to fill the storage cache with data registered in Rucio Data Management system and we test it, running a physics analysis that gets its input data from the cache. Thus we demonstrate that the use of such a system can be useful for diskless sites with a local cache only, allowing to optimize the distribution and analysis of experimental data.
Evolution of the Hadoop Platform and Ecosystem for High Energy Physics
Authors: Baranowski Zbigniew , Kleszcz Emil , Kothuri Prasanth , Canali Luca , Castellotti Riccardo , Martin Marquez Manuel , Matos de Barros Nuno Guilherme , Motesnitsalis Evangelos , Mrowczynski Piotr , Luna Duran Jose Carlos
Published: September 2019

The interest in using scalable data processing solutions based on
Apache Hadoop ecosystem is constantly growing in the High Energy Physics
(HEP) community. This drives the need for increased reliability and availability
of the central Hadoop service and underlying infrastructure provided to the
community by the CERN IT department. This paper reports on the overall status
of the Hadoop platform and related Hadoop and Spark service at CERN,
detailing recent enhancements and features introduced in many areas including
the service configuration, availability, alerting, monitoring and data protection,
in order to meet the new requirements posed by the users’ community.
Parallel Event Selection on HPC Systems
Authors: Paterno Marc , Kowalkowski Jim , Sehrish Saba
Published: September 2019

In their recent measurement of the neutrino oscillation parameters,
NOvA uses a sample of approximately 25 million reconstructed spills to search
for electron-neutrino appearance events. These events are stored in an n-tuple
format, in 250 thousand ROOT files. File sizes range from a few hundred KiB to
a few MiB; the full dataset is approximately 1.4 TiB. These millions of events
are reduced to a few tens of events by the application of strict event selection
criteria, and then summarized by a handful of numbers each, which are used in
the extraction of the neutrino oscillation parameters.

The NOvA event selection code is currently a serial C++ program that reads
these n-tuples. The current table data format and organization and the selection/
reduction processing involved provides us with an opportunity to explore
alternate approaches to represent the data and implement the processing. We
represent our n-tuple data in HDF5 format that is optimized for the HPC environment
and which allows us to use the machine’s high-performance parallel
I/O capabilities. We use MPI, numpy and h5py to implement our approach and
compare the performance with the existing approach. We study the performance
implications of using thousands of small files of different sizes as compared
with one large file using HPC resources. This work has been done as part of the
SciDAC project, “HEP analytics on HPC” in collaboration with the ASCR teams
at ANL and LBNL.
Continuous Integration service at Fermilab
Authors: Di Benedetto Vito , Podstavkov Vladimir , Fattoruso Michele , Coimbra Bruno
Published: September 2019

This paper describes the current architecture of Continuous Integration (CI) service developed at Fermilab, encountered successes and difficulties, as well as future development plans. The current experiment code has hundreds of contributors that provide new features, bug fixes, and other improvements. Version control systems help developers to collaborate in contributing software for their experiments, while the CI system helps developers keep their code healthy. The Fermilab CI service allows experiments and projects to test and validate their offline production and analysis code on the supported platforms. It is built on top of Jenkins, designed to be set up from a configuration file that provides implementation for each phase of the CI workflow, and able to validate experiment code through grid jobs. This CI service provides a dashboard for easy access to logs and statistical graphs. Since the CI service has been adopted by Fermilab experiments/projects, it proved to be very useful to intercept issues in their code early on and get them fixed before running it in production. Currently the CI service is in use by the ArgoNeuT, DUNE, g-2, LArIAT, MINERvA, mu2e, NOvA, SBND and uBooNE experiments and by the following projects: ART and LArSoft software suites, GENIE, and Glidein-WMS. The CI service is under active development and planning to support code profiling.
I/O in the ATLAS multithreaded framework
Authors: Cranshaw Jack , Malon David , Nowak Marcin , Van Gemmeren Peter
Published: September 2019

Scalable multithreading poses challenges to I/O for the ATLAS experiment. The performance of a thread-safe I/O strategy may depend upon many factors, including I/O latencies, whether tasks are CPU- or I/O-intensive, and thread count. In a multithreaded framework, an I/O infrastructure must efficiently supply event data to and collect it from many threads processing multiple events in flight. In particular, on-demand reading from multiple threads may challenge caching strategies that were developed for serial processing and may need to be enhanced. This I/O infrastructure must also address how to read, make available, and propagate in-file metadata and other non-event data needed as context for event processing. We describe the design and scheduling of I/O components in the ATLAS multithreaded control framework, AthenaMT, for both event and non-event I/O. We discuss issues associated with exploiting the multithreading capabilities of our underlying persistence technology, ROOT, in a manner harmonious with the ATLAS framework?s own approach to thread management. Finally, we discuss opportunities for evolution and simplification of I/O components that have successfully supported ATLAS event processing for many years from their serial incarnations to their thread-safe counterparts.
GNA: new framework for statistical data analysis
Authors: Fatkina Anna , Gonchar Maxim , Kalitkina Anastasia , Kolupaeva Liudmila , Naumov Dmitry , Selivanov Dmitry , Treskov Konstantin
Published: September 2019

We report on the status of GNA — a new framework for fitting large-scale physical models. GNA utilizes the data flow concept within which a model is represented by a directed acyclic graph. Each node is an operation on an array (matrix multiplication, derivative or cross section calculation, etc). The framework enables the user to create flexible and efficient large-scale lazily evaluated models, handle large numbers of parameters, propagate parameters’ uncertainties while taking into account possible correlations between them, fit models, and perform statistical analysis.

The main goal of the paper is to give an overview of the main concepts and methods as well as reasons behind their design. Detailed technical information is to be published in further works.
The BmnRoot framework for experimental data processing in the BM@N experiment at NICA
Authors: Batyuk Pavel , Gertsenberger Konstantin , Merts Sergey , Rogachevsky Oleg
Published: September 2019

A new generation of experiments for the relativistic nuclear physics is expected to be started up in the nearest years at the Nuclotron-based Ion Collider fAcility (NICA) under construction at the Joint Institute for Nuclear Research in Dubna. The main part of the facility is the essentially modernized accelerator Nuclotron. BM@N (Baryonic Matter at Nuclotron) is considered as a first stage towards realization of physics program available at NICA. It is a fixed target experiment aimed to work with the Nuclotron extracted beams of different species. The experiment had a set of technical runs since 2015. For a successful realization of the BM@N physics program, a well developed and tested software for simulation, digitization, reconstruction and analysis of collision events and other additional tasks is of utmost importance. The BmnRoot software developed in order to operate the mentioned tasks is described in this article. It includes modules for data digitizing obtained from BM@N detector systems, realistic simulation of signals in detectors, alignment of detectors, reconstruction of multiparticle interaction events, as well as all necessary systems for maintaining the databases of the experiment, visualization and providing information support for the experiment.
Shared Memory Transport for ALFA
Authors: Rybalchenko Alexey , Klein Dennis , Al-Turany Mohammad , Kollegger Thorsten
Published: September 2019

The high data rates expected for the next generation of particle physics experiments (e.g.: new experiments at FAIR/GSI and the upgrade of CERN experiments) call for dedicated attention with respect to design of the needed computing infrastructure. The common ALICE-FAIR framework ALFA is a modern software layer, that serves as a platform for simulation, reconstruction and analysis of particle physics experiments. Beside standard services needed for simulation and reconstruction of particle physics experiments, ALFA also provides tools for data transport, configuration and deployment. The FairMQ module in ALFA offers building blocks for creating distributed software components (processes) that communicate between each other via message passing.

The abstract "message passing" interface in FairMQ has at the moment three implementations: ZeroMQ, nanomsg and shared memory. The newly developed shared memory transport will be presented, that provides significant per-formance benefits for transferring large data chunks between components on the same node. The implementation in FairMQ allows users to switch between the different transports via a trivial configuration change. The design decisions, im-plementation details and performance numbers of the shared memory transport in FairMQ/ALFA will be highlighted.
A Historic Data Quality Monitor (HDQM) tool for the CMS Tracker Detector
Authors: Kyriakis Aristotelis
Published: September 2019

Monitoring the time evolution of data related observables is important for the successful operation of the LHC experiments. It permits keeping control on data quality during LHC running and also effectively checking the influence on data of any detector calibration performed during the year. The Historic Data Quality Monitor (HDQM) of the CMS experiment is a framework developed by the Tracker group of the CMS collaboration that permits a web-based monitoring of the time evolution of interesting quantities (i.e. signal to noise ratio, cluster size) in the Tracker Silicon micro-strip and pixel.
A Parallelised ROOT for Future HEP Data Processing
Authors: Piparo Danilo , Canal Philippe , Amadio Guilherme , Guiraud Enrico , Naumann Axel , Valls Xavier , Tejedor Enric
Published: September 2019

In the coming years, HEP data processing will need to exploit parallelism on present and future hardware resources to sustain the bandwidth requirements. As one of the cornerstones of the HEP software ecosystem, ROOT embraced an ambitious parallelisation plan which delivered compelling results. In this contribution the strategy is characterised as well as its evolution in the medium term. The units of the ROOT framework are discussed where task and data parallelism have been introduced, with runtime and scaling measurements. We will give an overview of concurrent operations in ROOT, for instance in the areas of I/O (reading and writing of data), fitting / minimization, and data analysis. This paper introduces the programming model and use cases for explicit and implicit parallelism, where the former is explicit in user code and the latter is implicitly managed by ROOT internally.
DQM4hep
Authors: Ete Rémi , Pingault Antoine
Published: September 2019

Data quality monitoring is the first step in the certification of data recorded for offline physics analyses. Many experiments have developed their own dedicated monitoring system in the past. Most of them rely on their own event data model, which leads to a strong dependency on the data format and storage. We present here a generic data quality monitoring system, DQM4hep, that has been developed without any assumptions on the underlying event data model. This reduces the code maintenance and increases the portability and reusability across other experiments. We first introduce the framework architecture and the various core components as well as tools provided by the software package. We then give an overview of the different experiments using DQM4hep and the foreseen integration in future other experiments. We finally present the ongoing and future software development for DQM4Hhep and long-term prospects.
Performance optimization of the air shower simulation program for the Cherenkov Telescope Array
Authors: Arrabito Luisa , Bernlöhr Konrad , Bregeon Johan , Maier Gernot , Langlois Philippe , Parello David , Revy Guillaume
Published: September 2019

The Cherenkov Telescope Array (CTA), currently under construction, is the next-generation instrument in the field of very high energy gamma-ray astronomy. The first data are expected by the end of 2018, while the scientific operations will start in 2022 for a duration of about 30 years. In order to characterize the instrument response to the Cherenkov light emitted when cosmic ray showers develop in the atmosphere, detailed Monte Carlo simulations will be regularly performed in parallel to CTA operation. The estimated CPU time associated to these simulations is very high, of the order of 200 millions HS06 hours per year. Reducing the CPU time devoted to simulations would allow either to reduce infrastructure cost or to better cover the large phase space. In this paper, we focus on the main computing step (70% of the whole CPU time) implemented in the CORSIKA program, and specifically on the mod-ule responsible for the propagation of Cherenkov photons in the atmosphere. We present our preliminary studies about different options of code optimization, with a particular focus on vectorization facilities (SIMD instructions). Our proposals take care, as automatically as possible, of the hardware portability constraints introduced by the grid computing environment that hosts these simulations. Performance evaluation in terms of running-time and accuracy is provided.
Monitoring tools for the CMS muon detector: present workflows and future automation
Authors: Calabria Cesare
Published: September 2019

The CMS Muon System has been operated successfully during the two LHC runs allowing to collect a very high fraction of data with a quality that fulfils the requirements to be used for physics analysis. Nevertheless, the workflows used nowadays to operate and monitor the detector are rather expensive in terms of human resources. Focus is therefore being put on improving such workflows, both by applying automated statistical tests and exploiting modern machine learning algorithms, in view of the future LHC runs. The ecosystem of tools presently in use will be presented, together with the state of the art of the developments toward more automatized monitoring and the roadmap for the future.
Anomaly detection using Deep Autoencoders for the assessment of the quality of the data acquired by the CMS experiment
Authors: Pol Adrian Alan , Azzolini Virginia , Cerminara Gianluca , De Guio Federico , Franzoni Giovanni , Pierini Maurizio , Siroký Filip , Vlimant Jean-Roch
Published: September 2019

The certification of the CMS experiment data as usable for physics analysis is a crucial task to ensure the quality of all physics results published by the collaboration. Currently, the certification conducted by human experts is labor intensive and based on the scrutiny of distributions integrated on several hours of data taking. This contribution focuses on the design and prototype of an automated certification system assessing data quality on a per-luminosity section (i.e. 23 seconds of data taking) basis. Anomalies caused by detector malfunctioning or sub-optimal reconstruction are difficult to enumerate a priori and occur rarely, making it difficult to use classical supervised classification methods such as feedforward neural networks. We base our prototype on a semi-supervised approach which employs deep autoencoders. This approach has been qualified successfully on CMS data collected during the 2016 LHC run: we demonstrate its ability to detect anomalies with high accuracy and low false positive rate, when compared against the outcome of the manual certification by experts. A key advantage of this approach over other machine learning technologies is the great interpretability of the results, which can be further used to ascribe the origin of the problems in the data to a specific sub-detector or physics objects.
The use of adversaries for optimal neural network training
Authors: Hawthorne-Gonzalvez Anton , Sevior Martin
Published: September 2019

B-decay data from the Belle experiment at the KEKB collider have a substantial background from e+e- -h> qq¯ events. To suppress this we employ deep neural network algorithms. These provide improved signal from background discrimination. However, the deep neural network develops a substantial correlation with the ∆E kinematic variable used to distinguish signal from background in the final fit due to its relationship with input variables. The effect of this correlation is reduced by deploying an adversarial neural network. Over-all the adversarial deep neural network performs better than a Boosted Decision Tree algorithimn and a commercial package, NeuroBayes, which employs a neural net with a single hidden layer.
New Machine Learning Developments in ROOT/TMVA
Authors: Albertsson Kim , Gleyzer Sergei , Huwiler Marc , Ilievski Vladimir , Moneta Lorenzo , Shekar Saurav , Estrade Victor , Vashistha Akshay , Wunsch Stefan , Zapata Mesa Omar Andres
Published: September 2019

The Toolkit for Multivariate Analysis, TMVA, the machine learning package integrated into the ROOT data analysis framework, has recently seen improvements to its deep learning module, parallelisation of multivariate methods and cross validation. Performance benchmarks on datasets from high-energy physics are presented with a particular focus on the new deep learning module which contains robust fully-connected, convolutional and recurrent deep neural networks implemented on CPU and GPU architectures. Both dense and convo-lutional layers are shown to be competitive on small-scale networks suitable for high-level physics analyses in both training and in single-event evaluation. Par-allelisation efforts show an asymptotical 3-fold reduction in boosted decision tree training time while the cross validation implementation shows significant speed up with parallel fold evaluation.
REANA: A System for Reusable Research Data Analyses
Authors: Šimko Tibor , Heinrich Lukas , Hirvonsalo Harri , Kousidis Dinos , Rodríguez Diego
Published: September 2019

The revalidation, reinterpretation and reuse of research data analyses requires having access to the original computing environment, the experimental datasets, the analysis software, and the computational workflow steps which were used by researchers to produce the original scientific results in the first place.

REANA (Reusable Analyses) is a nascent platform enabling researchers to structure their research data analyses in view of enabling future reuse. The analysis is described by means of a YAML file that captures sufficient information about the analysis assets, parameters and processes. The REANA platform consists of a set of micro-services allowing to launch and monitor container-based computational workflow jobs on the cloud. The REANA user interface and the command-line client enables researchers to easily rerun analysis workflows with new input parameters. The REANA platform aims at supporting several container technologies (Docker), workflow engines (CWL, Yadage), shared storage systems (Ceph, EOS) and compute cloud infrastructures (Ku-bernetes/OpenStack, HTCondor) used by the community.

REANA was developed with the particle physics use case in mind and profits from synergies with general reusable research data analysis patterns in other scientific disciplines, such as bioinformatics and life sciences.
Optimization of the SHiP Spectrometer Tracker geometry using the Bayesian Optimization with Gaussian Processes
Authors: Alenkin Oleg , Hushchyn Mikhail , Ustyuzhanin Andrey , Baranov Alexander
Published: September 2019

One of the most important aspects of data processing at SHiP [1] experiments is tracks pattern recognition. The purpose of the SHiP Spectrometer Tracker (SST) is efficient reconstruction of charged particle tracks originating from decays of neutral New Physics objects. The reconstruction performance strongly depends on the tracker design and should be considered as an objective to define the best SST geometry parameters. In this study the SHiP Spectrom eter Tracker geometry optimization using Bayesian optimization with Gaussian processes in considered. The study have been done on MC data. The first results of the optimization are also considered.
The TrackML high-energy physics tracking challenge on Kaggle
Authors: Kiehn Moritz , Amrouche Sabrina , Calafiura Paolo , Estrade Victor , Farrell Steven , Germain Cécile , Gligorov Vava , Golling Tobias , Gray Heather , Guyon Isabelle , Hushchyn Mikhail , Innocente Vincenzo , Moyse Edward , Rousseau David , Salzburger Andreas , Ustyuzhanin Andrey , Vlimant Jean-Roch , Yilnaz Yetkin
Published: September 2019

The High-Luminosity LHC (HL-LHC) is expected to reach unprecedented collision intensities, which in turn will greatly increase the complexity of tracking within the event reconstruction. To reach out to computer science specialists, a tracking machine learning challenge (TrackML) was set up on Kaggle by a team of ATLAS, CMS, and LHCb physicists tracking experts and computer scientists building on the experience of the successful Higgs Machine Learning challenge in 2014. A training dataset based on a simulation of a generic HL-LHC experiment tracker has been created, listing for each event the measured 3D points, and the list of 3D points associated to a true track.The participants to the challenge should find the tracks in the test dataset, which means building the list of 3D points belonging to each track.The emphasis is to expose innovative approaches, rather than hyper-optimising known approaches. A metric reflecting the accuracy of a model at finding the proper associations that matter most to physics analysis will allow to select good candidates to augment or replace existing algorithms.
Fair Share Scheduler for OpenNebula
Authors: Bagnasco Stefano , Vallero Sara , Zaccolo Valentina
Published: September 2019

A small Cloud infrastructure for scientific computing likely operates in a saturated regime, which imposes to optimize the allocation of resources. Tenants typically pay a priori for a fraction of the overall resources. Within this business model, an advanced scheduling strategy is needed in order to optimize the data centre occupancy. FaSS, a Fair Share Scheduler service for OpenNebula, addresses this issue by satisfying resource requests according to an algorithm, which prioritizes tasks according to an initial weight and to the historical resource usage of the project. In this proceedings, we are going to describe the implementation of FaSS Version 1.0, released in March 2017 as a product of the INDIGO-DataCloud project. We are also going to discuss the results of FaSS functional and stress tests performed at the Cloud infrastructure of the INFN-Torino computing centre.
Running Oracle WebLogic on containers
Authors: Aparicio Cotarelo Borja , Nappi Antonio , Gedvilas Lukas , Rodriguez Fernández Luis , Rizopoulos Theodoros , Wiecek Artur
Published: September 2019

CERN IT department is providing production services to run containers. Given that, the IT-DB team, responsible to run the Java based platforms, has started a new project to move the WebLogic deployments from virtual or bare metal servers to containers: Docker on Kubernetes allow to improve the overall productivity of the team by reducing both operations time and time-to-delivery. Additionally, this framework allows us deploy our services in a fully-reproducible fashion. The scope of the project goes from the design and production of Docker images, deployment environments based on Kubernetes as well as all procedures and operations including the needed tools to hand over to users the management of their deployed applications. This article illustrates how at CERN we have faced all the technical and design aspects to run it in a production environment. That means the implementation of the solutions needed including monitoring, logging, high availability, security and traceability.
Building and using containers at HPC centres for the ATLAS experiment
Authors: Benjamin Douglas , Childers Taylor , Lesny David , Oleynik Danila , Panitkin Sergey , Tsulaia Vakho , Yang Wei , Zhao Xin
Published: September 2019

The HPC environment presents several challenges to the ATLAS experiment in running their automated computational workflows smoothly and efficiently, in particular regarding issues such as software distribution and I/O load. A vital component of the LHC Computing Grid, CVMFS, is not always available in HPC environments. ATLAS computing has experimented with all-inclusive containers, and later developed an environment to produce such containers for both Shifter and Singularity. The all-inclusive containers include most of the recent ATLAS software releases, database releases, and other tools extracted from CVMFS. This helped ATLAS to distribute software automatically to HPC centres with an environment identical to those in CVMFS. It also significantly reduced the metadata I/O load to HPC shared file systems. The production operation at NERSC has proved that by using this type of containers, we can transparently fit into the previously developed ATLAS operation methods, and at the same time scale up to run many more jobs.
Containerized Batch System Monitoring
Authors: Gellrich Andreas , Hartmann Thomas , Lewendel Birgit , Prelogović David , Voss Christian
Published: September 2019

Running a batch system for grid jobs and for local users, we investigate a generic solution to monitor the resource usages of jobs. We extend a standard toolkit for monitoring container performance metrics also for non-containerized applications, so that we can make easy use of a popular industry solution. By concentrating on the basic kernel features also used by containers frameworks, we envisage to use the same tools on non-containerized batch systems as well as container orchestrators.

For deployment, we encapsulate the tools as Singularity containers and distribute them via CVMFS.
Experience with dynamic resource provisioning of the CMS online cluster using a cloud overlay
Authors: Andre Jean-Marc , Behrens Ulf , Branson James , Brummer Philipp , Chaze Olivier , Cittolin Sergio , da Silva Gomes Diego , Darlea Georgiana-Lavinia , Deldicque Christian , Demiragli Zeynep , Dobson Marc , Doualot Nicolas , Erhan Samim , Richard Fulcher Jonathan , Gigi Dominique , Gladki Maciej , Glege Frank , Gomez-Ceballos Guillelmo , Hegeman Jeroen , Holzner Andre , Lettrich Michael , Mecionis Audrius , Meijers Frans , Meschi Emilio , K.Mommsen Remigius , Morovic Srecko , O’Dell Vivian , Orsini Luciano , Papakrivopoulos Ioannis , Paus Christoph , Petrucci Andrea , Pieri Marco , Rabady Dinyar , Racz Attila , Rapsevicius Valdas , Reis Thomas , Sakulin Hannes , Schwick Christoph , Simeleviciu Dainius , Stankevicius Mantas , Vazquez Velez Cristina , Wernet Christian , Zejdl Petr
Published: September 2019

The primary goal of the online cluster of the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) is to build event data from the detector and to select interesting collisions in the High Level Trigger (HLT) farm for offline storage. With more than 1500 nodes and a capacity of about 850 kHEPSpecInt06, the HLT machines represent similar computing capacity of all the CMS Tier1 Grid sites together. Moreover, it is currently connected to the CERN IT datacenter via a dedicated 160 Gbps network connection and hence can access the remote EOS based storage with a high bandwidth. In the last few years, a cloud overlay based on OpenStack has been commissioned to use these resources for the WLCG when they are not needed for data taking. This online cloud facility was designed for parasitic use of the HLT, which must never interfere with its primary function as part of the DAQ system. It also allows to abstract from the different types of machines and their underlying segmented networks. During the LHC technical stop periods, the HLT cloud is set to its static mode of operation where it acts like other grid facilities. The online cloud was also extended to make dynamic use of resources during periods between LHC fills. These periods are a-priori unscheduled and of undetermined length, typically of several hours, once or more a day. For that, it dynamically follows LHC beam states and hibernates Virtual Machines (VM) accordingly. Finally, this work presents the design and implementation of a mechanism to dynamically ramp up VMs when the DAQ load on the HLT reduces towards the end of the fill.
Apache Spark usage and deployment models for scientific computing
Authors: Castro Diogo , Kothuri Prasanth , Mrowczynski Piotr , Piparo Danilo , Tejedor Enric
Published: September 2019

This talk is about sharing our recent experiences in providing data analytics platform based on Apache Spark for High Energy Physics, CERN accelerator logging system and infrastructure monitoring. The Hadoop Service has started to expand its user base for researchers who want to perform analysis with big data technologies. Among many frameworks, Apache Spark is currently getting the most traction from various user communities and new ways to deploy Spark such as Apache Mesos or Spark on Kubernetes have started to evolve rapidly. Meanwhile, notebook web applications such as Jupyter offer the ability to perform interactive data analytics and visualizations without the need to install additional software. CERN already provides a web platform, called SWAN (Service for Web-based ANalysis), where users can write and run their analyses in the form of notebooks, seamlessly accessing the data and software they need. The first part of the presentation talks about several recent integrations and optimizations to the Apache Spark computing platform to enable HEP data processing and CERN accelerator logging system analytics. The optimizations and integrations, include, but not limited to, access of kerberized resources, xrootd connector enabling remote access to EOS storage and integration with SWAN for interactive data analysis, thus forming a truly Unified Analytics Platform. The second part of the talk touches upon the evolution of the Apache Spark data analytics platform, particularly sharing the recent work done to run Spark on Kubernetes on the virtualized and container-based infrastructure in Openstack. This deployment model allows for elastic scaling of data analytics workloads enabling efficient, on-demand utilization of resources in private or public clouds.
Digital repository as a service: automatic deployment of an Invenio-based repository using TOSCA orchestration and Apache Mesos
Authors: Antonacci Marica , Brigandì Alberto , Caballer Miguel , Cetinić Eva , Davidovic Davor , Donvito Giacinto , Moltó Germán , Salomoni Davide
Published: September 2019

In the framework of the H2020 INDIGO-DataCloud project, we have implemented an advanced solution for the automatic deployment of digital data repositories based on Invenio, the digital library framework developed by CERN. Exploiting cutting-edge technologies, such as Docker and Apache Mesos, and standard specifications to describe application architectures such as TOSCA, we are able to provide a service that simplifies the process of creating and managing repositories of various digital assets using cloud resources. An Invenio-based repository consists of a set of services (e.g. database, message queue, cache, workers and frontend) that need to be properly installed, configured and linked together. These operations, along with the provisioning of the resources and their monitoring and maintenance, can be challenging for individual researchers or small-to-moderate-sized research groups. To this purpose, the INDIGO-DataCloud platform provides advanced features for orchestrating the deployment of complex virtual infrastructures on distributed cloud environments: it is able to provision the required resources automatically over heterogeneous and/or hybrid cloud infrastructures and to configure them automatically ensuring dynamic elasticity and resilience. This approach has been successfully adapted to support the needs of the researchers and scholars in the domain of the Digital Arts and Humanities.
Advanced features of the CERN OpenStack Cloud
Authors: Castro León José
Published: September 2019

The CERN OpenStack cloud has been delivering a wide variety of services to the whole laboratory since it entered in production in 2013. Initially, standard resources such as Virtual Machines and Block Storage were offered. Today, the cloud offering includes advanced features such as Container Orchestration (for Kubernetes, Docker Swarm mode, Mesos/DCOS clusters), File Shares and Bare Metal, and the Cloud team is preparing the addition of Networking and Workflow-as-a-Service components. In this paper, we will describe these advanced features, the OpenStack projects that provide them, as well as some of the main usecases that benefit from them. We will show the ongoing work on those services that will increase functionality, such as container orchestration upgrades and networking features such as private networks and floating IPs.
Exploiting private and commercial clouds to generate on-demand CMS computing facilities with DODAS
Authors: Spiga Daniele , Antonacci Marica , Boccali Tommaso , Ceccanti Andrea , Ciangottini Diego , Di Maria Riccardo , Donvito Giacinto , Duma Cristina , Gaido Luciano , López García Álvaro , Palacio Hoz Aida , Salomoni Davide , Tracolli Mirco
Published: September 2019

Minimising time and cost is key to exploit private or commercial clouds. This can be achieved by increasing setup and operational efficiencies. The success and sustainability are thus obtained reducing the learning curve, as well as the operational cost of managing community-specific services running on distributed environments. The greater beneficiaries of this approach are communities willing to exploit opportunistic cloud resources. DODAS builds on several EOSC-hub services developed by the INDIGO-DataCloud project and allows to instantiate on-demand container-based clusters. These execute software applications to benefit of potentially “any cloud provider”, generating sites on demand with almost zero effort. DODAS provides ready-to-use solutions to implement a “Batch System as a Service” as well as a BigData platform for a “Machine Learning as a Service”, offering a high level of customization to integrate specific scenarios. A description of the DODAS architecture will be given, including the CMS integration strategy adopted to connect it with the experiment’s HTCondor Global Pool. Performance and scalability results of DODAS-generated tiers processing real CMS analysis jobs will be presented. The Instituto de Física de Cantabria and Imperial College London use cases will be sketched. Finally a high level strategy overview for optimizing data ingestion in DODAS will be described.
FaaM: FPGA-as-a-Microservice - A Case Study for Data Compression
Authors: Ojika David , Gordon-Ross Ann , Lam Herman , Patel Bhavesh
Published: September 2019

Field-programmable gate arrays (FPGAs) have largely been used in communication and high-performance computing and given the recent advances in big data and emerging trends in cloud computing (e.g., serverless [18]), FPGAs are increasingly being introduced into these domains (e.g., Microsoft’s datacenters [6] and Amazon Web Services [10]). To address these domains’ processing needs, recent research has focused on using FPGAs to accelerate workloads, ranging from analytics and machine learning to databases and network function virtualization. In this paper, we present an ongoing effort to realize a high-performance FPGA-as-a-microservice (FaaM) architecture for the cloud. We discuss some of the technical challenges and propose several solutions for efficiently integrating FPGAs into virtualized environments. Our case study deploying a multithreaded, multi-user compression as a microservice using the FaaM architecture indicate that microservices-based FPGA acceleration can sustain high-performance compared to straightforward implementation with minimal to no communication overhead despite the hardware abstraction.
CVMFS: Stratum 0 in Kubernetes
Authors: Schultz David , Skarlupka Heath , Brik Vladimir , Merino Gonzalo
Published: September 2019

IceCube is a cubic kilometer neutrino detector located at the south pole. CVMFS is a key component to IceCube’s Distributed High Throughput Computing analytics workflow for sharing 500GB of software across datacenters worldwide. Building the IceCube software suite across multiple platforms and deploying it into CVMFS has until recently been a manual, time consuming task that doesn’t fit well within an agile continuous delivery framework. Within the last 2 years a plethora of tooling around microservices has created an opportunity to upgrade the IceCube software build and deploy pipeline. We present a framework using Kubernetes to deploy Buildbot. The Buildbot pipeline is a set of pods (docker containers) in the Kubernetes cluster that builds the IceCube software across multiple platforms, tests the new software for critical errors, syncs the software to a containerized CVMFS server, and finally executes a publish. The time from code commit to CVMFS publish has been greatly reduced and has enabled the capability of publishing nightly builds to CVMFS.
Container support in Vac
Authors: McNab Andrew
Published: September 2019

During 2017 support for Docker and Singularity containers was added to the Vac system, in addition to its long standing support for virtual machines. All three types of "logical machine" can now be run in parallel on the same pool of hypervisors, using container or virtual machine definitions published by experiments. We explain how CernVM-FS is provided to containers by the hypervisors, to avoid any need to operate the containers in privileged modes. Finally we describe how LHCb Docker containers use this model and have been used to validate the Vac implementation.
Quasi-online accounting and monitoring system for distributed clouds
Authors: Seuster Rolf , Berghaus Frank , Casteels Kevin , Driemel Colson , Ebert Marcus , Leavett-Brown Colin , Paterson Michael , Sobie Randall
Published: September 2019

The HEP group at the University of Victoria operates a distributed
cloud computing system for the ATLAS and Belle II experiments. The system
uses private and commercial clouds in North America and Europe that run
OpenStack, Open Nebula or commercial cloud software. It is critical that we
record accounting information to give credit to cloud owners and to verify our use of commercial resources. We want to record the number of CPU-hours of the virtual machine. We continuously collect the CPU usage and an estimate of the HEPSpec06 units of the VM obtained during the boot of the VM and
uploads it into an Elastic Search database. The information is processed and published as soon as it is available. The data is published in tables and plots in Kibana and as a cross check in ROOT. We have found the system to be useful beyond gathering accounting information and can be used for monitoring and diagnostic purposes. For example, we can use it to detect if the payload jobs are stuck in a waiting state for external information. We will report on the design and performance of the system, and show how it provides important accounting and monitoring information on a large distributed system.
Long-term experiences in keeping balance between safety and usability in research activities in KEK
Authors: Murakami Tadashi , Yuasa Fukuko , Baba Ryouichi , Nakamura Teiji , Hashimoto Kiyoharu , Suzuki Soh Y. , Nishiguchi Mitsuo , Kaneko Toshiaki
Published: September 2019

This work aims to provide KEK general-purpose network to support various research activities in the fields of high-energy physics, material physics, and accelerator physics. Since the end of the 20th century, on a daily basis, networks experience cyber-attacks and the methods of attack have rapidly evolved to become more sophisticated over the years. Security measures have been developed to mitigate the effects of cyber-attacks. While security measures may improve safety, restrictions might reduce usability. Therefore, we must keep a balance between safety and usability of the network for a smooth running of research activities.

Herein, we present our long-term experience with keeping a balance between safety and usability in KEK research activities. The key points are reasonably ensuring traceability and security management. We have been using security devices, such as firewalls, intrusion detection systems, and vulnerability management systems, to achieve a balance between safety and usability. Also, we present activities of the computer security incident response team (CSIRT) and collaborative activities among research organizations.
INFN Tier–1: a distributed site
Authors: dell’Agnello Luca , Boccali Tommaso , Cesini Daniele , Chiarelli Lorenzo , Chierici Andrea , Dal Pra Stefano , De Girolamo Donato , Falabella Antonio , Fattibene Enrico , Maron Gaetano , Michelotto Diego , Morganti Lucia , Prosperini Andrea , Sapunenko Vladimir , Zani Stefano
Published: September 2019

The INFN Tier-1 center at CNAF has been extended in 2016 and 2017 in order to include a small amount of resources (∼ 22 kHS06 corresponding to ∼ 10% of the CNAF pledges for LHC in 2017) physically located at the Bari-ReCas site (∼ 600 km distant from CNAF). In 2018, a significant fraction of the CPU power (∼ 170 kHS06, equivalent to ∼ 50% of the total CNAF pledges) is going to be provided via a collaboration with the PRACE Tier-0 CINECA center (a few km from CNAF), thus building a truly geographically distributed (WAN) center. The two sites are going to be interconnected via an high bandwidth link (400-1200 Gb/s), in order to ensure a transparent access to data residing on CNAF storage; the latency between the centers is small enough not to require particular caching strategies. In this contribution we describe the issues and the results of the production configuration, focusing both on the management aspects and on the performance provided to end-users.
Challenges, solutions and lessons learnt in 7 years of Service Management at CERN
Authors: Martín Clavo David , Brugger Barbara , Cremel Nicole , Delamare Catherine , Fernandez Gonzalez Isabel , Lienard Eric , Martens Reinoud , Moller Mats , Salter Wayne , Toteva Zhechka , Van der Vossen Olaf
Published: September 2019

CERN has been using ITIL Service Management methodologies and ServiceNow since early 2011. Initially a joint project between just the Information Technology and the General Services Departments, now most of CERN is using this common methodology and tool, and all departments are represented totally or partially in the CERN Service Catalogue. We introduce a summary of the current situation of Service Management at CERN, as well as its recent evolution. We discuss service onboarding, user experience, and tool configuration, outlining the challenges faced, solutions adopted and some important lessons learnt. Some of the most important points relate to CERN’s common process and service catalogue, which have enabled scalability; hiding complexity from users in order to improve user experience; and the good practices to configure CERN’s Service Management application which have been learnt in the 8 years of the project. Finally, we present ongoing and future work, such as the upgrade of the CERN Service Portal and bringing even more new services board.