<figure><img src="image_85.png" /><figcaption>**Figure 75.** Hit time within the orbit of randomized hits sent at fixed rate to HPTDC inside a TOF crate operated in continuous readout mode. No holes are seen in the distribution, which means that random hits are received in all 3564 bunch crossings through the whole LHC orbit.</figcaption></figure>

FPGA (Microsemi IGLOO2). Overall, it replaces the connections to the central readout and from the central trigger in the DRM1, which were based respectively on the Detector Data Link (DDL) [61] and the TTC system [4], with just one bidirectional GBT link (see section 2.2). The GBTx ASIC is hosted on the DRM2 and the GBT protocol is implemented in the FPGA on the CRU at the receiving end. The new system for each DRM2 has a user bandwidth to the central readout system (CRU) of 3.2 Gb/s, corresponding to the bandwidth available on a single GBT link.

As mentioned, the readout is implemented with special TOF triggers at fixed bunch crossing values with $f_T = 33$ kHz, setting a matching window of 30 µs in the HPTDC installed in the TRMs to achieve continuous readout. The same link is also used for receiving triggers and a low-jitter clock, which is distributed to the front-end electronics as primary clock. For the TOF detector, the quality of this clock is crucial, and a campaign of measurements on the clock received from the common readout unit has been carried out. A clock jitter (RMS) as low as ~10 ps was measured in the laboratory, which is compatible with the requirements. Nevertheless, a dedicated line of clock distribution is available (same as during Runs 1 and 2) with a similar jitter.

A picture of the DRM2 card is shown in figure 76. It is a narrow 9U VME card (16 cm × 33 cm) with the same form factor as the DRM1 and the TRM boards. The heart of the board is a Microsemi Flash-based IGLOO2 FPGA (M2GL090-FG676 with silicon revision 3), which drives the trigger and data flows inside the crate. This device has been chosen since the expected TID (Total Ionizing Dose) for the board (placed at ≈ 4 meters from the beam pipe) is 0.13 krads in 10 years, which is acceptable for such a device. The advantage is that the FPGA configuration memory is immune to single event upsets, so that scrubbing is not needed. Results of irradiation tests on several components of the DRM2, including the IGLOO2 FPGA, commercial optical transceivers, and staging RAM have been reported in [62].

The FPGA's GBTx connection consists of a single 40-bit large parallel lane of 80 MHz differential signals. The same configuration was previously tested on a GBTx test board developed before designing the DRM2, where a bit-error ratio lower than $10^{-14}$ and a total jitter on the received clock around 50 ps had been measured [63]. As on the DRM1, an additional optical Slow Control
<figure><img src="image_86.png" /><figcaption>**Figure 76.** The DRM2 card: on the left the VTRX transceiver and the GBTx ASIC (covered by a heat dissipating panel). On the right the ARM piggy-back card is visible. The additional optical receivers for the SCL and the LHC clock are in the middle of the front panel.</figcaption></figure>

Link (SCL) is implemented. The SCL has a dual role: configuration and monitoring. It provides a firmware implementation of the CONET2 protocol developed by CAEN [64]. Via this link, all DRM2 are connected to commercial A3818 PCIe cards, housed in Linux machines hosted in the DCS network. The SCL is used for configuration of the front-end electronics and programming of all VME cards. In addition, while the data collected are immediately sent to the CRU via the GBT link, the firmware also stores them in the staging RAM (1M × 36 bit SSRAM from Cypress: CY7C1460KV33) for transfer (1 MB buffers) to the DCS machines. From these data (only a portion of all data are inspected via the SCL), some values such as temperatures are stripped and reported in the DCS via DIM servers. In addition, Quality Control programs run on these data.

A block of hardware inherited by DRM1 is the ARM microprocessor mounted on the A1500 piggy-back card provided by CAEN. This CPU implements, via JTAG over the VME backplane, the programming of the Actel APA750 and APA600 installed in the TRM and LTM cards, respectively. The connection on the front panel to the console port of this CPU was improved, with respect to the DRM1, by using a commercial RS232-USB interface to provide a more modern USB interface to connect a laptop computer. The ARM CPU is also able to program the firmware of the IGLOO2 FPGA. As for the A1500 mounted on the DRM1, thanks to the modified Ethernet interface (which has been validated to operate in magnetic field), all the firmware updates of the VME cards (DRM2, TRM and LTM) can be remotely executed.

Finally, the DRM2 distributes the clock to all VME cards inside the crate. This is an entirely new functionality with respect to the DRM1. Previously, only every second TOF crate had a clock distribution module (Clock and Pulse Distribution Module, CPDM). This complicated the power-up and configuration sequences, and created single points of failure (e.g. via the dependence on the crate power supply). The DRM2 distributes a local clock to all cards in the VME crate in a user-selectable way, using either the clock received via GBTx or the clock received directly from the LHC interface. For the latter, an optical receiver from PD-LD/NECSEL is used, with ST plug-type, with pinout compatible with the Truelight TRR-1B43-000, which was previously widely used for TTC applications. This configuration minimizes the jitter of the clock distributed to the TRM cards.

All the DRM1s were removed and disassembled during the first months of 2019. The A1500 ARM piggy-back cards were tested and prepared for installation on the DRM2 cards. The procedure
for validation and test of the DRM2 production (completed during 2019) is described in [65]. The installation of all DRM2 cards, partially delayed by the Covid-19 pandemic, was ended in June 2020 with full commissioning starting in September 2020. The MRPC high voltages were ramped to the nominal value (V = 6500 V on each stack) in the same month. The TOF detector actively participated to the data taking with the LHC pilot beams in October 2021.

The front-end software needed for configuration of the electronics was upgraded and deployed in a staged way during spring 2020.

### 3.8.3 Additional upgrades in low voltage and quality control systems

During LS2, several other TOF systems linked to the readout upgrade were subject to key improvements and maintenance to prepare for the intense data taking foreseen in Run 3. Among many interventions, we highlight in particular:

* The DC/DC systems (CAEN modules A1395 and A1396): these modules supply power to the four crates of each TOF supermodule. They receive a DC 48 V power supply via bus-bars from outside the L3 Magnet and provide LV power supply for the VME boards and the front-end cards on the MRPC modules. A solid state fuse that was subject to frequent breakdown was replaced. Additionally, a study via proton irradiation at the Centro di Protonterapia in Trento in 2019 investigated the cause of SEU events registered in 2018 at high irradiation-rates that produced sudden loss of communication with the module. The addition of a filter capacitor on the reset line of the microprocessor on the A1396 fixed the problem. A full refurbishment of all modules (entailing dismounting 216 modules from the detector) was completed in 2019-2020.

* All DRM2s are equipped with an ARM microprocessor (AT91RM9200 from Atmel) running Linux. During Runs 1 and 2 these CPUs were used exclusively to perform firmware upgrades on the VME boards via the JTAG interface on the VME backplane using Actel software for APA FPGAs. Using the cross-platform development tools provided, a full slow control DIM server has been deployed on these CPUs. This provides an additional channel to monitor voltages and temperature on the cards (even if the SCL is not connected). More importantly, thanks to a different hardware implementation on the DRM2 with respect to the DRM1, via the server running on ARM CPUs it will be possible to reset the CONET link and to reset the Microsemi FPGA of the DRM2. These two emergency resets may be used in case of loss of communication with the DRM2 (on the SCL), without the need of executing a power cycle. This may be useful especially because the DRM2 provides the primary clock to all TRM cards and a DRM2 power cycle would cause the loss of the clock and therefore the need of a power cycle in all VME slots in the crate.

* The procedure for the control and validation of the recorded data has been integrated into the O² framework under the project of the Quality Control (QC). For example, counters reporting errors detected in some HPTDC on the TRM are provided, as well as information on the hit rate on all 150 000 channels of the TOF detector. Noisy channels, identified by a hit rate greater than 1 kHz, can be disabled individually. Quality control tasks monitor the reconstruction and calibration process at various levels in order to provide a detailed insight
into the various steps of the data processing. The QC code runs on dedicated computing nodes (FLPs or EPNs) to monitor the TOF data stream, and in the DCS system, sampling data through the SCL. For the SCL part, the QC task is primarily intended to monitor temperatures of front-end electronics while readout is ongoing, as well as to monitor the hit rate, turning off quickly very noisy channels that could prevent the continuous readout (on average each event has to be read out in 30 µs, see section 3.8.1). While a certain degree of duplication exists, the FLP and EPN QC provides more aggregated information for the whole TOF, like the average multiplicity per matching window.

* The data flow from the CRUs is processed by the CPUs to perform the first level of data decoding and some preprocessing, which produces a second level of raw data, providing a zero-suppressed data stream where the relevant information is stored in a compact format. This effectively reduces the output bandwidth from the FLP to the EPNs by a factor of four for very high multiplicity events, and by a much larger factor for low multiplicity events. This allows the framework to make the best use of the available computing resources on the FLPs by performing low-level data monitoring. The QC system is able to access the preprocessed data directly on the FLPs for monitoring of the raw data stream as early as possible.

## 3.9 High-momentum particle identification

### 3.9.1 Introduction

The ALICE High Momentum Particle Identification (HMPID) detector [1, 66] is designed to identify hadrons at $p_T > 1$ GeV/c. It is based on Ring Imaging Cherenkov detectors (RICH). Seven MWPCs, equipped with CsI segmented photocathodes, detect Cherenkov patterns. Together with the momentum measured by the TPC, they allow the determination of the particle mass. During Run 3, the main goal of the detector is to identify light nuclei and corresponding anti-nuclei at high transverse momenta in the central rapidity region, up to 12 GeV/c for the deuteron and triton, and up to 10 GeV/c for $^3$He. During LS2, the readout firmware was upgraded to allow an increase of the event readout rate. In addition, to measure the inelastic cross section of (anti-)deuterons in the momentum range 0.2 to 2.2 GeV/c, two aluminum absorbers of 8 cm thickness, were installed in front of two RICH modules. In one module, two out of three Cherenkov radiator gas vessels were leaking, whereas the second RICH was located in a favourable position for the installation of the second absorber, needed for the required statistical abundance of the measurement. The consequent loss of acceptance of the PID measurement is largely compensated with the remaining modules by an event readout rate ten times higher than in Run 2. The detector stayed in place during the long shutdown, since moving it to a laboratory for repairing and upgrades was considered too risky. In turn a re-design of the readout firmware was carried out as explained in the next section.

### 3.9.2 Upgrading of readout firmware and trigger

Figure 77 shows the block diagram of the HMPID readout chain. The Readout Control Board (RCB) houses the readout FPGA, the TTCRx mezzanine card and the Source Interface Unit (SIU) interfacing the trigger and DAQ systems. The FPGAs firmware (FW) synchronises the trigger and data readout and is a key element of the readout performance. The C-RORC cards are installed on O² First Level Processor (FLP) computers and connected via optical links to the HMPID readout electronics.
<figure><img src="image_87.png" /><figcaption>**Figure 77.** From left to right: HMPID Front-End Electronics (FEE), Readout Control Board (RCB) with the readout FPGA, the TTCRx and the Source Interface Unit (SIU). On the right, the C-RORC cards installed on $O^2$ FLP computers.</figcaption></figure>

<figure><img src="image_88.png" /><figcaption>**Figure 78.** HMPID full DAQ structure. Four C-RORC boards are installed on two First Level Processor (FLP) computers of the $O^2$ data acquisition environment. Fourteen optical links connect the 14 RCBs, two per RICH module.</figcaption></figure>

The block diagram for the HMPID DAQ system is shown in figure 78. Fourteen optical links, two per RICH module, are connected to four C-RORC cards, on two FLPs.

### 3.9.3 New readout firmware and readout rate

In Run 2, the detector was operated at lower readout rates, in line with the TPC (2 kHz and 800 Hz, respectively, in pp and Pb-Pb collisions). As a result, the effective increase in data sample size between Run 2 and 3 can be up to a factor 14 and 10, respectively, in pp and Pb-Pb collisions. In order to improve the readout speed in Run 3, a new readout firmware was designed, tested in the laboratory, and finally deployed on the RICH modules. In laboratory tests, the HMPID has reached a readout rate up to 28 kHz for pp collisions (readout data headers only, a factor five improvement with respect to Run 2) and 9 kHz in Pb-Pb collisions (about a factor three improvement with respect to Run 2).

The readout firmware runs on the ALTERA Stratix II EP2S15F484C5 FPGA housed on the Readout and Control Board (RCB). The firmware improvements listed below resulted in higher and more stable data acquisition rates. The improvements are:

* shortening of the event readout time by 90 µs due to the omission in Run 3 of the L2a trigger level with its long latency,

* skipping of empty readout columns (in pp collisions only one out of seven events has a track in the HMPID acceptance, which results in an occupancy of 0.17%),

* the digitization and data transfer from FEE cards to the column memory buffer, which is now carried out in parallel to the decoding of the L1 trigger message in the TTCRx card. On average, it completes after 15.6 µs from the arrival of the LM trigger,

* masking of failing electronic columns, which is applied during the configuration of the FEE.
The final readout performance, as measured in the ALICE experiment with simulated occupancy, is shown in figure 79. With 0.17% occupancy (pp collisions), the measured acquisition rate is 22 kHz, whereas with 2% occupancy (in Pb-Pb collisions) it is 9.6 kHz. During the LHC pilot beam campaign in October 2021, the HMPID recorded events at 11.2 kHz with the zero bias trigger, coherently with the measured readout performance. In fact, the LHC filling schema had two colliding bunches with a ~20 µs separation. On average, only the first one was accepted in the HMPID, whereas the second colliding bunch was usually rejected by the busy time of 45 µs. Differently spaced colliding bunches, as expected during the normal operation of the LHC, will allow the full exploitation of the readout performance.

<figure><img src="image_89.png" /><figcaption>**Figure 79.** Event rate as a function of the detector occupancy, which is on average about 0.17% and 2% in pp and Pb-Pb collisions, respectively.</figcaption></figure>

### 3.9.4 Detector calibration formalism

The detector-calibration method has changed with respect to Run 2. During a standalone run without zero suppression, a dedicated workflow at the EPN level computes the average values of the pedestal distributions and the corresponding standard deviation. These values are archived in the Calibration and Constants Data Base (CCDB, see section 5). The DCS retrieves these data from the CCDB and uploads the pedestal values and the standard deviations in the readout electronics, via the ALFRED mechanism (see section 5.6). The granularity of this calibration mechanism is at the level of a single readout column, with a single calibration file per column. Each C-RORC link configures 24 columns, which correspond to the right or left half of a RICH module.

### 3.9.5 Other subsystems

**Detector Control System.** The DCS for the HMPID was upgraded providing the following new features:
<figure><img src="image_90.png" /><figcaption>**Figure 80.** Expected relative statistical uncertainty on the anti-deuteron absorption cross section in p-Pb and Pb-Pb collisions for the full Run 3 data sample and for one half of the Pb-Pb sample.</figcaption></figure>

• uploading the pedestal and sigma values in the readout electronics via the new ALFRED formalism;

• monitoring the busy time of a single RO link;

• automatically ramp up tripped HV channels.

**Absorbers for anti-deuteron inelastic cross section measurement.** Another important achievement during LS2 is the installation of two aluminium absorbers of 8 cm thickness, corresponding to half an interaction length for the anti-deuteron inelastic cross section measurement in the momentum range of 0.2 to 2.2 GeV/c. The (anti-)deuterons impinging on the two absorbers will be identified using the $dE/dx$ measured by the TPC and the time-of-flight measured by the TOF detector. The detection of secondary particles produced in the hadronic interaction with the target nuclei will be carried out by the pad-segmented cathodes of the HMPID-MWPCs installed right behind the absorbers.

During Run 3 the expected statistical precision of this measurement is expected to be in the range 5–10% in the momentum interval $0.2 < p < 2.2 \text{ GeV/c}$ for p-Pb collisions at $\sqrt{s_{NN}} = 8.8 \text{ TeV}$ and 5–8% in the momentum interval $0.2 < p < 1.4 \text{ GeV/c}$ for Pb-Pb collisions at $\sqrt{s_{NN}} = 5.5 \text{ TeV}$ (see figure 80). A systematic uncertainty of maximum 5.5% is expected based on a conservative estimate.

In ALICE an effort is ongoing to measure the (anti-)deuteron inelastic cross section, also at low momentum, using the TRD as an absorber. This detector has an average mass number $\langle A \rangle \approx 8$ considering the gas mixture, the active detector materials, and the support structure. The measurement using aluminum ($A = 27$) as a target will provide complementary information to other existing approaches and these measurements will allow the study of the mass-number dependence of the (anti-)deuteron inelastic cross section at low momentum, which is currently unknown.
### 3.10 Electromagnetic Calorimeter

The Electromagnetic Calorimeter (EMCal) [67, 68] was designed for the measurements of electrons from heavy-flavor hadron decays, of the electromagnetic component of jets, and of direct photons and neutral mesons. The calibration procedures and achieved performance during Runs 1 and 2 are described in detail in [69].

The calorimeter remains a trigger detector during Run 3 and will continue to provide L0, L1-$\gamma$ and L1-jet triggers (described in section 5.5). It is also possible to read out the calorimeter with any other trigger provided by the CTP (e.g. FIT minimum bias trigger). The hardware did not need any modifications to comply with the Run 3 requirements. However, in order to be able to operate during Run 3, the following actions were taken: spare hardware was produced in order to ensure operation during Runs 3 and 4, and the front-end electronics firmware was upgraded in order to satisfy the specifications for Run 3. Details will be given below.

The EMCal is a shashlik-type lead-scintillator sampling calorimeter comprising 4416 individual modules that are grouped into 20 Super Modules (SM). Each of the modules is composed of four optically isolated towers, resulting in 17 664 individual towers in total. The optical readout of each tower is provided using wavelength shifting fibers coupled to an Avalanche Photo Diode (APD).

The front face dimensions of the towers are 6 × 6 cm² resulting in individual tower acceptance of $\Delta\eta \times \Delta\varphi \approx 0.0143 \times 0.0143$ at $\eta = 0$. The towers are arranged within the SMs such that each tower is approximately projective to the interaction vertex in $\eta$ and $\varphi$. The towers are operated at ~25°C ambient temperature with a nominal APD gain of ≈ 30, to achieve a 14-bit effective dynamic energy range from ~16 MeV to ~250 GeV per tower.

The overall design of the calorimeter is heavily influenced by its integration within the ALICE setup [1]. SMs of 3 different sizes are used: full-size, 2/3-size and 1/3-size. Each full-size SM consists of 12 × 24 = 288 modules arranged in 24 strip modules of 21 × 1 modules each. The 1/3 and 2/3 size SMs consist of 4 × 24 = 96 and 12 × 16 = 192 modules, respectively.

The detector consists of two parts, that cover two different regions in azimuth, as illustrated in figure 81 (see also figure 1). The main segment of the EMCal consists of ten full-size SMs and two 1/3-size SMs covering $|\eta| < 0.7$ in azimuth and $80° < \varphi < 187°$ in azimuth, while six 2/3-size SMs and two 1/3-size SMs are installed around the PHOS detector, covering $0.22 < |\eta| < 0.7, 260° < \varphi < 320°$ and $|\eta| < 0.7, 320° < \varphi < 327°$. The latter part of the detector is sometimes referred to as Di-Jet Calorimeter (DCal). In the following, we will use the term EMCal to refer to the full system, and DCal only when the distinction between the two segments is useful.

The SMs are located at $R \approx 4.3$ m in radial distance from the beamline, inserted into support frames situated between the time-of-flight detector and the ALICE L3 magnet. The weight of a single full-size SM is ≈ 7.7 tons, and the total weight of all 20 SMs is ≈ 120 tons. More details regarding the mechanical structure and Front-End Electronics (FEE) can be found in [67, 68].

An individual EMCal tower is read out with an avalanche photodiode and preamplifier mounted on the tower. The preamplifier signal is split into energy and trigger shaper channels on the FEE boards [70]. The energy shaper signals are sampled at 10 MHz with 10-bit resolution using ALTRO chips [71]. Prior to digitisation, each energy signal is split into high and low gain channels, each shaped separately, with a gain ratio of 16 to provide an effective dynamic range of 14 bit. Each FEE board provides readout of the high and low gain channels from 32 towers.
<figure><img src="image_91.png" /><figcaption>**Figure 81.** Schematic view of the EMCal, consisting of two disjunct detector segments, in the top-left hemisphere and the bottom-right hemisphere (DCal), covering approximately opposite locations in azimuth. The PHOS calorimeter inside the DCal segment is indicated in brown.</figcaption></figure>

The ALTRO chips are configured to record 15 10-bit time (pre-)samples per readout channel per event to cover the 1.5 µs integration window. The data are compressed by discarding samples close to the reference level (pedestal) that contain no useful information (“zero suppression”), reducing substantially the data volume. The pedestals are obtained from special runs with no pre-programmed pedestal or signal present.

### 3.10.1 The readout system

Each SM is equipped with a readout concentrator, the so called Scalable Readout Unit (SRU) [72]. The SRU interconnects with each FEE board through a custom daughter card which was designed for the EMCal FEE board. It provides interface compatibility between the SRU and the EMCal FEE board to provide the Data, Trigger, Clock and Control (DTC) links. The maximum bandwidth of a DTC link on the SRU is 2 Gb/s. In the EMCal application, the bandwidth of the DTC link is conservatively limited to 20 MB/s due to the hardware capability of the rather outdated FEE board FPGA (Altera ACEX 1K Family EP1k100QC208-3). However, the DTC link does not limit the EMCal data throughput.

Each SRU has a total of 40 point-to-point links to connect to 37 FEE and three TRU boards for the full size EMCal SMs, and sends the data to an FLP through two Detector Data Links (DDL), see section 5.2. The SRU board integrates a TTCrx (LHC Trigger, Timing, and Control receiver) [73], which can receive trigger and timing information from the ALICE Trigger system. It also has three SFP+ ports directly connected to the FPGA's high speed serial transceivers for serial data transport at up to 5 Gb/s and an additional SFP+ port that provides a 10 Gb Ethernet link. For the EMCal application, one of these transceivers is used for the Ethernet connection to the ALICE detector control system, while the other two transceivers are used for the two DDL links to transmit the data to an FLP.
### 3.10.2 Trigger

The EMCal provides inputs to the L0 and L1 trigger decisions in ALICE (section 5.5). The trigger subsystem resides in specific hardware boards. The analog signals of 2 × 2 adjacent towers are summed in the FEE boards and transmitted to a Trigger Region Unit (TRU) board, where the 2 × 2 tower sums from twelve FEE cards (2 × 2 sums from 96 channels) are digitized at the LHC clock frequency of 40 MHz [74]. The digitized 2 × 2 tower sums are summed over time samples with pre-sample pedestal subtraction to provide an integral energy measurement, referred to as time sum. Finally, overlapping 4 × 4 tower digital sums are formed within each TRU and a peak finding algorithm is used to find a signal peak. Each 4 × 4 sum signal peak amplitude is then compared against a threshold to provide a L0 trigger output that indicates the presence of a high energy shower in the TRU region (1 TRU covers 1/3 of the area for a full-size SM). The L0 trigger decision from each TRU is passed to a Summary Trigger Unit (STU), which performs the logical OR of the L0 outputs from all TRUs to provide a single L0 input to the ALICE CTP.

Upon reception of an accepted L0 trigger from the CTP, the digitized time-summed 2 × 2 tower sums from each TRU are passed to the STU. In the STU the 4 × 4 overlapping tower sums are formed again, but across TRU boundaries over the full acceptance to provide an improved L1 high energy shower trigger referred to as L1-γ trigger [75]. At the same time, tower sums over a large 8 × 8 trigger channel window (16×16 towers) and a 16×16 trigger channel window (32×32 towers) are also formed to provide a L1 jet trigger. Both L1 triggers allow to define two thresholds for the event selection.

In order to reduce the bias due to multiplicity fluctuations in heavy-ion collisions, there is a direct communication between the STUs of the main EMCal segment and the DCal to consider the underlying event background in the online L1 trigger decision. The background is estimated based on the median of the energies deposited in 8 × 8 trigger channel (16 × 16 towers) windows in the opposing segment of the detector. The background is subtracted from the signal amplitude and then compared against a threshold to provide L1 triggers.

### 3.10.3 Spare production

In order to guarantee a smooth operation of the detector through Run 3, additional FEE boards were produced. They are identical to the ones used during Runs 1 and 2. A total of 100 front-end cards and six TRUs, amounting to 15% of the units used in the experimental cavern, were produced. In addition, 2 STUs were produced as spares.

### 3.10.4 Front-end electronics firmware upgrade

The firmwares of SRU and of STU had to be upgraded in order to satisfy the requirements concerning the readout rate for Run 3. In particular, in order to increase the readout rate for the anticipated 50 kHz minimum bias Pb-Pb interaction rate for Run 3, a multi-event buffering (MEB) logic was implemented in the SRU firmware, allowing to accept a trigger while the data from the previous trigger is being processed. The importance of multi-event buffering for the data recording rate as a function of interaction rate is shown in figure 82. The left-hand plot shows the predictions expected from Monte Carlo simulations, and the results from measurements with black events are shown on the right-hand plot. To estimate the SRU readout rate, which depends on detector occupancy, some Pb-Pb data from Run 2 were used. Pedestal data are used to create the load expected for minimum
<figure><img src="image_92.png" /><figcaption>**Figure 82.** SRU readout rate as a function of interaction rate for different Multi-Event Buffering (MEB) schemes. The left panel shows simulated data, while the right panel shows measured performance.</figcaption></figure>

bias Pb-Pb collisions. For early readout rate estimates [59], the detector occupancy was emulated by masking ALTRO channels in the FEE configuration (open markers on the left-hand side plot). This was improved later by applying a relatively high value for the baseline and by suppressing the data in the ALTRO channels (full markers). This procedure yields results that are closer to real data. The readout rate decreased by ~10% compared to preliminary expectations for both single- and multi-event buffering configurations. With four event buffers, a readout rate of ~35 kHz is expected for minimum bias Pb-Pb collisions at 50 kHz, and the results from Monte Carlo simulations are in a good agreement with the results from measurements at Point 2.

In addition, further improvements were implemented in the SRU firmware for increasing the readout stability and the physics performance. In particular, a synchronization between the LHC 40 MHz clock and the ALTRO 10 MHz clock was implemented in order to perform online time calibration during Run 3.

The firmware of the STU was upgraded to conform to the Run 3 trigger and DAQ protocols. The STU readout time highly depends on the data size to be sent from STU to DAQ. It is ~ 50 µs when sending the data from all trigger channels needed for the full QC. However, for physics analysis only the channels contributing in the trigger decision are selected by the STU FPGA, resulting in a readout time of ~ 20 µs. During normal operation for physics data taking, the data from all channels will be recorded for only ≤ 1% of events, and the average readout time is expected to be close to 20 µs.

### 3.10.5 Data compression

In order to reduce the amount of data written to tape, a fit of the time series of raw ADC signals is performed per tower, extracting amplitude and peak time. This fit is performed during the synchronous reconstruction. Amplitude and time, as well as tower index and gain type are encoded in a 48-bit word per tower. Further compression can be achieved by removing low tower energy signals which are rejected in the clusterization process and will therefore not contribute to physics measurements.
### 3.10.6 Calibration

The calibration procedure is based on existing calibration procedures used during Run 2. At the beginning of the data taking process of Run 3, a sample of events will be used to determine the absolute energy scale for each tower, based on the comparison of the reconstructed $\pi^0$ mass to the nominal mass. Identification of bad channels, which need to be removed in the analysis process, and calibration of the time measurement with respect to the collision time are based on the event-by-event tower energy and time measurement and are performed for all data blocks. Calibration histograms are filled and calibration parameters are determined in an automatized procedure during the synchronous and asynchronous reconstruction using the $O^2$ calibration software framework. The remaining time-dependence of the energy calibration resulting mainly from the sensitivity to the temperature are calibrated using the light-emitting diode (LED) system, by generating an ultra-bright blue light triggered by the CTP [67].

### 3.10.7 Quality Control

Monitoring of the data quality is based on the Quality Control framework within the ALICE $O^2$ computing framework. The EMCal Quality Control is designed to provide sufficient information in order to identify problems during the data taking process, and to decide on the usability of data blocks for physics measurements. Quality Control for EMCal consists of the following tasks:

* Raw Data level: a fraction of the raw data is inspected in order to find problematic parts of the detector, and to check for errors in the raw stream

* Digit level: tower-based quantities (energy, position, time) after the fit to the raw data are monitored in order to find inactive or noisy regions of the detector.

* Cluster: A calorimeter cluster, an aggregate of adjacent calorimeter cells with energy above the noise threshold, is the main object delivered by the reconstruction software. Basic EMCal cluster observables, such as energy, position, number of contributing towers, and others, are monitored at the level of the synchronous and asynchronous reconstruction stages for a fraction of the data.

* Trigger level: trigger-level digits are monitored for a small fraction of the data in order to identify noisy regions in the trigger system.

The Quality Control is histogram-based. The histograms are further processed to produce derived observables which are monitored continuously to follow the time evolution of the detector state.

### 3.11 Photon Spectrometer

The Photon Spectrometer (PHOS) [76] is a precise electromagnetic calorimeter which specialises in the detection of photons with high energy and spatial resolutions. PHOS covers a limited acceptance at mid rapidity $|y| < 0.13$ and azimuthal angle $250^\circ < \varphi < 320^\circ$. The layout of the PHOS detector surrounded by the DCal is depicted in figure 81. The physics objectives of the PHOS detector are the measurement of direct photon yields in the energy range from $\approx 0.1$ to $100$ GeV, azimuthal anisotropy of photon emission, photon-hadron correlations, as well as measurements of light neutral mesons $\pi^0$, $\eta$, $\omega$ with transverse momenta $p_T$ above $\approx 0.6$ GeV/c, with the upper $p_T$ limit being driven mainly by available statistics.
### 3.11.1 Detector layout

PHOS consists of 4 modules assembled from the active detection elements consisting of lead tungstate (PbWO₄) crystals with avalanche photodiode (APD) photodetectors and preamplifiers. These detection elements, called cells, compose rectangular matrices which are called modules. Three PHOS modules covering the azimuthal angle range 260° < φ < 320° consist of 64 × 56 cells, and one module at the angles 250° < φ < 260° is a matrix of 32 × 56 cells. Figure 83 depicts one cell and cells stacked into a module matrix. The front surface of each crystal is positioned at a distance of 460 cm from the beam axis.

<figure><img src="image_93.png" /><figcaption>**Figure 83.** PHOS detection element consisting of a PbWO₄ crystal and APD with preamplifier (left) and a fraction of a cell matrix of one PHOS module (right).</figcaption></figure>

The active detection elements of the cells are made of lead tungstate, PbWO₄, an inorganic crystalline scintillator which has a high density ρ = 8.3 g/cm³, a radiation length $X_0 = 0.89$ cm, and Molière radius $R_M = 2.0$ cm. The light yield of the crystal is about 0.3% of the light yield of NaI. The luminescence of PbWO₄ has a wide spread in the region of visible photons with a maximum at $\lambda_{max} = 420$ nm. The light yield of the crystals changes by -2.5% for every Kelvin temperature change around the operating temperature.

The PHOS modules are operated at a temperature of -25°C to achieve an increase of the light yield by a factor of three compared to room temperature. In order to ensure these working conditions and to provide thermal stabilization of 0.1%, the crystal matrices of the PHOS modules are housed in a thermoinsulated volume cooled down by C₆F₁₄ for which the flow is provided by the cooling plant installed outside the ALICE solenoid magnet at about 10 m from the PHOS modules. The PHOS modules also contain a so-called “warm volume” with front-end electronics. The warm volume and the cold crystal volume are contained in air-tight boxes, through which dry nitrogen is blown in order to maintain a low humidity. The environment inside the modules is monitored by a set of temperature and humidity sensors.

### 3.11.2 Readout

For LHC Run 3, like in the previous Runs 1 and 2, PHOS remains a triggered detector, i.e. acquiring data upon receiving a trigger from the ALICE trigger system. The new ALICE trigger protocol provides 2-level triggers with the level-0 trigger generated 0.8 µs after the collision, followed by the
level-1 trigger with a latency of 6.5 µs. The PHOS readout system receives the L0 trigger generated by one of the ALICE trigger detectors (FIT, EMCal, TOF, PHOS, etc.). The choice of the L0 trigger source is configured by the central trigger processor (CTP). After receiving the L0 trigger, the PHOS readout raises the busy signal to block further reception of triggers until the accepted trigger is processed and shipped to the FLP upon receiving the L1 trigger. The busy signal is lowered after sending the whole data payload to the FLP. The dead time depends on the payload size and varies from 20 to 55 µs. Shipping the L1 trigger is performed by the CTP, and the absence of the L1 trigger within the time window corresponding to the L0–L1 latency is considered as a rejection of the triggered event.

Energy deposited in each PHOS cell by high-energy particles is detected by an APD Hamamatsu S8664-55 with a sensitive area of 5 × 5 mm². The APD gain is adjusted to the nominal value of 50 by setting the bias voltage with an accuracy of 10⁻³ in the range from 210 to 400 V. The APD signals are passed to charge-sensitive preamplifiers with an output signal proportional to the APD charge conversion. Dual-gain shapers with an average gain ratio of 16.7 generate semi-Gaussian signals with a rise time of 2.1 µs, which are further digitized by a 10-bit sampling ADC (ALTRO [71]) at a sampling rate of 10 MHz. The dynamic range of photons detected in PHOS spans from 5 MeV to 5 GeV in the high-gain channels and from 80 MeV to 80 GeV in the low-gain channels. The number of samples is configurable via ALTRO registers and is chosen to be 37 in order to cover the rising edge of the signal and its maximum. The sampled digitized waveform of a signal in one channel is shown in figure 84. One front-end board processes 64 signals generated by high-gain channels and low-gain channels from 32 PHOS cells. Data collection from ALTRO, FEE board configuration and data shipping to the readout units is provided by the Altera ACEX 1 K Family EP1k100 FPGA. The design of the PHOS front-end board is described in [77].

<figure><img src="image_94.png" /><figcaption>**Figure 84.** Digitized signal waveform from one channel of the PHOS FEE board. Sampling time is 100 ns, digitization of sampled amplitude is 10 bits.</figcaption></figure>

The PHOS generates triggers at the L0 and L1 levels to select events with high-energy deposition in the PHOS cells. The input to the trigger decision starts from the analog sum of the amplitudes of the group of 2 × 2 cells implemented in the FEE boards as “fast-OR” signals. Each FEE board produces 8 such “fast-OR” signals. The L0 trigger is produced by the Trigger Region Units (TRU) [74] covering an area of 16 × 28 cells. The TRU measures the energy deposits in a sliding window of 2 × 2 “fast-OR” channels, or 4 × 4 cells. If the energy in at least one window exceeds the
configurable threshold, the TRU generates the L0 trigger. The whole detector is inspected by 28 TRUs. All 28 L0 triggers generated by the TRUs are collected by the Summary Trigger Unit [78] (STU), which performs a logical OR operation of the inputs and generates the common L0 trigger if at least one TRU generated a trigger. The TRU boards deployed by PHOS are similar to those designed for the EMCal detector, with the only difference being the number of channels: PHOS TRU has 112 channels, whereas the EMCal one has 96 channels. While the STU boards are electronically identical for PHOS and EMCal, different firmware is used in the two cases.

FEE boards are read out by a point-to-point protocol via the designated DTC links (described earlier in the EMCal section) to the readout concentrator, the Scalable Readout Unit (SRU) [72]. One SRU can serve up to 40 front-end boards connected to its 40 DTC ports. The PHOS readout topology is closely related to the geometry of the PHOS modules, using 28 FEE boards and 2 TRU per SRU. The whole PHOS detector is read out by 14 SRUs. Triggering and synchronization of the SRU is provided by the TTC signal distributed by the ALICE central trigger. The TTC clock and trigger is propagated by the SRU to each FEE board or TRU via the DTC links. The SRU raises the busy signal upon receiving the L0 trigger via TTC and releases the busy signal when all FEE boards and TRU are read out. Data collected by the SRU are shipped to the FLP via the DDL link with a bandwidth of 2.125 Gb/s. All electronic modules (FEE, TRU, SRU) remain unchanged compared to their hardware state during Run 2. However, the upgrade concerned the SRU firmware which was adapted to comply with the new trigger protocol and was modified from the 3-level trigger sequence in Run 2 to the L0-L1 trigger sequence described above. Shipping data from SRU to FLP immediately after receiving the L1 trigger results in a significant reduction of the busy time, allowing an increase of the readout rate.

### 3.11.3 Performance

While the PHOS active detection elements, photodetectors and front-end electronics in Run 3 remain the same as they were during Run 2, the physics performance of the detector achieved during Run 2 remains valid for the upcoming Run 3.

The high light yield, short radiation length and small Molière radius of lead tungstate, enable high energy and spatial resolutions for the PHOS. The energy resolution measured in a beam test is parameterized by the equation [79]

$$ \frac{\sigma_E}{E} = \sqrt{\left(\frac{a}{E}\right)^2 + \frac{b^2}{E} + c^2} \qquad (3.1) $$

with $a = 0.013$ GeV, $b = 0.036$ GeV$^{1/2}$, $c = 0.011$ and the photon energy $E$ is expressed in GeV. Spatial resolution was evaluated in Monte Carlo simulations and indirectly confirmed by the mass resolution of $\pi^0$ mesons in data collected by PHOS during LHC Runs 1 and 2. The value of the noise parameter $a$ of the energy resolution (3.1) is determined by the APD intrinsic noise and design of the APD preamplifiers and FEE boards. The stochastic term $b$ is driven by the light yield of the PbWO$_4$ scintillator at the nominal working temperature and by the light collection efficiency defined by the surface area of the APD. Since the PHOS electronics did not change since Run 1 and 2, the energy resolution parametrized by (3.1) remains valid for Run 3.
The spatial resolution is parametrised as [80]

$$ \sigma_x = \sqrt{A^2 + \frac{B^2}{E}}, \qquad (3.2) $$

where parameters A and B depend on the photon incident angle; averaging over all angles yields A = 0.96 mm, B = 2.29 mm·GeV¹/². One of the key performance parameters of PHOS is the two-photon separation distance, i.e. the distance at which individual photons striking the PHOS surface can be identified. As discussed in [80], PHOS can distinguish photon showers split by at least one PHOS cell, δr = 2.2 cm. This feature is especially important in the high-multiplicity environment of heavy-ion collisions, as well as for resolving single photons from π⁰ decay at high pT.

The mass resolution for π⁰ and η mesons in pp collisions at √s = 13 TeV in Run 2 is discussed in [81]. The mass resolution is affected by large incident angles of photons at low transverse momenta and by the splitting procedure of overlapping showers at high pT. Photon showers from π⁰ decay start to overlap in PHOS at pT > 25 GeV/c, and shower splitting efficiency is reduced with the growth of the π⁰'s pT. At pT ≈ 50 GeV/c the most probable distance between decay photons from the π⁰ becomes to be one PHOS cell, then the photons cannot be resolved anymore. At such high pT, the reconstructed π⁰ mass is distorted by overlapping showers, and the mass resolution becomes rather large. The widths of the π⁰ and η mesons measured with PHOS in Run 3 will remain the same, σπ⁰ = 4.5 MeV/c² and ση = 15 MeV/c².

Photon identification in PHOS is based on charged-particle background rejection using matching between charged-particle tracks and clusters, as well as selections of cluster-shape parameters [80] to discriminate between electromagnetic and hadronic showers. Measurements of the arrival time could also be a strong criterion to identify fast neutral clusters produced by photons and slow clusters produced by heavy neutral hadrons such as neutrons and antineutrons. The intrinsic time resolution of PbWO₄ is rather good and can reach σt = 0.15 ns at photon energy E = 1 GeV. However, the front-end electronics deployed by PHOS is not designed for precise time measurement. Timing-resolution dependence on photon energy, measured using physics data collected by PHOS during Run 2, is shown in figure 85. At energies below a few hundred MeV, the time resolution σt rises above 10 ns. The best time resolution of 2 ns is achieved for an energy E ≈ 5 GeV, but then deteriorates because the low-gain channel is used for larger signals. The achieved time resolution is sufficient to separate photons of energies E > 1 GeV produced in different bunch crossings with 25-ns intervals.

As mentioned above, the PHOS performance will not change in Run 3. Hence, the PHOS energy and time resolutions and the related systematic uncertainties on photon and neutral meson measurements achieved in Run 2 will not improve. However, the ability to collect larger data samples with the new readout strategy will allow to improve the statistical uncertainties by a factor of two to five with respect to Run 2.

## 3.12 Zero-Degree Calorimeter

The aim of the ZDC upgrade is to cope with the high collision rate foreseen for Runs 3 and 4. Since the zero-degree calorimeters behaved well with the irradiations during the Run 1 and 2 operations, the calorimeter stacks are unchanged [1]. However, the infrastructure had to be consolidated and the readout system upgraded.
<figure><img src="image_95.png" /><figcaption>**Figure 85.** PHOS time resolution dependence on photon transverse momentum during the Run 2 data taking.</figcaption></figure>

Concerning the infrastructure, two actions were performed. Firstly, the control electronics of the movable platform was upgraded. This platform is used to move the ZDC calorimeters in a garage position where it is shielded from potential beam losses during beam injection or adjustment operations. Moreover, it allows to align them with the neutron (proton) average impact position during data taking. Secondly, additional power supplies for the voltage dividers of the ZDC photomultipliers were installed, in order to stabilize the gain in the high event rate conditions that are foreseen.

The main upgrade activity concerned a new readout system based on faster electronics. In fact the Run 1 and 2 readout electronics were based on VME charge-to-digital converters with a conversion time of ~ 10 µs that cannot sustain an event rate of 50 to 100 kHz without dead time (taking also into account a possible luminosity increase beyond the LS3 baseline). Moreover, in order to fully exploit the ALICE physics potential in ultra-peripheral heavy-ion collisions, the ZDC aims to take data in continuous (autotrigger) readout mode. This operating condition is particularly challenging since the acceptance of the ZDC not only covers nucleon emission from hadronic interactions but also the ones resulting from electromagnetic dissociation [82–84] that have ~ 50 times higher cross sections for Pb-Pb collisions at LHC energies. The designed Pb-Pb readout rate of 100 kHz will be accompanied by an additional ~ 5 MHz event rate, mostly uncorrelated among the two neutron ZDCs (ZNA on side “A” and ZNC on side “C” of the experiment, at positive and negative pseudorapidities, respectively), resulting from electromagnetic interactions that do not involve barrel detectors.

Because of the low number of channels to be instrumented, the new readout system is based on commercial digitizers, in particular ALICE will use VITA 57 FPGA Mezzanine Card (FMC) digitizers, that allow a continuous sampling of the signal waveform followed by a real time analysis on an FPGA. The adequate bandwidth available through the FMC connection from the digitizer to the FPGA allows for the full waveform to be analyzed. Fast trigger and selection algorithms are executed on the FPGA and the relevant portions of the waveform (see below) are transferred to the acquisition and reconstruction system via optical GBT links (see section 2.2).

To preserve the time and charge resolution and to match the bandwidth of the ZDC signals, the digitizers should have about 12 bit resolution (with an effective number of bits of ~ 10 bit) with a sampling frequency of 0.5 ÷ 1 GHz. Since the photomultiplier signal is unipolar the digitizer has to be DC coupled. After evaluating a few modules, the ADC3112 FMC [85] mounting digitizer ADS5409 [86] was chosen. Thanks to the shielded location of the readout electronics there is no
requirement of radiation hardness. The FMC is hosted on the carrier "Intelligent FPGA Controller" IFC_1211 [87] with a Kintex UltraScale XCKU40 FPGA. The ADC3112 on-board oscillator is locked to the LHC revolution frequency recovered from a GBT link and dispatched through the FMC connector. Since the ADC will acquire 24 samples per bunch crossing, it will run at a frequency of ~ 960 MHz. Internally each ADC channel is acquired by two digitizers which work in interleaved mode. In order to reduce the data size the low pass filtering with digital downsampling is enabled on the ADC. This has the benefit of improving the measurement accuracy by averaging over the even and odd samples removing the need to correct for the slightly different gains and offsets between the two circuits. The data throughput to the FPGA will therefore be reduced to 12 samples per bunch crossing at ~ 480 Msps, simplifying the firmware design.

A critical aspect of the ZDC operation in Run 3 is triggering at high rates in Pb-Pb with the bunch spacing reduced to 50 or 25 ns since the duration of the photomultiplier signals will be comparable or longer than the bunch spacing. This is complicated by the large signal dynamics (from 1 to ~ 60 neutrons in the acceptance of the neutron calorimeters). In order to identify the presence of a signal, a differential trigger algorithm was developed. Samples at different times are compared (sample $y_i$ with sample $y_{i+shift}$ where shift is a tunable parameter from 3 to 5 samples). If two consecutive differences are above threshold, the trigger condition is satisfied, effectively rejecting fake triggers due to electronic noise, and the bunch is flagged for acquisition. This autotrigger condition drives the acquisition in continuous readout mode while in triggered mode the readout system acquires data regardless of the autotrigger flag. The same flags are used also to measure the interaction rate that is used to estimate the instantaneous luminosity.

The measurements of signal arrival time and amplitude need to take into account the baseline (pedestal) oscillations and the possible presence of a signal in an earlier bunch crossing (pile-up).

Two methods for pedestal evaluation were implemented. Given the bunch structure of LHC that alternates "trains" of colliding bunches to "gaps" where no collisions can occur, it is possible to measure the pedestal considering portions of the digitized data where no collision can occur. These are prescribed by a filling map uploaded on the front-end at each fill. Using this information the pedestal average for each LHC orbit is computed and then transmitted on GBT. This allows taking into account a possible low frequency drift of the baseline and obtaining an accurate reference. A second method allows to effectively subtract the pedestal in presence of noise at higher frequencies. For each trigger (or autotrigger), in addition to the bunch where the signal peaks ($BC_0$), the 12 samples of the preceding bunch crossing will be transferred in order to evaluate and correctly subtract the pedestal in case of a significant discrepancy in the orbit average computed with the first method.

For what concerns the pile-up from a signal in an earlier bunch crossing, in autotrigger mode all ZDC signals are transmitted and reconstructed, allowing to identify and correct for pile-up. On the other hand, in triggered mode, the firmware ensures that the information on the signal inducing pile-up is not lost due to trigger selectivity. Consequently, for each triggered bunch crossing, up to four bunch crossings will be transferred: the triggered and the preceding one (pedestal evaluation) and additionally $BC_{-2}$ and $BC_{-3}$ in case a pile-up signal is detected.

During Pb-Pb data taking in 2018 a prototype of the ZDC system was tested in parallel to the ALICE data acquisition by using a custom system based on Labview reading the ADC_3112 mounted on a Xilinx evaluation board or using the IOxOS IFC_1210 carrier. An example of the achieved performances is shown in figure 86. The resolution on 2.76 TeV single neutron emission
<figure><img src="image_96.png" /><figcaption>**Figure 86.** Performance of the digitizer during Pb-Pb 2018 data taking in the operating conditions chosen for Run 3. On the left plot: the lower part of the triggered spectrum of ZNC common photomultiplier in Pb-Pb collisions where the emission of a single 2.76 TeV neutron and multiples are visible. The spectrum is fitted to a superposition of gaussian functions whose peak positions $\mu_i$ are related to the neutron multiplicity by the relation $\mu_i = \mu_{1n} \times i$ and their widths by the relation $\sigma_i = \sigma_{1n} \sqrt{i}$, where i is the neutron multiplicity and $\mu_{1n}$ and $\sigma_{1n}$ are the mean and the r.m.s. of the single neutron peak, respectively. The autotrigger algorithm effectively rejects pedestal events. On the right plot: the arrival time of ZNC common photomultiplier signals w.r.t. the reference ALICE L0 trigger signal.</figcaption></figure>

detected by ZNC is ~17%, resulting in an improvement w.r.t. the ~20% of the previous electronics. The time resolution w.r.t. the ALICE L0 trigger is ~0.35 ns, a value that is comparable with the performance of the previous system.

## 4 Mechanics and integration

The principal layout and infrastructure of the original ALICE detector is described in ref. [1]. During LS1 (2013 to 2014), the DCal detector had been added as an extension of the electromagnetic calorimeter on a 60 degree azimuthal acceptance opposite of the EMCal detector. For this purpose, new support rails and a new support structure holding the PHOS and the DCal modules were installed in the bottom part of the L3 magnet. These new support rails are also used for injecting 10000 m³/h of cold air into the L3 magnet volume for stabilising the air temperature around the ALICE detector.

For the LS2 upgrade, the global mechanical structures of ALICE remained unchanged. The most important modification was related to the support of the beam pipe and the ITS2 detector. In the original ALICE setup, the TPC had to be moved to the parking position in order to carry out maintenance of the ITS2 detector. This required the disconnection of about 30% of all ALICE services and would therefore only have been possible in a long shutdown of more than one year. In addition, the beam pipe, ITS2, and TPC were connected in a way that did not allow relative adjustment, so alignment of the beam pipe with the nominal LHC beamline required the adjustment of the TPC or even of the entire ALICE experiment. Such an operation had been carried out in 2008.
For the ALICE 2 detector, the support structures of the ITS2 and the beam pipe inside the TPC were therefore completely re-designed. The cage, a support structure made from carbon fiber material, was installed inside the TPC as shown in figure 87. The cage holds the beam pipe and has a rail system that allows the installation of the ITS2 and MFT detectors with the TPC in place. This makes it possible to perform maintenance of the ITS2 detector during a year end technical stop of about three months. In addition, it allows the alignment of the beam pipe with the nominal beamline within a range of ±4 mm without the need to move the TPC.

<figure><img src="image_97.png" /><figcaption>**Figure 87.** The cage: a support structure for beam pipe, ITS2, and MFT. Shown is the cage (magenta) with the bottom half of ITS and MFT as well as the beam pipe already installed.</figcaption></figure>

The new ALICE beam pipe has a central beryllium section with a length of 888 mm, an outer diameter of 36 mm, and wall thickness of 0.8 mm (figure 88).

<figure><img src="image_98.png" /><figcaption>**Figure 88.** Beam pipe installed for the ALICE 2 detector with an outer diameter of 36 mm.</figcaption></figure>
# 5 Readout and data processing

In this section, an overview of the readout concepts and the data flow is given. Subsequently, the individual systems for experiment and detector control, triggering, data acquisition, synchronous and asynchronous event reconstruction, and the processing of analysis object data are discussed.

## 5.1 Readout data flow

In order to minimise the costs and requirements for data processing and storage, the ALICE computing model for Runs 3 and 4 is designed for a maximum compression of the data volume read out from the detectors synchronously with data taking [88]. In order to compress the large data flow from the TPC, tracks are reconstructed online. Moreover, data for detector calibration are extracted during online processing avoiding additional offline calibration passes over the full data set. Online data processing is performed in two steps on the ALICE online/offline facility (O²) located at Point 2. The facility consists of two types of computing nodes: the First Level Processor (FLP) located in the experiment access shaft (CR1), and the Event Processing Nodes (EPN) in dedicated computing containers (CR0), see figure 89. The facility provides also the network for data distribution, large disk storage capacity as well as interfaces with the GRID and the permanent data store at the Tier 0 computing center.

The upgraded online system supports both continuous and triggered readout. Legacy subsystems not upgraded to continuous readout are not capable of reading the full event rate and thus require a hardware trigger signal. These detectors are therefore read out whenever they are not busy. Triggered readout for all detectors is also used for commissioning and calibration runs. Data produced by the detectors are transferred to the Common Readout Units (CRU) (see section 2.2) where they are compressed, multiplexed, and then transferred to the memory of the FLPs.

During the revolution period of the LHC (~88.92 µs = “LHC orbit”) there is an LHC filling scheme dependent number of bunch crossings (BC) at which collisions can occur. The ALICE data stream is divided into so called heartbeat frames (HBF) which have a duration of one LHC orbit and are synchronized with the LHC clock. A configurable number of HBFs form a time frame (TF), which represents the data container for data processing and replaces the traditional event entity. The nominal TF length is 128 orbits (~11.4 ms). At 50 kHz interaction rate, it contains on average 569 Pb-Pb collisions. Continuous and triggered data are tagged by HBF and BC identifiers.

The FLPs perform a first level of data compression to 900 GB/s by zero suppression. In addition, they have the possibility to perform calibration tasks based on local information from the part of the detector they serve. One example is the TPC for which a first calibration step is already performed on the CRU. The signals from the GEM readout detectors feature an ion tail and at high occupancy a common baseline shift, that is best removed as early as possible. A Sub Time Frame (STF) comprises all HBFs belonging to a TF from one FLP. After all FLPs have built their STFs of an individual TF, an available EPN is selected and all STFs are sent there and the full TF is built.

A dedicated FLP is used to collect and process data from the Detector Control System (DCS) in two workflows. The first one processes DCS data shipped via the ALICE datapoint server and stores detector conditions like voltages, temperature, and pressure in compact objects (see also section 5.6.3). The second one processes configuration files sent by detectors as well as LHC information. The calibration objects are stored in the condition and calibration database (CCDB)
and from there they are read by the following processing stages. Another dedicated FLP is used to collect all trigger signals sent by the CTP to the detectors.

<figure><img src="image_99.png" /><figcaption>**Figure 89.** Overview of the components of the O² data read out and processing systems and the main data flows.</figcaption></figure>

The EPN farm consists of 280 servers hosting 8 GPUs and 64 CPU cores each. The capacity has been dimensioned such that it can achieve a first reconstruction pass (referred to as synchronous reconstruction), extraction of calibration objects for subsequent asynchronous reconstruction passes, and data compression. The compressed data are aggregated into so-called compressed time frames (CTF) replacing the original raw data and written to a disk buffer at an output rate of about 130 GB/s. The disk buffer has a raw capacity of 150 PB and is managed by the EOS system [89]. The erasure coding configuration used for storage protection reduces the usable capacity to about 120 PB.

Calibration data from EPNs are aggregated on dedicated nodes, processed, and stored in the CCDB. CCDB objects are distributed back to the whole O² farm through multi-casting and migrated to the offline CCDB as well as to GRID storage elements, for usage by the ongoing synchronous reconstruction steps and for the later asynchronous processing and simulation, respectively.

The CTFs are transferred to the GRID for archiving. After data taking and full detector calibration, two or more asynchronous reconstruction passes are performed on the GRID as well as on the EPN farm. The output of these reconstruction passes is stored as Analysis Object Data (AOD), the input for physics analysis. For specific physics signals, a further data size reduction and speed-up of the corresponding analyses is achieved by filtering out events of interest and writing out only the minimum event information needed. The processing of pp data follows the same chain with an additional step of selection of interesting collisions during an asynchronous reconstruction pass and reduction of the CTFs by keeping only the data associated to these collisions. Reconstruction passes are followed by Monte Carlo production cycles taking into account the time dependent detector conditions.

Besides the computing infrastructure, a common software framework has been developed within which all online and offline components are operated [90]. It consists of three main layers. The Transport Layer has been developed in collaboration with GSI (FAIR) and it uses the FairMQ message passing toolkit [91] with FairMQDevices as its main building blocks. It enables efficient parallelism
by providing abstraction of network and inter-process communication as well as by supporting
shared memory backed message passing for devices on the same node. The data model provides
language agnostic and extensible descriptions of messages that are passed between devices [92]. It
provides support for various back-ends such as a so called zero-copy format (a format that optimises
performance by allowing to efficiently map files or portions thereof to memory and to share buffers
between processes), serialisation based on the ROOT data analysis framework [93], and Apache
Arrow [94] for analysis and integration with external tools. Finally, the Data Processing Layer (DPL)
abstracts computation as a set of data processors organized in a logical data flow specifying how data
are transformed. Depending on the deployment environment, the data flow is mapped to a concrete
topology and from there to a set of processes running FairMQ devices.

<figure><img src="image_100.png" /><figcaption>**Figure 90.** Synchronous reconstruction workflow.</figcaption></figure>

5.1.1 Synchronous reconstruction

A schematic representation of the synchronous reconstruction workflow is shown in figure 90. The main objectives of the synchronous processing are the reduction of the data rate from the TPC, which accounts for most of the raw data volume and the extraction of data for calibration. This is achieved by performing clustering and full track reconstruction in the TPC and removing background hits from the data. Moreover, cluster space point coordinates are stored as relative coordinates, thus reducing the entropy and allowing for efficient ANS entropy encoding [95] of the data. The TPC space charge distortion calibration uses the information of fully reconstructed barrel tracks including ITS, TOF, and TRD information. However, only a small fraction of all tracks needs to be fully
reconstructed to gain sufficient data. Hence, full TPC reconstruction needed for data compression is the most demanding step in terms of computing time. The online processing makes extensive use of Graphic Processing Units (GPU), which provide a significant speed-up by about a factor of 50 [96] compared to one CPU core in an EPN server, without compromising the physics performance.

The TPC reconstruction code has been developed starting from the existing Run 2 High Level Trigger (HLT) algorithms. It starts with the cluster finding and is followed by tracking comprising the track finding, track merging, fitting, and compression steps. The presence of Space Charge Distortions (SCD) of up to 10 cm represents a particular challenge for the reconstruction of continuous data. In absence of triggers, which provide reference for the drift time estimate, the z-positions of clusters are unknown. However, this information is needed for z-dependent quantities used during track reconstruction: the corrections of the SCDs, the magnetic field strength, and the cluster error parameterisation. Therefore, TPC tracking is first performed without these corrections. Since the distortion effects are smooth, the track finding is not strongly affected. Track seeds are extrapolated to the beam line and the most probable z-coordinate is calculated under the assumption that the track is from a primary particle and the vertex is at the interaction point. If the track turns out to be a secondary, an average pseudorapidity is assumed. The track is refitted with the corresponding corrections. The average SCD corrections require a first order correction map obtained from simulation or a previous reference run which is scaled by the instantaneous luminosity. In addition, the 1D integrated digital currents containing information about the fluctuations of the number of ion pile-up events and of the track multiplicity are used to achieve a partial correction of SCD fluctuations. During synchronous reconstruction, cluster positions can be corrected with a precision of $O(\text{mm})$ which is sufficient for correct cluster associations to tracks. The full correction with a precision of $O(100 \mu\text{m})$ will be performed during asynchronous reconstruction.

Two options for TPC data compression are supported by the software. In the first option (A) clusters from background (for example from noisy pads or charge clouds related to low momentum protons) and clusters that are associated to or in the proximity of background tracks are rejected. Background tracks include those from very low momentum particles spiralling around the magnetic field lines, track segments with large inclination with respect to the TPC pad rows, and clusters from secondary legs of looping low-momentum tracks used for physics. However, clusters in a tube around good tracks are protected. For the second option (B), only clusters that are attached to, or in the proximity of identified good tracks that may be used for physics analysis are kept. The estimated rejection fractions for options A and B are 12.5 – 39.1% and 37 – 53%, respectively. While option B yields lower data size it bears the risk that in case the SCD corrections are not precise enough track merging and partially also track following might lose good tracks or parts thereof. Optimal performance of option A requires identification of hits from particles with momenta below 10 MeV/c, since they contribute about 15% of all TPC hits. Tracking in this momentum region is challenging and is currently under development.

Further data size compression is achieved by converting the cluster properties from the single-precision floating point format used in reconstruction to custom integer and floating point formats with exactly as many bits as needed for the intrinsic TPC resolution. The entropy is reduced before the ANS encoding for further data compression. This includes the following steps. Coordinates of hits that are not assigned to tracks are sorted by geometrical coordinates and the difference to the previous hit is stored. Raw coordinates (row, pad, time) of hits assigned to tracks are stored relative
to the extrapolated track (Track Model Compression). Cluster properties, maximum charge, total charge, and cluster size, are encoded together in order to profit from their correlation.

Synchronous data processing of the remaining detectors is performed on CPU cores in parallel with the GPU processing. For the ITS and the muon spectrometer system (MFT, MCH, MID), processing starts with space point reconstruction (clustering). For the barrel calorimeters EMCal, DCal, and PHOS, the cell properties (time, amplitude) are determined by fitting the raw time distributions. Clusterization is performed in order to select cells to write to the CTF, while final clustering is performed during analysis. Data for time calibration and dead-channel maps are extracted. For FT0, the reconstruction of collision times is performed for the needs of barrel global tracking and vertexing. FT0, FV0, and FDD digits converted from raw data are stored in the CTF.

For a subsample of tracks selected from peripheral collisions (about 1% of all tracks), full tracking including all barrel detectors is performed, i.e. ITS tracking after clustering, matching of ITS tracks to TPC tracks, and finally track matching to TRD and TOF. As in Run 2, residuals between global tracks and TPC clusters are used to create 3-dimensional space charge distortion maps with a granularity of 1-2 minutes when running with Pb beams and 10 minutes in pp collisions. These maps together with the TPC integrated digital currents recorded during synchronous processing become part of the calibration used in asynchronous processing.

Global barrel tracks are also used to obtain fast TPC drift time and TRD calibration (gain, $t_0$, $E \times B$, and drift velocity). Moreover, the drift of the LHC clock with time (due to temperature changes that impact the fiber refractive index and the distribution of the LHC clock time to the experiments), which affects the reference for the time of flight measurement as a global offset, is calibrated using global tracks matched to TOF. At the same time, the TOF channel-level offset in the measured times related to the cable lengths and electronics is determined. In addition, other calibration algorithms are running during online reconstruction, particularly, for the determination of the interaction region, calorimeter bad channels, and gain parameters. The general way to perform these online calibrations is to extract for every TF compact data related to the parameters being calibrated and send them to dedicated aggregator servers. The workflows running on these servers attribute the incoming calibration data to time slots with a granularity characteristic for each calibration type and automatically create a CCDB object for every slot once they have accumulated enough data for processing. During synchronous processing, also input data are accumulated for those calibration constants that need a large amount of data or are too demanding to be determined synchronously. One example is the TOF channel time slewing. The corresponding calibration information is extracted before the asynchronous reconstruction takes place, and the CCDB is updated.

The final processing step consists in compressing all data stored in the CTF using the rANS algorithm, a variant of Asymmetric Numeral System coders, which allows to reach the entropy limit [95, 97].

## 5.2 First-Level Processors

The O²/FLP subsystem includes the First-Level Processors (FLPs) detector readout farm, the data quality control system, and the services for control, configuration, monitoring, logging, and bookkeeping.
### 5.2.1 The FLP detector readout farm

The readout farm consists of 199 nodes, with 488 readout cards (472 new CRUs and 16 legacy C-RORC) to transfer the data from each detector to the $O^2$ system. The number of FLP nodes and readout cards associated with each detector is given in table 10. The total nominal readout bandwidth amounts to 3.5 TB/s from the detector electronics to the readout cards where it is compressed before the transfer to the memory of the FLP servers. Most of the detectors use the GBT link and the CRU (see section 2.2) adopted for this upgrade. The system is also backward compatible with the Detector Data Link (DDL) [61] and the Common ReadOut Receiver Card (C-RORC) [6] used during the LHC Runs 1 and 2.

<table><caption>**Table 10.** FLP readout farm used to transfer the data from the detectors to the $O^2$ system.</caption><thead><tr><th rowspan="2">Detector</th><th rowspan="2">Link type</th><th colspan="2">Readout links</th><th colspan="2">Readout boards</th><th>Readout nodes</th></tr><tr><th>DDL</th><th>GBT</th><th>C-RORC</th><th>CRU</th><th>FLPs</th></tr></thead><tbody><tr><td>CPV</td><td>GBT</td><td></td><td>16</td><td></td><td>1</td><td>1</td></tr><tr><td>CTP</td><td>GBT</td><td></td><td>14</td><td></td><td>1</td><td>1</td></tr><tr><td>EMC</td><td>DDL</td><td>40</td><td></td><td>8</td><td></td><td>2</td></tr><tr><td>FIT</td><td>GBT</td><td></td><td>34</td><td></td><td>3</td><td>3</td></tr><tr><td>HMP</td><td>DDL</td><td>14</td><td></td><td>4</td><td></td><td>2</td></tr><tr><td>ITS</td><td>GBT</td><td></td><td>432</td><td></td><td>22</td><td>11</td></tr><tr><td>MCH</td><td>GBT</td><td></td><td>550</td><td></td><td>30</td><td>11</td></tr><tr><td>MFT</td><td>GBT</td><td></td><td>304</td><td></td><td>11</td><td>5</td></tr><tr><td>MID</td><td>GBT</td><td></td><td>32</td><td></td><td>2</td><td>1</td></tr><tr><td>PHS</td><td>DDL</td><td>16</td><td></td><td>4</td><td></td><td>2</td></tr><tr><td>TOF</td><td>GBT</td><td></td><td>72</td><td></td><td>4</td><td>2</td></tr><tr><td>TPC</td><td>GBT</td><td></td><td>5832</td><td></td><td>361</td><td>145</td></tr><tr><td>TRD</td><td>Custom</td><td></td><td>1044</td><td></td><td>36</td><td>12</td></tr><tr><td>ZDC</td><td>GBT</td><td></td><td>1</td><td></td><td>1</td><td>1</td></tr></tbody><tfoot><tr><td>Total</td><td></td><td>69</td><td>9291</td><td>16</td><td>472</td><td>199</td></tr></tfoot></table>

The server selected for the FLPs is the Dell Poweredge R740. The selection has been done after numerous hardware and software tests [98] and a competitive tender. Each FLP is equipped with 96 GB of DDR memory and two CPUs. The CPUs are of two different flavours of the Intel Cascade Lake generation (the Silver 4210 or the Gold 6230 with 10 or 20 hardware cores, respectively) depending on the processing needs of the detector. Each FLP hosts up to three CRUs, up to four C-RORCs, and one Infiniband network interface, each using one PCIe Gen3 x16 slot. The readout software performance allows data to be transferred from three CRUs simultaneously to the FLP memory for a total input throughput of 330 Gb/s corresponding to 85% of the maximum PCIe Gen3 bandwidth. The maximum output bandwidth available to the Infiniband network is 100 Gb/s.

The first layer on top of the PCIe interface of the cards is the PDA (Portable Driver Architecture) UIO (Userspace IO) kernel module [99]. PDA also provides a user space library in C [100] which supports PCIe device enumeration and provides a handle to PCI devices. The readout software includes the readout program and the readoutCard library [101] which orchestrate the simultaneous
data transfers from the GBT links to the FLP memory as shown in figure 91. The transfer of data to the EPN farm is handled by the $O^2$ data distribution (see section 5.3).

<figure><img src="image_101.png" /><figcaption>**Figure 91.** Simultaneous dataflows inside the FLP from the CRUs to the DDR memories and from the memory to the Infiniband network to the EPN farm.</figcaption></figure>

### 5.2.2 Data quality control

The online execution of the calibration and the reconstruction and the replacement of the raw input data by compressed data make reliable data Quality Control (QC) mandatory. Its main purposes are to quickly identify and overcome problems during data taking and to provide good quality data for physics analyses. It is also crucial to ensure that the data processing behaves as expected, especially when running synchronously with the data taking.

The $O^2$ QC system [102] includes a distributed software framework as shown in figure 92.

Data samples are selected following a pseudo-random sampling and configurable policies at key points in the dataflow and are dispatched to local (on the FLPs and the EPNs) or remote (on QC servers) QC tasks executing detector-specific algorithms. Their results are published as QC objects, for example hit distributions in sub-detectors, which are typically represented as ROOT [103] histograms. The results of the QC tasks running in parallel on many nodes are assembled by the mergers. Checkers evaluate the quality of the objects, resulting in QC qualities, that summarise e.g. whether the hit distributions are good or bad. The QC qualities can optionally be aggregated and are stored together with the QC objects in the QC repository. This database has reused the software developed for the CCDB of ALICE $O^2$. The post-processing component encompasses asynchronous tasks such as correlation and trending of data derived from QC objects and qualities. It is triggered periodically, manually, or on certain events (e.g. start of run or end of fill). The Machine Learning component will be a particular type of post-processing. QC and quality objects are accessible to shifters and experts through a web-based QC GUI.
<figure><img src="image_102.png" /><figcaption>**Figure 92.** $O^2$ Quality Control design.</figcaption></figure>

### 5.2.3 Services

**Web User Interface framework** The Web User Interface (Web UI) framework provides the core functionalities and building blocks to easily create rich web applications. The server side features REST and WebSocket API, authentication via CERN Single Sign-On and authorisation using CERN e-groups. The client-side features Cascading Style Sheet building blocks for the user interface, asynchronous data fetching (Ajax), and bidirectional sockets (WebSockets). Several $O^2$/FLP GUIs are based on this web interface: AliECS, InfoLogger, QC, and Bookkeeping.

**Control and configuration.** The ALICE Experiment Control System (AliECS) [104] integrates the experiment control and configuration, the FLP farm control, and a high-level control interface to the $O^2$/EPN cluster. It implements a distributed state machine to represent the aggregated state of the constituent $O^2$ processes of a data-driven workflow. Furthermore, it allows reconfiguration of running processes and simultaneous operation of multiple workflows, with easy reallocation of resources among workflows. Finally, it reacts to inputs, handling events from the user, the LHC, the trigger system, the DCS, and the FLP cluster itself with a high degree of autonomy.

Figure 93 shows the architecture of the system. The AliECS core is the control scheduler implementing the distributed state machine communicating over the google Remote Procedure Call (gRPC) protocol with the operator using the GUI and other interactive applications based on the AliECS Command Line Interfaces (CLI). The AliECS also uses a variety of communication protocols for the exchanges with other systems: DIP with the LHC, gRPC with the trigger system, and DIM for the communication with the DCS. Apache Mesos [105] is used by AliECS as cluster resource management system for the management of $O^2$/FLP components, resources, and tasks inside the $O^2$/FLP facility, effectively enabling the developer to program against the datacenter (i.e., the $O^2$/FLP facility at LHC Point 2) as if it was a single pool of resources. AliECS supports two $O^2$ Configuration and Control (OCC) interfaces to Mesos agents: either through the OCC library or through an OCC plugin for all the processes based on FairMQ, part of ALFA [106], which is the common $O^2$ transport layer for physics data.
AliECS interfaces with Consul [107], a key-value store which acts as the configuration repository of the system. Once acquired by the AliECS core, configuration information is processed into an in-memory hierarchical key-value store, and from there it is fed into a template system in order to generate task deployment and configuration structures.

Most components of AliECS are written in Go [108], a statically typed general purpose programming language in the tradition of C, which is particularly suitable for distributed system development because of its advanced synchronization and threading facilities.

<figure><img src="image_103.png" /><figcaption>**Figure 93.** AliECS design.</figcaption></figure>

**Monitoring.** The monitoring subsystem [109, 110] provides a complete overview of the overall system health and detects performance degradation and component failures by collecting, processing, storing, and visualising values from hardware and software sensors and probes. As presented in figure 94, metrics are sent to the system from both Telegraf [111] (for system metrics) and the C++ monitoring library (via Telegraf, for application metrics). These metrics are processed in an Apache Kafka [112] cluster and later written to an InfluxDB [113] time-series database for permanent storage.

The InfluxDB time-series database supports downsampling, which decreases the value resolution over time reducing the total database size. It is planned to keep high resolution metrics for several days. After that, time metrics will be downsampled in order to decrease the number of points and store them until the end of the calendar year.

The system includes a data visualisation interface based on Grafana [114] and channels for alarms and reporting.

**Logging.** The logging system has been adapted from the ALICE Run 2 DAQ software [115]. A new web-based user interface has been developed in addition to the existing GUIs.

**Bookkeeping.** A new bookkeeping system [116] has been developed. It unifies two functionalities: gathering, storing, and presenting metadata associated with the operations of the ALICE detector and tracking the asynchronous processing of the physics data. The front-end and back-end are based
<figure><img src="image_104.png" /><figcaption>**Figure 94.** The O² computing system monitoring design.</figcaption></figure>

on the WebUI framework like the other applications and are adaptive to various clients such as tablets, mobile devices, and other screens. The back-end includes a relational database and a REST API specified in the OpenAPI standard that allows to easily build bindings in various languages (as C++ and Go).

### 5.2.4 Installation and commissioning

The O²/FLP system has replaced the former DAQ system used during the LHC Run 1 and 2 in the Counting Room 1 (CR1) located in the access shaft of the ALICE experimental cavern at the LHC Point 2. All the optical fibers transferring the data from the detectors to the CRUs have been installed in four campaigns: February–June 2019, October–November 2019, February–March 2020, and November–December 2020.

The FLP specification was reviewed in April and May 2019 and the purchase order made in August 2019. The FLPs were delivered in several batches from September to November 2019. The FLPs have then been prepared to house the CRUs which required a mechanical modification of the chassis from September 2019 to February 2020. The connection of fibers to the CRUs (readout and trigger) was performed from April to August 2020 and the cabling to the network from July to September 2020.

The FLP software has been released as a coherent set of packages monthly since July 2019 and weekly from August 2021.

The test of the FLP system with detector electronics started first in the laboratories in June 2018. Its commissioning with (large pieces of) individual detectors started on the surface in June 2019, and in the experimental cavern at the LHC Point 2 in March 2020. The global tests with several detectors began in July 2021 and the first realistic experience with beams was collected during the LHC pilot beam in October 2021.

## 5.3 Event Processing Nodes

The EPN farm is designed to perform a first online data reconstruction pass, extract detector calibration objects, and reduce the data volume in order to fit into the available storage buffer space of about 80 PB. The compression algorithms rely on data reconstruction properties, and the subsequent asynchronous reconstruction passes rely on the calibration objects. During periods when data is not being collected, the EPN farm, in addition to other resources, will be used for the asynchronous
reprocessing of the data and will contribute computing resources to the physics analysis of the
previously recorded data.

5.3.1 EPN farm

Due to the increased Run 3 computing needs and the resulting space and cooling requirements [88], a new data centre for the EPN farm was built on the surface at Point 2 of the LHC, close to the ALICE detector. The new EPN data centre shown in figure 95 consists of four modules for standard Information Technology (IT) equipment and one infrastructure module. Each IT module has a cooling capacity of 525 kW and allows for power densities of up to 1 kW computing load per rack height unit (4.5 cm). This design allows the use of highly integrated servers, with the maximum number of supported GPUs per server.

<figure><img src="image_105.png" /><figcaption>**Figure 95.** The ALICE CR0 data centre which houses the EPN farm.</figcaption></figure>

During data taking, the EPN farm will receive up to ~ 900 GB/s from the FLP farm. This data rate will then need to be reduced to ~ 130 GB/s in real time, in order to write the data to the final storage space, see figure 89. The EPN farm is connected to the FLPs via a fast InfiniBand HDR network with a total throughput of 14.4 Tbit/s. Connectivity to the disk buffer in the CERN IT data centre is realized via Ethernet with 100 Gbit/s link speed and a total bandwidth of 2.4 Tbit/s, in a high availability setup.

The EPN farm consists of 280 EPN servers, which provide the necessary computing power for
the synchronous Run 3 data processing, required by the O² software. The servers were dimensioned
benchmarking the compute performance with simulated data via the O² software. This determined
the required CPU cores, number of GPUs, and size of the memory per server. In the current state
of the computing and software infrastructure, 230 EPNs are needed to process the 50 kHz Pb-Pb
data, replaying simulated data. A 4U Supermicro GPU server was chosen for its capability to house
eight double-width GPUs as well as an InfiniBand HDR host adapter. The servers are equipped
with two 32 core AMD Rome CPUs (PSE-ROM7452-0057) with 512 GB DDR4-3200 RAM, eight
AMD MI50 GPUs with 32 GB Memory, a 1 TB NVMe disk, along with an InfiniBand high data rate (HDR) host channel adaptor (HCA) operated at 100 Gbit/s.

### 5.3.2 EPN installation

The first containers for the data centre were delivered at the end of September 2018, the last two containers at the end of July 2019. Extensive load tests were performed to commission the control of the cooling system, before installing the IT equipment. The first usage of the data centre was via a test system using old Run 2 servers, called vertical slice, in October 2019. The first batch of the final network was installed and enabled the commissioning of the infrastructure. The vertical slice commissioning allowed the testing of the complete chain, from FLP to EPN to the EOS Open Storage system hosted in the computing center at CERN, in a reduced capacity. In June 2020, the final network for the EPN farm was installed in preparation for the arrival of the EPN servers, and following the Production Readiness Review for the EPN servers in August 2020, the order was prepared. The installation of the servers into the final rack positions was performed in January 2021. Another round of tuning for the cooling system was done with the production servers at the beginning of 2021, to optimize settings to the final Run 3 system. The EPNs were used for commissioning at Point 2 since beginning of 2021. The final part of the network, the gateways from the InfiniBand network to the Ethernet network of the storage facility, was finalised by June 2022 due to delays in the availability of the gateways and issues with operating multiple gateways in a high-availability cluster.

### 5.3.3 $O^2$ data distribution

The data flow in the $O^2$ system starts with the CRU performing direct memory access (DMA) transfers of the detector data into the memory of the FLP. The CRU DMA region is mapped as a shared memory region, which allows efficient intra-node communication between readout, data distribution, and local synchronous processing tasks. The zero-copy network transfers are implemented using remote direct memory access (RDMA) protocols of the InfiniBand network interface. Detector data corresponding to the configured number of HeartBeat Frames (HBF), nominal 128 and up to 256, are aggregated from all FLPs on a single EPN node, forming the Time Frame (TF), which is the input for synchronous reconstruction.

**$O^2$ data distribution network.** By far the largest data bandwidth requirement in the $O^2$ network comes from the readout data stream from FLPs to EPNs. As the data moves from all FLPs to a single EPN node, as mandated by synchronous processing, the flow must be actively regulated to avoid any points of congestion.

The network diagram of the entire $O^2$ facility is shown in figure 96. To satisfy individual FLP data rates, which vary from FLP to FLP depending on, e.g. the connected subdetector and number of installed CRUs, a 100 Gb/s InfiniBand network interface was chosen. The same interface is used for EPN nodes. The architecture of the InfiniBand network is implemented using a two-level folded-Clos topology network, often referred to as a fat tree. The network is built using 40 port switches for both core and top-of-the-rack (ToR) level switches. Two nodes interface at 100 Gb/s using a single switch port utilizing copper splitter cables. The core of the network is implemented using fiber optic cables operating at 200 Gb/s. Following the requirements, the network fabric features non-blocking communication from FLPs to EPNs, but implements a high blocking ratio within individual EPN and
<figure><img src="image_106.png" /><figcaption>Figure 96. Network diagram of the Run 3 $O^2$ facility.</figcaption></figure>

FLP sub-segments, where the bandwidth is not required. Additionally, three InfiniBand to Ethernet gateways, with 8 links at 100 Gb/s each, provide the required throughput for mass storage and other services for offline processing.

**$O^2$ data distribution software.** The ALICE detector implements a readout scheme with both continuous and triggered readout where the full online reconstruction is performed during the data acquisition. The data stream from the detectors is grouped into time frames, where one particular TF is processed on a single EPN. The detector data arrive into the cluster of 199 input nodes (FLP).

Given the large number of FLPs, such a scheme has to implement a deep pipeline, where in the worst case scenario the number of TFs in-building approaches the number of FLP senders. However, as the transfers use RDMA read primitives, the EPNs are able to pull Sub-Time Frames (STFs) close to the line rate of network interfaces, without creating congestion in the network or the receivers. This allows for optimal transfers of the STFs and reduces the TF building pipeline. The data rates across FLPs from different detectors range from 10 kB/s up to 8 GB/s, where the size of STFs depends on number of HBFs in a STF and can fluctuate in presence of changing conditions or equipment faults. Therefore, the size of time slices can vary strongly in time and the processing time of time slices is also quite variable. Therefore, the data distribution scheduling framework has to take such fluctuations into account. There is a trade-off between the implementation of larger buffers on the processing nodes, allowing to average some of the fluctuations, and the overall latency of the time-slice processing. Further, the system has to be stable against failures, such as the crash of a processing job or failure of a processing node. In such cases, the data loss is localized only to the TFs currently being processed by the affected process or node. The data distribution system is designed to accommodate changing processing requirements by allowing the addition or removal of EPNs into the ongoing data taking run. This enables the use of processing nodes for offline processing during times when there is a low load on the online system.

The high-bandwidth many-to-one data flow that is needed to assemble full TFs on the EPNs is managed by a data distribution scheduling framework. Figure 97 shows the main components of the developed data distribution system: sub-time-frame builder (StfBuilder) and sender (StfSender) on FLP nodes, time frame builder (TfBuilder) on EPN nodes, and time frame scheduler (TfScheduler) on an EPN infrastructure node which orchestrates the data distribution components and regulates the
<figure><img src="image_107.png" /><figcaption>Figure 97. Data distribution software framework.</figcaption></figure>

data flow. Detector data (HBFs) are transmitted to the StfBuilder, which publishes STF objects for the local processing and the quality control. The StfSenders report the availability and information about STFs to the scheduler, which selects a suitable EPN node where the TF can be built, once all STF components are available. Therefore, the scheduler keeps track of the utilization of all EPNs and buffer states of sending nodes. As several TFs are being built at the same time, due to the many-to-one traffic nature of TF aggregation, the scheduler keeps track of network fabric utilization to avoid creating congestion hotspots. Once TfBuilders are given information about all the STFs, they fetch the data from FLPs using remote DMA (RDMA), decreasing the CPU utilization on both the sender and receiver side. In the case when the data cannot be scheduled to any EPN, the StfSenders are instructed to drop the data in order to avoid creating back pressure. The dropping of complete TFs will occur only when there is not enough processing power available, e.g. insufficient number of EPNs in the partition, or in the presence of failures, e.g. frontend or readout misconfiguration or network issues.

The TF scheduling task keeps track of the utilization of TF destination buffers on EPNs and utilization of the shared memory (readout) buffer on FLPs. In case EPN or FLP buffer utilization is reaching the high watermark (configurable, and typically 32 GiB for FLP and 112 GiB for EPN), the TF scheduler throttles the transfers. This ensures the network transfers are not propagating back pressure to the FLP readout processes. Overflowing FLP buffers could be a result of misconfiguration of the readout card or frontend, where an FLP would generate more than 100 Gb/s of raw data or an unstable network link resulting in reduced network bandwidth. Sufficient EPN buffers might not be available if the number of allocated EPNs is not sufficient to process the incoming TFs. In the nominal case, when there is no back pressure in the whole data flow and processing chain, the TF scheduler assigns an EPN for each individual TF, maintaining even EPN and network utilization.

## 5.4 Physics data processing

### 5.4.1 Asynchronous reconstruction

Pb-Pb and pp data taking with synchronous reconstruction is followed by an about four to six week period during which the final calibration constants are evaluated. For some detectors, this requires also a short calibration pass over CTF data before the full asynchronous production passes can start.
The final calibration is performed during reconstruction passes, in particular targeting full correction of TPC space-charge distortions and nominal resolution. At this stage, all detectors are included in the reconstruction. The TPC tracks are matched to ITS tracks and propagated to the outer detectors. The global tracks are established by combining information from multiple detectors, and improved track fits are performed. Primary vertices are reconstructed and secondary vertices are identified in order to reconstruct V0 and cascade candidates. For long-lived particles decaying at large radius and producing TPC tracks unconstrained by other detectors, continuous readout poses an additional challenge. Every pair of unconstrained TPC tracks needs to be tested for multiple hypotheses of V0s from different primary vertices compatible with the allowed time (or z) range of the tracks. Since the TPC track corrections depend on their z position, this may even require on-the-fly re-calibration and refit of TPC tracks. In a final step, the particle identification hypothesis is assigned, based on combined information from all detectors. For the muon spectrometer system, stand-alone tracking is performed for MFT and MCH, followed by matching of MFT-MCH track-segments and track selection to form global muon tracks. At least two full passes of asynchronous reconstruction are planned to achieve the full performance.

For pp data at full energy, the first reconstruction pass includes an event selection procedure in order to reduce the overall data size. In addition to physics events of interest, such as heavy-flavor, high-multiplicity, or diffractive events, events needed for the TPC distortion calibration are selected. The CTF size is reduced by only keeping the clusters associated to tracks that point to the primary vertex of a selected collision within ±30 cm in z. The goal is an event rejection factor of 1000 leading to a CTF reduction to 1.2% of the original size. Data from reference runs with pp collisions at the same centre-of-mass energy as the Pb-Pb data taking will not be pre-selected, but fully transfered to mass storage.

When the EPN farm is not (fully) used for synchronous processing, e.g. outside data taking periods, it will be used for asynchronous reconstruction. During asynchronous reconstruction, the number of processing steps is larger than in synchronous reconstruction and without further code running on the GPU processing is CPU bound. To make optimal use of the GPU resources on the EPN farm also during asynchronous reconstruction, ALICE aims to offload more processing steps onto the GPU with the final goal to run the complete barrel tracking on GPUs. The reconstruction code is written using generic C++ code and can run on different GPU hardware. This opens also the possibility to run reconstruction efficiently on heterogeneous computing platforms that become available on the GRID.

Reconstruction passes are followed by Monte Carlo (MC) simulation productions. Physics analysis will be performed using GRID computing resources and on dedicated analysis facilities, using AODs from collision data and from MC productions as input and produces additional physics objects like fully reconstructed charmed hadrons and jets.

### 5.4.2 Simulation

Physics simulation comprises primary event simulation, the transport of particles through the detector geometry, detector response simulation, and digitisation of the detector signals. The O² software framework for simulation [117] has been developed within the ALFA project, an ALICE/FAIR collaborative effort based on common components such as FairRoot [118] and FairMQ [91].
GEANT4 [119] is employed as the main transport engine. As for AliRoot in Runs 1 and 2 [120], the $O^2$ simulation framework uses external transport codes through the Virtual Monte Carlo (VMC) layer [121]. This allows also the use of GEANT3 [43] and FLUKA [122] with the same user code, for example for studies of systematic uncertainties or radiation calculations. The detector geometry is described using ROOT/TGeo and the detector response using the VMC API and callbacks. As part of new developments for $O^2$ the VMC interface has been extended to interfacing fast detector simulation components which can replace detailed simulation in parts of the detector or for certain particle types [123]. Preserving the VMC interface has allowed efficient porting of detector code from AliRoot into $O^2$.

The $O^2$ simulation framework has been developed with two main objectives in mind: the possibility to leverage opportunistic resources, in particular High Performance Computing (HPC) facilities, which frequently offer only very short processing time windows, and performance optimization through parallelism on top of the capability of individual parts, going beyond standard event multi-threading of GEANT4. To this end, the simulation process is broken up into individual subprocesses: primary particle generation (event server), detector simulation, and I/O processes. These subprocesses run in parallel and interact with each other via sending and receiving messages. Parallelism is improved by further dividing the event simulation task into the processing of sub-events. Multiple independent detector simulation worker devices are instantiated at the same time. Each of them asks the event server for work chunks to process, where a chunk is either a full event or a sub-event. Hence, the system is able to process multiple events in parallel or collaborate on the simulation of a single event concurrently. A strategy based on so called late forking makes optimal use of common memory between the different processes. Processing speed-up as a function of simulation workers shows ideal strong scaling up to the physical number of cores. By reducing the processing time for a unit of work, the framework naturally supports the usage of opportunistic resources providing short processing time windows. In addition, there is the possibility to combine the results of various smaller transport simulations during digitization, so that a large and costly timeframe simulation can be split across multiple smaller jobs if necessary.

The output of detector response simulations are hits typically containing space-point and energy loss information of particles passing sensitive detectors. They serve as the input to the digitization step in which also the Time Frames are created. Since at the peak Pb-Pb luminosity on average five events can overlap within the drift-time of the TPC, a simulated time frame cannot be assembled from independently digitized events. Hence, the digitisation workflow takes into account the contributions from different collisions to the same digits.

Since the full simulation of Pb-Pb collisions is very time consuming, an optimized simulation strategy, named embedding, has been developed for AliRoot and used in production during Run 2. The background events are reused multiple times and overlaid with rare signal events Owing to the overlapping features mentioned above, the $O^2$ simulation framework supports this strategy naturally. The maximum time gain by embedding is limited by the time spent in digitization, in particular for the TPC. For this reason, effort has been put into reducing digitization time to a minimum. The fraction of digitization time of the total simulation time is ≈ 10% without embedding and reaches 40% with embedding.

In addition to strategies such as embedding, an efficient MC workload execution engine based on a directed-acyclic graph scheduler was developed. This engine performs dynamic scheduling
of tasks in the MC processing chain with the goal to make optimal use of multi-core GRID resources. Moreover, it naturally brings novel features, such as checkpointing or start-stop-continue possibilities to the processing. This is important for debugging or to split the processing over multiple GRID jobs.

### 5.4.3 Analysis

In Run 3, about 4 PB of AODs will be produced per Pb-Pb running period and a total of about 50 PB will be accumulated in Runs 3 and 4. Considering a typical analysis turnaround cycle of a few days for the full data set and assuming that all data is read only once, a data throughput of the order of up to 100 GB/s is required. In order to meet this requirement, an optimized analysis model as well as a new analysis framework have been designed.

In order to achieve a fast turnaround cycle for analysis code validation and cut optimization, 10% of each data set (including simulated data) are copied to dedicated analysis facilities with exclusive access for ALICE (at the time of writing GSI, Darmstadt and Wigner, Budapest). Each of these consists of 20000 CPU cores, equipped with fast local storage and an internal network capable of sustaining high rates of data transfer from the storage to the computing nodes. The facility only analyzes local data, to reduce problems due to slow external network connections or remote storage instabilities. The fast internal network allows for data to be moved quickly from the storage to the nodes. Analysis task validation on the analysis facility before running over full data sets on the GRID avoids inefficiencies in the most costly stage of processing. Moreover, a large reduction of processing time is expected from the systematic usage of so called derived data sets of reduced size. This is in particular the case for analysis of rare processes. Derived data sets can be obtained through event selection (filtering) and/or event data reduction (selection of only those quantities strictly needed for a specific analysis).

The new data analysis framework [124] fully leverages the DPL and is built on top of it offering an even higher level of abstraction for the benefit of analysis code writers. As in Run 2, analysis is organized in trains consisting of wagons, the individual analysis tasks [125]. In the new framework wagons correspond to a group of DPL devices allowing to process the tasks in parallel and to remove crashing tasks from the train. Data are represented in memory as flat tables similar to a relational database and stored as flat ROOT trees. This saves the processing time for de- serialization needed for the nested C++ objects used in the old framework. In order to keep the size on disk small, a number of quantities are recomputed automatically when the data are read from disk. Significant development has been done to perform these operations transparent to the user (building on C++17 extensions). The in-memory tables are implemented using Apache Arrow, an Open Source cross-language development platform [94]. It provides interoperability between external tools like Python Pandas [126], Apache Spark [127], and many others. The compatibility with ROOT is guaranteed by using the TArrowDS data source which allows using Arrow with RDataFrame. Besides I/O efficiency, the new data format naturally allows for optimized vectorized processing and declarative analysis. The frameworks API isolates many of the advanced features from the user and data access methods are similar to the ones used in the analysis framework in Runs 1 and 2. This facilitates porting of user analysis code into the new framework.
## 5.5 Central Trigger System

ALICE operates at an interaction rate of 50 kHz for Pb-Pb collisions and up to 1 MHz for pp and p-Pb collisions. The majority of ALICE detectors are read out continuously. A minimum bias trigger signal is recorded along with the continuous data in order to flag collisions. For legacy detectors that are not upgraded to continuous readout, the minimum bias trigger initiates the readout if their readout electronics are available and not busy from a previous readout operation. For pp running, event filtering based on fully reconstructed events will run on the EPN farm. The upgraded central trigger system (CTS) [128] provides clock, timing, and trigger signals.

### 5.5.1 Requirements of the Central Trigger System

The CTS supports continuous and triggered readout. Detectors upgraded to continuous readout use triggered mode for commissioning and dedicated runs only. The CTS governs the continuous readout by sending regular heartbeat (HB) triggers to the front-end of upgraded detectors to synchronise data streams of the detectors and to adjust the data-taking bandwidth by either sending a HB accept (HBa) or a HB reject (HBr) trigger message. The CTS also provides minimum bias triggers at three different latencies, referred to as LM, L0, and L1, depending on the timing requirements of each detector. The CTS operates without dead-time by processing trigger inputs and distributing the corresponding trigger output signal for each bunch crossing. The CTS is connected to the ALICE readout via its own dedicated CRU, such that trigger decisions are recorded together with the detector data. In addition, the system may be used to monitor the status of all CRUs and is able to throttle the readout rate depending on the status of the CRU buffers.

### 5.5.2 Trigger hardware and interfaces

Just like the trigger system for Runs 1 and 2 [1], the CTS system is located in the experimental cavern. It employs a two-stage distribution system, which includes a Central Trigger Processor (CTP, see figure 98) and Local Trigger Units (LTU). The CTP receives the LHC timing signals and the trigger input signals, and is connected via bidirectional TTC-PON optical links [5] to up to 18 Local Trigger Units (LTU), one for each detector. The standard CTS timing and trigger signal distribution path is from the CTS via the detector specific CRUs to the detector front-ends via bidirectional, radiation-tolerant GBT links [34]. Detectors that require latency-critical trigger signals receive these trigger signals additionally on a direct path from the CTS to the detector front-ends on GBT links. Legacy detectors not supporting continuous readout are read out via C-RORC readout cards [6, 101] and require a hardware trigger signal to initiate the readout. They receive the clock and trigger signals via the legacy TTC system [4]. In commissioning runs, the LTUs may be decoupled from the CTP and used to emulate CTP signals for testing purposes.

The CTP and LTU are based on identical PCBs. Each board contains one Xilinx Kintex Ultrascale FPGA, two 1 GB DDR4 memories, two Si5345 PLLs, one FME-HPC connector, two six-fold SFP+ cages, a single SFP+ cage, and two UCD90120A power controllers. The CTP and LTU boards only differ by the installed FPGA: the CTP board is equipped with the more performant XCKU060-2FFVA1156E; the LTU uses the pin-compatible XCKU040-2FFVA1156E FPGAs. The boards feature a triple-width 6U VME format using the VME backplane for power supply only, with all data interfaces being available via the front panel.
<figure><img src="image_108.png" /><figcaption>**Figure 98.** Photograph of a CTP module.</figcaption></figure>

The CTP board utilises an FPGA mezzanine card with a total of 72 LVDS I/O connections: two differential links are used for clock signals, 48 for trigger inputs (12 LM, 12 L0, and 24 L1), four for BUSY inputs from legacy detectors, and two for direct LM trigger outputs for detectors requiring a minimum latency trigger. Some of the LTUs are equipped with a commercial FMC S-18 card to extend the number of optical connections to the detectors from 12 to 19. The LHC clock and orbit signals are connected via Lemo connections. The cards are remotely programmable via their JTAG ports connected to an Ethernet adapter and are controlled via an IPbus interface [129]. The CTS allows monitoring of internal counters, including trigger inputs, subdetector BUSY signals, and an internal snapshot memory.

### 5.5.3 Trigger protocol and data format

The minimum bias trigger input signals are delivered to the CTP by the FIT detector. The TOF, EMCal, DCal, and PHOS detectors also deliver trigger inputs for dedicated run scenarios. The CTS aligns the trigger inputs and synchronises them to the BC clock. The trigger algorithm is applied using a lookup table and produces the trigger output signals. The latencies for the trigger input signals to reach the CTS are 425 ns, 1200 ns, and 6100 ns for LM, L0, and L1, respectively. The CTS processing and signal propagation time is about 150 ns. The total latency from interaction to output trigger signal is 575 ns. The CTS can generate internal triggers controlled by software which are used for debugging and detector calibration.

The CTS allows grouping of detectors into up to 18 clusters, forming a data acquisition partition independent from other clusters. Naturally, it also foresees the inclusion and exclusion of individual detectors depending on run conditions. Similar to Runs 1 and 2, the trigger signal distribution for the triggered legacy detectors is protected by a BUSY signal, which communicates whether a detector is ready to receive the trigger signal. The trigger message transmitted to the CRUs and detector front-ends consists of the trigger type (32 bit), the LHC orbit counter (32 bit), and bunch crossing counter (12 bit). The CTP is read out similar to a detector and its data is merged into the continuous data stream. The CTS readout contains information on the trigger messages sent to the detectors, the trigger input mask (specifying the active CTS trigger inputs), and the trigger mask (specifying the trigger conditions
<figure><img src="image_109.png" /><figcaption>Figure 99. Trigger system overview.</figcaption></figure>

and the active detectors) for each bunch crossing. The transmission of the status information of all the CRU buffers sent upstream to CTP from CRUs will be implemented at a later stage.

A general overview of the trigger system is shown in figure 99.

In the continuous readout scheme, the readout rate may be downscaled by introducing periodic HBr triggers. That data rate can be adjusted by changing the ratio between HBa and HBr triggers as required. At present, the system allows the application of a pre-defined sequence. Dynamic modification of the ratio depending on the CRU buffer status will be implemented at a later stage, once the operation of the ALICE system is fully tuned. This functionality uses the transmission of the state of the CRU buffers in the HB acknowledge messages that are sent by the CRUs to the CTS upon reception of a HBa or HBr message. The HB acknowledge messages of all CRUs are assembled into a HB map in the CTS which is used to assess the buffer states of all CRUs and to decide on modulating the HBa/HBr rate accordingly.

## 5.6 Detector Control System

The ALICE Detector Control System (DCS) [130] ensures safe and stable operation of the experiment. Its architecture is derived from the previous versions used in Run 1 and Run 2, but significant extensions were developed and deployed for Run 3 to allow for integration of new readout electronics. The internal conditions data flow was also modified, to provide continuous streaming of conditions data in real time [131].

An optimized standalone DCS system is available for each detector. These systems are built by detector teams, following the guidelines prepared by the central DCS team. The central DCS further integrates the detector DCS into one distributed system, which can be operated by a single operator.

During the design phase of the DCS great attention was given to the selection of hardware components and software tools. All systems are based on the commercial SCADA (Supervisory Control and Data Acquisition) system WINCC OA [132] provided by Siemens. SCADA systems are
based on industrial standards and are widely used to efficiently supervise processes by monitoring and controlling the devices. High level of standardization allows for deployment of common solutions which largely simplify the development cycle and reduce deviations from operational standards adopted in ALICE:

* The CERN Joint Control Project (JCOP) provides a common framework for integration of standard devices such as power supplies, embedded logical controllers (ELMB) [133], magnetic field sensors, etc. It is developed as a joint effort of LHC experiments. ALICE contributes with a component for integration and control of ISEG power supplies. The JCOP framework also contains, for example, all necessary tools for integration of CERN standard protocols DIM [134] and DIP [135] as well as the State Management Interface (SMI++) [136] used to model the device operation as FSMs.

* The ALICE framework further extends the JCOP framework with tools specific to ALICE such as a unified user interface, or FRED framework for integration of front-end modules.

Despite its complexity, the whole DCS can be operated by one person using a single user interface. The FSM mechanism guarantees that commands are executed in a correct order and experiment conditions (such as status of cooling or power systems) are always taken into account. Predefined states were implemented as a response to different conditions of the experiment. For example, the requirements for high voltage settings during the beam injections differ from those during data taking.

To minimize the human factor in the operation, most actions are executed by the system without the need of manual intervention. The operator specifies the desired state (for example, move ALICE to a state compatible with beam injection or magnet ramps) and the system determines and executes the corresponding sequence of commands. Direct interaction of the operator with the system is required only in case of exceptions (for example the recovery from power cut or detector trip). Almost all operations could be executed automatically, however certain checkpoints where operator response is required were introduced to ensure that sufficient attention to the operation is given by the shift crew.

### 5.6.1 DCS computing hardware upgrades

The core of DCS computing is installed in ALICE counting room CR 3. Three rows of racks originally hosting the DCS cluster were replaced with 26 new racks in two rows, each equipped with water cooled doors. All DCS computers were replaced with new hardware. In total 200 new servers, mostly running WINCC OA [132], were installed and configured. These servers run the central distributed SCADA system and front-end control servers, and provide services required for DCS operation (DNS, BootP servers providing boot images to diskless controllers, fileservers, databases, etc.). Old servers were kept operational alongside the new servers in order to provide a smooth transition to the new hardware for the detector groups. Using this approach, the DCS maintained almost uninterrupted operation during the whole LS2 period.

Files shared between several systems are residing on a new redundant cluster of file servers. These are also hosting installation repositories and system backups. The fileservers are mirrored in a cluster installed outside of the ALICE experiment network.
The DCS configuration data (device settings, front-end configuration, etc.) and conditions data (temperatures, currents, pressures, system states, etc.) are stored in an ORACLE database. This mission critical service is provided by a new cluster consisting of four servers and highly redundant storage. The whole database is replicated using ORACLE Active Data Guard technology (ADG) [137] to a twin cluster installed outside of the ALICE detector network. The ADG provides data availability and protection by mirroring data to a standby database which can replace the ALICE online cluster in case of severe failure. The standby database further provides ALICE data in read-only mode, to offload heavy load operations from the production cluster.

All computer racks hosting the DCS cluster are equipped with switches with 10 Gigabit Ethernet uplinks to the router. This provides sufficient bandwidth for DCS operation also beyond Run 3. A total of 144 multi-mode and 48 single-mode patch panels are installed in the DCS counting room. Two high-speed (40 Gb/s) Ethernet links connect the DCS cluster with the FLP farm. These links carry the traffic to the front-end electronics and are used for the streaming of DCS data to $O^2$ as explained below.

The DCS counting room network renewal is part of the overall $O^2$ network upgrade. The DCS network services counting rooms, control rooms, surface areas (gas system, EPN containers), and the whole cavern. The infrastructure put in operation before Run 1 reached the end of its lifetime and was entirely replaced. All routers and switches were installed in parallel to the installation and commissioning of detectors. New infrastructure was installed alongside the old one and was put in operation in phases, in order to minimize the impact on the commissioning activities of the experiment. The network upgrade process lasted two years and the old network was decommissioned recently.

One of the key factors affecting the selection of communication buses is the distance between the servers installed on the surface and the devices in the cavern. To provide stability in a harsh environment, the Controller Area Network (CANbus) [138] has been adopted. It is based on a serial bus designed for robust performance over long distances. A majority of the devices controlled by CANbus are commercial power supplies or Embedded Logical Monitoring Boards (ELMB) mainly used for environment monitoring and rack control. CANbus is also used in the ITS controls, where it provides a redundant channel for hardware access and also served the interlock control during the on-surface commissioning. All CANbus controllers, previously based on USB or PCI devices, were replaced with ANAGATE CANbus-Ethernet gateways [139]. This change allowed to remove the dependency on a physical connection between the DCS server and CAN controller. In case of a server failure, a physical intervention on the CANbus network is no longer required.

### 5.6.2 DCS software upgrades

The DCS software is structured into several layers as shown in figure 100.

The driver layer is connected directly to controlled hardware. It provides the device specific low-level interface. To integrate the numerous hardware configurations used in ALICE, this layer also provides hardware abstraction, which hides all device-specific details. The devices are presented to the DCS by standardized interfaces. Most commercial devices use the industrial communication standard OPC [140] for this purpose. An OPC server communicates with the hardware and exposes its functionality in a form of standard commands and services carrying the monitored data.
Custom hardware modules developed for ALICE are controlled by software exposing its functionality to WINCC OA using the CERN DIM protocol. Similar to OPC, the DIM servers convert the device specific interface to a standardized set of commands and services. Both technologies require that either an OPC or DIM client is deployed on the WINCC system, however, this component is common for all detectors.

The controls layer implemented in WINCC OA performs the basic control and monitoring tasks. It sends commands to devices and reads back the responses. Scripted actions allow to execute procedures, based on the received response. Monitored values are in addition compared with predefined settings and in case of deviations a message can be sent to an alert system. In most cases an automated procedure can regulate the settings without any human intervention.

The logical layer encodes detector specific operations in a form of a Finite State Machine (FSM) organized in a hierarchical way. This layer is entirely encoded in SMI++ language, which allows for definition of stable states and rules for the transitions between them. The FSM logic encodes experience gained over years of operation and defines reaction of detectors to different experiment conditions (such as beam injections, data taking, ALICE magnet ramps, etc.)

Finally, the user interface layer presents the DCS to the operator using a unified interface. All DCS functionality can be reached from a single panel in an intuitive way. Each action sent from the user interface is verified by high level scripts and protects the experiment from incorrect commands sent while detectors are not in compatible condition.

<figure><img src="image_110.png" /><figcaption>**Figure 100.** DCS software architecture.</figcaption></figure>

A significant part of the software upgrade concerned the communication with the devices. In the past years the widely used commercial OPC standard evolved and a new OPC UA technology [140] emerged. It replaces the older OPC DA technology used in Run 1 and 2. During the LS2, the new standard was tested and adopted in ALICE. Currently, a total of 49 OPC servers control commercial hardware (power supplies, PLCs, etc.).

Upgraded front-end electronics in ALICE are read out through GBT links described in section 2.2. Those are controlled by CRUs, installed in FLP servers. The same links are used to configure, control, and monitor the detector front-end electronics. This grouping of functionality has an impact
on DCS, because the FLPs are not part of the DCS domain. As a result, the DCS does not have direct control over the front-end electronics.

A client-server based architecture, named ALFRED, was developed to address the link sharing between the readout and the DCS. The low-level link access is established through ALF (ALICE Low-level Access) module and executed on the FLPs. This software is detector agnostic and transmits data received from FRED (Front-End Device server) [141] to the front-end electronics. If the communication with the front-end electronics is based on the Slow Control Adapter (SCA) protocol, commands are sent through a dedicated communication channel using reserved bits in the transmitted GBT frames. For faster controls (mass configurations), the SCA channel does not provide sufficient throughput. Custom protocols, such as Single Word Transactions (SWT) are mastered by ALFRED. The data are transmitted in dedicated GBT frames in this case. The response produced by the detector front-end electronics is propagated from ALF back to FRED as illustrated in figure 101.

<figure><img src="image_111.png" /><figcaption>**Figure 101.** Access to the hardware implemented in ALF-FRED mechanism.</figcaption></figure>

The FRED component is a framework provided by central DCS. Detector specific code is embedded inside the FRED server framework. The framework functionality covers the ALF-to-FRED communication and synchronization. FRED also provides a communication layer to interface the system with WINCC OA. Using this architecture, a large uniformity has been achieved even between the largely heterogeneous detector front-ends. Currently eight detectors use FRED to communicate with electronics through GBT links and one detector uses FRED to control the user part of the CRU firmware.
Profiting from the flexibility of the ALFRED framework, the ITS detector developed an interlock system controlled via CANbus. A modified version of ALF (CANALF) communicates with the devices using the ANAGATE gateway instead of the CRU. The ITS electronics can also use CANbus in place of GBT as a fallback solution, bypassing the FLP. Due to reduced bandwidth of CANbus, this solution cannot fully replace the standard communication over GBT link, however, it provides the necessary level of redundancy. Using a command originating in WINCC OA, FRED can redirect communication from ALF to CANALF and continue detector operation still using the same user code.

Finally, FRED is a fully scalable and parallel framework. For large detectors, this can result in high amounts of data to be processed by a single instance of FRED. Depending on detectors size, the deployment of FRED ranges from single FRED instance on a single server, through multiple FRED instances on the same server, up to several instances distributed across multiple servers.

### 5.6.3 DCS conditions data

Data collected from detectors are stored by WINCC OA systems in an ORACLE database. Part of these data are used in the $O^2$ online processing. During LHC Runs 1 and 2, these data were retrieved by a dedicated process after each run (a period of stable data taking during a LHC fill). The collected data were merged with detector data acquired by the DAQ system and further processed by the offline framework. The Run 3 conditions processing expects continuous data processing, and the DCS conditions data must be provided in real time. This challenge was addressed by the newly developed ADAPOS (Alice DAtaPOint Service) system [142].

The ADAPOS server collects a requested set of conditions data from individual WINCC OA systems and assembles a so-called Full Buffer Image (FBI). Each value stored in the FBI is updated whenever the WINCC OA system provides a new measurement. The FBI represents an up-to-date image of current DCS conditions.

The ADAPOS system is able to periodically send the up-to-date FBI for further processing with an update frequency of 20 Hz. However, most of the values do not usually change significantly, and most records in the FBI remain unchanged for longer periods of time. To save resources, ADAPOS can operate in transparent mode, where the FBI is maintained in the ADAPOS memory, but only parameters that recently changed are forwarded to the consumer. The mode selected for operation is a hybrid one, which complements the transparent mode with a full FBI sent at regular intervals to reinforce data integrity at the point of processing.

To ensure stability and high availability, several ADAPOS servers can operate in parallel and provide full system redundancy. Implementation of ADAPOS required significant changes inside the WINCC OA system. In the standard configuration used at CERN, the communication with the ORACLE database is handled by a specialized manager (RDB manager). This connects WINCC OA directly with the database. Part of this data stream requested by ADAPOS had to be duplicated and transferred from WINCC OA to ADAPOS using a DIM protocol.

A new technology developed by Siemens and further extended at CERN allowed a novel approach. The Next Generation Archive (NGA) manager plugs into the core of the system and collects data tagged for archival. The NGA can split the data stream between several backends. A CERN developed ORACLE backend handles all communication with ORACLE and fully replaces the RDB manager. The ADAPOS backend, developed in ALICE, forwards condition parameters to ADAPOS. To add new parameters to the ADAPOS data stream, the detector expert needs to tag the
corresponding data, and all related configuration will be executed on the fly by the individual software
components. Figure 102 shows the data flow from WINCC OA to ORACLE and ADAPOS using the
NGA manager and corresponding backends. Each WINCC OA system is built as a collection of highly
specialized managers. The data flow is managed by the Data manager (DM) and Event manager (EM).

<figure><img src="image_112.png" /><figcaption>Figure 102. DCS conditions data flow to ORACLE and ADAPOS.</figcaption></figure>

The described mechanism represents the main data flow from WINCC OA systems to ORACLE or ADAPOS, respectively. The data streaming, however, is not optimal for large data sets, such as masks, chip configurations, etc., that usually do not change during the run. A dedicated service called FilePush allows the injection of such data directly to the CCDB. A similar service called FilePull moves data from the CCDB to the DCS configuration database. This service was created to handle configuration data produced on the O² data processing side.

A large part of the data is consumed by various online displays and user interfaces. This part of the data flow is covered by WINCC OA tools. Analysis and mitigation of operational issues and analysis of long term performance require retrieval of large amounts of data from the database. The online cluster in ALICE offers data recorded since 2008. Retrieval of large historical records is, however, inefficient when only WINCC OA built-in tools are used. The DCS was therefore extended by a new system, named DARMA (the DCS Archive Manager).

DARMA is a web-based system allowing the retrieval of historical values from the database. To protect the online ORACLE cluster from possible overload, DARMA is configured to access data from the online replica of the ALICE production database. Based on ORACLE RAC ADG, this replica contains up-to-date values in real time mode. The user of DARMA can choose between a web-based GUI and a scripting interface to configure the requests. Finally, the DARMA system is extended by a plugin, allowing the Grafana system to access the DCS ORACLE archive. The full DCS data flow from/to external services is shown in figure 103.
<figure><img src="image_113.png" /><figcaption>Figure 103. DCS data exchange with $O^2$ and external consumers.</figcaption></figure>

### 5.6.4 DCS operator environment

Even if the DCS provides fully automated operation and is available at all times with minimum downtime (about two or three days during the whole year for scheduled updates and backups), human supervision is maintained during detector operation. The main task of the operator is to react to anomalies indicated by the alert system and to assist the experts in mitigation. An intuitive and reliable operator environment is an important part of the DCS design.

The complexity of the control system presents a challenge for the system designers when creating user interfaces: about one million parameters can be accessed through DCS tools and the global DCS logic is encoded in 70000 finite state machines. Central operations require large sequences of steps that need to be executed by the operator. To facilitate these tasks and protect the systems from operator errors, an ALICE DCS UI component was developed. It provides a coherent way for accessing all ALICE parameters from one user interface. Navigation is based on detector hierarchy, which makes this process intuitive.

All controlled objects are represented in a visual pane. Navigating through the hierarchy, the main user interface displays panels for selected objects. Instructions for the use of the panel are either available in the operator manual, or are embedded in the alert instructions if an anomaly occurred. Various operator panels embedded in the common user interface are shown in figure 104.

Complex High-level operations related to detector safety, preparation of the detectors for data taking, or communication with LHC are further embedded in macro commands. The DCS operator is guided through the operations, and the procedure evaluates the status of all components in parallel and prevents human error. For example, a detector not yet fully recovered from a condition compatible with a magnet ramp operation will not allow the operator to ramp up the high voltage until the recovery steps are executed. Similar to this, a detector in a state not compatible with beam injection ('not beam safe') will not allow the operator to grant the injection permits to the LHC and hence blocks the injection until the detector is brought to a state safe for beam operation.

The new ALICE DCS UI component is a key operational tool. More than 300 operators were already trained for the Run 3 period, most of them being colleagues with no prior experience with system controls. Use of a single user interface component hiding as many detector specificities as possible significantly reduces the learning curve and improves the operational stability.
<figure><img src="image_114.png" /><figcaption>**Figure 104.** ALICE DCS UI with various panels used by the operators.</figcaption></figure>

# 6 Physics performance

The physics programme motivating the upgrade was first established in the Letter of Intent [143] and the individual Technical Design Reports and was further expanded upon in the summary report of the heavy-ion physics programme at the LHC in Run 3 and 4 across all the LHC experiments [144]. A precise measurement of the long-wavelength behaviour, i.e. the macroscopic fluid-like evolution of a high-density and high-temperature system, allows the determination of transport properties, such as the viscosity of the QGP, and provides information about the equation of state. Probes that are sensitive to short wavelengths give access to the microscopic parton dynamics in a deconfined QGP state. A further goal is to study particle production in order to assess the validity of the fluid description and the role of collectivity in high-multiplicity pp collisions and in p-Pb collisions. Finally, the precise measurement of nuclear parton densities over a wide ($x, Q^2$) range is fundamental to constrain the initial conditions.

In the following, some examples of the physics performance with the upgraded detector will be briefly highlighted. Figure 105 shows the resolution in the distance of closest approach of tracks to the primary vertex, in both the transverse ($r\phi$, solid circle markers) and longitudinal ($z$, open square markers) directions in Pb-Pb collisions, as simulated with the full $O^2$ simulation framework for the upgraded detector (red points) compared to Run 2 data (blue points). The impact parameter resolution is seen to improve by a factor of approximately three in the transverse direction and a factor ten or more in the longitudinal direction.

Heavy quarks are produced in hard scatterings between incoming quarks and gluons, and lose energy as they interact with the quark-gluon plasma while propagating out of the collision zone. To quantify the effect of energy loss, the transverse momentum distributions of produced hadrons are measured both in proton-proton and Pb-Pb collisions at the same centre-of-mass energy per nucleon pair. The ratio of the $p_T$ spectra measured in Pb-Pb collisions to those in pp collisions scaled by the nuclear overlap function $T_{AA}$, which is proportional to the number of binary nucleon-nucleon
collisions, is then calculated:

$$R_{AA} = \frac{dN/dp_{T}|_{AA}}{T_{AA} d\sigma/dp_{T}|_{pp}}. \qquad (6.1)$$

The ratio $R_{AA}$ is called the nuclear modification factor. The left panel of figure 106 shows the expected precision for measurements of the nuclear modification factor for charm and beauty hadrons, which are used to determine the mass dependence of energy loss of heavy quarks propagating through the quark-gluon plasma [144]. The figure shows projections for the nuclear modification factor for the open charm D⁰ mesons and for measurements of beauty mesons via full hadronic reconstruction and measurements of production of non-prompt charm particles. The mass dependence of parton energy loss is most pronounced at transverse momenta around or below the quark mass, when the quark is moving through the QGP with a velocity that is significantly smaller than the speed of light. The upgraded detector provides measurements with uncertainties that are small enough to reveal the expected mass dependence in this momentum range.

The right panel of figure 106 shows the expected precision for measurements of the azimuthal asymmetry for charm mesons and baryons [144], as characterised by the elliptic flow $v_2$, which is the second harmonic coefficient of the Fourier expansion of the distributions of the azimuthal angles $\varphi$ of the hadrons with respect to the reaction plane angle $\Psi$:

$$\frac{dN}{d\varphi} \propto 1 + 2v_2 \cos 2(\varphi - \Psi). \qquad (6.2)$$

These measurements are in particular sensitive to the diffusion of charm quarks in the QGP and the path length dependence of parton energy loss effects.

<figure><img src="image_115.png" /><figcaption>**Figure 105.** Transverse (solid circle markers) and longitudinal (open square markers) impact parameter resolution in Pb-Pb collision data with ALICE 1 (blue points) and simulations the upgraded detector, ALICE 2 (red points).</figcaption></figure>
<figure><img src="image_116.png" /><figcaption>**Figure 106.** Performance projections for measurements of the nuclear modification factor $R_{AA}$ and elliptic flow coefficient $v_2$ (see text for details) for charm and beauty hadrons [144].</figcaption></figure>

<figure><img src="image_117.png" /><figcaption>**Figure 107.** Performance projection for the ratio of beauty baryon and meson yields in central Pb-Pb collisions with the total expected integrated luminosity of $10 \text{ nb}^{-1}$ (at full magnetic field) for Run 3 and 4. From [144].</figcaption></figure>

Figure 107 shows the expected performance for measurements of the beauty baryon-to-meson ratio. These measurements probe the mechanisms for meson and baryon formation. For example, hadronisation via quark coalescence (red and blue curve) is expected to lead to an increased production rate of beauty baryons relative to that of the $B^+$ meson [145–147]. Performing such measurements for the first time in the beauty sector will enhance the understanding of hadronisation mechanisms.

Figure 108 shows the expected precision of the measurement of electron-positron pair production. The left panel shows the mass distribution of all pairs, and the right panel shows the result after subtracting the calculated contributions from the decays of light hadrons, except the $\rho$ meson and heavy-flavour decays. The result in the right panel is sensitive to the modification of the $\rho$ meson spectral function in the high-density collision system (indicated by the red line), as well as thermal emission (orange line) from all stages of the collision, including the QGP phase [148], which provides a unique window on the system before hadronisation.
<figure><img src="image_118.png" /><figcaption>**Figure 108.** Projection of the invariant mass distribution of electron-positron pairs before (left panel) and after (right panel) subtraction of the contribution from decays of light and heavy flavour mesons, except the $\rho$ meson. From [144].</figcaption></figure>

<figure><img src="image_119.png" /><figcaption>**Figure 109.** Projected yield ratio of $\psi(2S)$ and $J/\psi$ in pp and Pb-Pb collisions, measured with the MFT and muon arm [143]. Model calculations from [150, 151].</figcaption></figure>

A further upgrade of the inner tracking system is planned to further improve the capabilities of ALICE for measurements of electron-positron pairs as well as heavy-flavour measurements. To achieve this, the three innermost layers of the ITS will be replaced with wafer-scale silicon sensors that are bent into a cylindrical shape, supported by carbon foam. More details can be found in the ITS3 Letter of Intent [149].

Figure 109 shows the expected performance for measurements of the production of the two main charmonium states with the MFT and the muon detectors. The MFT provides additional background rejection of non-prompt muons and an improved momentum resolution, which make
<figure><img src="image_120.png" /><figcaption>**Figure 110.** Projected limits on jet energy loss from the measurement of the yields of jets recoiling from high-$p_T$ trigger particles in different collision systems. From [152].</figcaption></figure>

a precise measurement of the production of the $\psi(2S)$ possible. The production rates of $J/\psi$ and $\psi(2S)$ are compared to gain further insight into the mechanisms for quarkonium dissociation and regeneration in the QGP.

Another important part of the physics program is the investigation of several effects that have been observed in high-multiplicity collisions of protons that are reminiscent of heavy-ion collisions [153], such as the increased production yields of multi-strange baryons with respect to pions and azimuthal asymmetry of produced particles. In particular, these observations raise the question whether quark-gluon plasma is formed in high-multiplicity pp and p-Pb collisions. Parton energy loss is a distinctive signature of QGP formation that was yet observed in small collision systems. The search for this effect will continue with the much larger pp and p-Pb samples of Run 3 and also using a short pilot run with oxygen-oxygen collisions, which provides similar multiplicities but larger geometrical size. Figure 110 shows the projected sensitivity limits on energy transported outside the jet cone based on the measurement of jet yield recoiling from high-$p_T$ particles in various smaller collision systems [152] compared to the values determined from existing measurements in p-Pb and Pb-Pb collisions [154, 155]. The large data sample of proton-proton collisions of Runs 3 and 4 (with a target integrated luminosity of 200 pb$^{-1}$) will also enable unique studies of perturbative QCD effects, such as the dead cone in gluon radiation off heavy quarks, of the residual strong interaction among hadrons pairs, including multistrange hadrons and light nuclei, and of hypernuclei production [153].

# 7 Conclusions and outlook

After the successful conclusion of data taking in the LHC Runs 1 and 2, the ALICE detector has undergone a major upgrade in order to enable new or more precise measurements in Runs 3 and 4. The inner tracking system was completely replaced and is now fully instrumented with silicon pixel sensors, which provide a much better pointing resolution and support continuous readout. The muon forward tracker uses the same pixel sensors and provides for the first time precise pointing information for forward-rapidity muons reconstructed in the muon chambers and the muon identifier. The time projection chamber is now read out using new detectors based on gas electron multiplication foils
that reduce ion backflow and enable continuous readout at much higher interaction rates. The new fast interaction trigger system provides the interaction trigger, as well as multiplicity measurements and time measurements for offline analysis. All other detector systems have undergone consolidation work and several systems have been equipped with upgraded readout electronics and/or new firmware to increase the readout speed or support continuous readout.

All upgrades were completed on schedule and have been commissioned with standalone runs as well as global commissioning runs in 2021 and 2022. The systems have been tested with pilot beams in October 2021 before the official start of Run 3 and are being operated successfully for routine data taking with proton-proton collisions at $\sqrt{s} = 13.6$ TeV since July 2022. High-rate tests are being performed to qualify the systems for the high data rates in Pb-Pb collisions. The upgraded detectors will strongly extend the physics reach of the experiment, in particular in the sector of open and hidden heavy-flavour probes, measurements of thermal radiation from the quark-gluon plasma, as well as measurements of light nuclear states, final-state interactions of hadrons, and the internal structure of jets.

Two additional upgrades are in preparation for Long Shutdown 3 (2026–2028), with the goal of further enhancing the physics reach of the experiment in Run 4: new inner layers for the inner tracking system (ITS3 [149]) and a forward calorimeter with high-granularity readout (FoCal [156]).

The ITS3 upgrade consists in the replacement of the three innermost layers of the ITS2 with three ultra-light truly-cylindrical layers made with curved large-area MAPS sensors [157]. The innermost layer will have a radius of 18 mm and will surround a new beam pipe with reduced radius (16 mm) and thickness. The pointing resolution will be better than that of the ITS2 by a factor of two up to a transverse momentum of $5 \text{ GeV/c}$, reaching down to about $12 \text{ µm}$ at $p_T = 1 \text{ GeV/c}$. The low-$p_T$ tracking efficiency will also improve. The ITS3 will strongly enhance the low-mass dielectron, heavy-flavour meson, and baryon production measurements.

The FoCal consists of an electromagnetic calorimeter with high-granularity readout for optimal separation of direct-photon showers from those of neutral pions at forward pseudorapidity ($3.4 < \eta < 5.8$), coupled to a hadronic calorimeter for additional hadron rejection. The required granularity is achieved using a combination of MAPS silicon pixel readout planes and silicon pad readout planes. The main physics goal of the FoCal is the study of gluon parton distribution functions in the lead nucleus at Bjorken $x$ values down to $10^{-6}$ using the nuclear modification factor of forward direct photons with transverse momentum $2 < p_T < 20 \text{ GeV/c}$ in p-Pb collisions.

For the LHC Runs 5 and 6, a completely new apparatus, named ALICE 3, is proposed [158]. The ALICE 3 detector consists of a vertexing and tracking system with unique pointing resolution over a large pseudorapidity range ($-4 < \eta < +4$), complemented by multiple sub-detector systems for particle identification, including silicon time-of-flight layers, a ring-imaging Cherenkov detector with high-resolution readout, a muon identification system, and an electromagnetic calorimeter. Unprecedented pointing resolution of $10 \text{ µm}$ at $p_T = 200 \text{ MeV/c}$ at midrapidity in both the transverse and longitudinal directions can be achieved by placing the first layers as close as possible to the interaction point, on a retractable structure to leave sufficient aperture for the beams at injection energy. This next-generation apparatus will, on the one hand, enable novel studies of the QGP and, on the other hand, open up important physics opportunities in other areas of QCD and beyond. The main new studies in the QGP sector focus on low-$p_T$ heavy-flavour production, including beauty hadrons, multi-charm baryons and charm-charm correlations, as well as on precise multi-differential measurements of dielectron emission to probe the mechanism of chiral-symmetry restoration and
the time-evolution of the QGP temperature. Besides QGP studies, ALICE 3 can uniquely contribute to hadronic physics, for example with femtoscopic studies of the interaction potentials between charm mesons and searches for nuclei with charm, to fundamental physics, with tests of the Low theorem for ultra-soft photon emission, and to searches for physics beyond the Standard Model, for example in the sector of axion-like particles and the anomalous magnetic moment of $\tau$ leptons. The programme aims to collect an integrated luminosity of about $35 \text{ nb}^{-1}$ with Pb-Pb collisions and $18 \text{ fb}^{-1}$ with pp collisions at top LHC energy. The potential to further increase the luminosity for ion running in the LHC by using smaller ions (e.g. $^{84}$Kr or $^{128}$Xe), as well as further runs with small collision systems to explore the approach to thermal equilibrium, are being explored.

## Acknowledgments

The ALICE Collaboration would like to thank all its engineers and technicians for their invaluable contributions to the construction of the experiment and the CERN accelerator teams for the outstanding performance of the LHC complex. The ALICE Collaboration gratefully acknowledges the resources and support provided by all Grid centres and the Worldwide LHC Computing Grid (WLCG) collaboration. The ALICE Collaboration acknowledges the following funding agencies for their support in building and running the ALICE detector: A.I. Alikhanyan National Science Laboratory (Yerevan Physics Institute) Foundation (ANSL), State Committee of Science and World Federation of Scientists (WFS), Armenia; Austrian Academy of Sciences, Austrian Science Fund (FWF): [M 2467-N36] and Nationalstiftung für Forschung, Technologie und Entwicklung, Austria; Ministry of Communications and High Technologies, National Nuclear Research Center, Azerbaijan; Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), Financiadora de Estudos e Projetos (Finep), Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) and Universidade Federal do Rio Grande do Sul (UFRGS), Brazil; Bulgarian Ministry of Education and Science, within the National Roadmap for Research Infrastructures 2020-2027 (object CERN), Bulgaria; Ministry of Education of China (MOEC), Ministry of Science & Technology of China (MSTC) and National Natural Science Foundation of China (NSFC), China; Ministry of Science and Education and Croatian Science Foundation, Croatia; Centro de Aplicaciones Tecnológicas y Desarrollo Nuclear (CEADEN), Cubaenergía, Cuba; Ministry of Education, Youth and Sports of the Czech Republic, Czech Republic; The Danish Council for Independent Research | Natural Sciences, the VILLUM FONDEN and Danish National Research Foundation (DNRF), Denmark; Helsinki Institute of Physics (HIP), Finland; Commissariat à l'Energie Atomique (CEA) and Institut National de Physique Nucléaire et de Physique des Particules (IN2P3) and Centre National de la Recherche Scientifique (CNRS), France; Bundesministerium für Bildung und Forschung (BMBF) and GSI Helmholtzzentrum für Schwerionenforschung GmbH, Germany; General Secretariat for Research and Technology, Ministry of Education, Research and Religions, Greece; National Research, Development and Innovation Office, Hungary; Department of Atomic Energy Government of India (DAE), Department of Science and Technology, Government of India (DST), University Grants Commission, Government of India (UGC) and Council of Scientific and Industrial Research (CSIR), India; National Research and Innovation Agency - BRIN, Indonesia; Istituto Nazionale di Fisica Nucleare (INFN), Italy; Japanese Ministry of Education, Culture, Sports, Science and Technology (MEXT) and Japan Society for the Promotion of Science (JSPS) KAKENHI, Japan; Consejo Nacional de Ciencia (CONACYT) y
Tecnología, through Fondo de Cooperación Internacional en Ciencia y Tecnología (FONCICYT) and Dirección General de Asuntos del Personal Academico (DGAPA), Mexico; Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO), Netherlands; The Research Council of Norway, Norway; Commission on Science and Technology for Sustainable Development in the South (COMSATS), Pakistan; Pontificia Universidad Católica del Perú, Peru; Ministry of Education and Science, National Science Centre and WUT ID-UB, Poland; Korea Institute of Science and Technology Information and National Research Foundation of Korea (NRF), Republic of Korea; Ministry of Education and Scientific Research, Institute of Atomic Physics, Ministry of Research and Innovation and Institute of Atomic Physics and University Politehnica of Bucharest, Romania; Ministry of Education, Science, Research and Sport of the Slovak Republic, Slovakia; National Research Foundation of South Africa, South Africa; Swedish Research Council (VR) and Knut & Alice Wallenberg Foundation (KAW), Sweden; European Organization for Nuclear Research, Switzerland; Suranaree University of Technology (SUT), National Science and Technology Development Agency (NSTDA) and National Science, Research and Innovation Fund (NSRF via PMU-B B05F650021), Thailand; Turkish Energy, Nuclear and Mineral Research Agency (TENMAK), Turkey; National Academy of Sciences of Ukraine, Ukraine; Science and Technology Facilities Council (STFC), United Kingdom; National Science Foundation of the United States of America (NSF) and United States Department of Energy, Office of Nuclear Physics (DOE NP), United States of America. In addition, individual groups or members have received support from: European Research Council, Strong 2020 - Horizon 2020, Marie Skłodowska Curie (grant nos. 950692, 824093, 896850), European Union; Academy of Finland (Center of Excellence in Quark Matter) (grant nos. 346327, 346328), Finland; Programa de Apoyos para la Superación del Personal Académico, UNAM, Mexico.

## References

[1] ALICE collaboration, *The ALICE experiment at the CERN LHC*, 2008 JINST 3 S08002.

[2] ALICE collaboration, *Performance of the ALICE Experiment at the CERN LHC*, Int. J. Mod. Phys. A 29 (2014) 1430044 [arXiv:1402.4476].

[3] ALICE collaboration, *The ALICE experiment — A journey through QCD*, arXiv:2211.04384.

[4] B.G. Taylor, *Timing distribution at the LHC*, in the proceedings of the 8th Workshop on Electronics for LHC Experiments, Colmar, France, 9–13 September 2002, pp. 64–74.

[5] E.B.S. Mendes et al., *The 10G TTC-PON: challenges, solutions and performance*, 2017 JINST 12 C02041.

[6] ALICE and ATLAS collaborations, *The C-RORC PCIe card and its application in the ALICE and ATLAS experiments*, 2015 JINST 10 C02022.

[7] J.P. Cachemiche et al., *The PCIe-based readout system for the LHCb experiment*, 2016 JINST 11 P02013.

[8] O. Bourrion et al., *Versatile firmware for the Common Readout Unit (CRU) of the ALICE experiment at the LHC*, 2021 JINST 16 P05019 [arXiv:1910.08804].

[9] ALICE collaboration, *The ALPIDE pixel sensor chip for the upgrade of the ALICE Inner Tracking System*, Nucl. Instrum. Meth. A 845 (2017) 583.
[10] W. Snoeys, CMOS monolithic active pixel sensors for high energy physics, Nucl. Instrum. Meth. A 765 (2014) 167.

[11] S. Senyukov et al., Charged particle detection performances of CMOS pixel sensors produced in a 0.18µm process with a high resistivity epitaxial layer, Nucl. Instrum. Meth. A 730 (2013) 115 [arXiv:1301.0515].

[12] H. Hernández et al., A Monolithic 32-Channel Front End and DSP ASIC for Gaseous Detectors, IEEE Trans. Instrum. Measur. 69 (2019) 2686.

[13] M. van Elzakker et al., A 10-bit Charge-Redistribution ADC Consuming 1.9 $V_{mu}$W at 1 MS/s, IEEE J. Solid-State Circuits 45 (2010) 1007.

[14] ALICE TPC collaboration, The upgrade of the ALICE TPC with GEMs and continuous readout, 2021 JINST 16 P03022 [arXiv:2012.09518].

[15] ALICE collaboration, Technical Design Report for the Upgrade of the ALICE Inner Tracking System, J. Phys. G 41 (2014) 087002.

[16] ALICE collaboration, ALPIDE, the Monolithic Active Pixel Sensor for the ALICE ITS upgrade, Nucl. Instrum. Meth. A 824 (2016) 434.

[17] ALICE collaboration, Readout of the upgraded ALICE-ITS, Nucl. Instrum. Meth. A 824 (2016) 465.

[18] J. Schambach et al., A Radiation-Tolerant Readout System for the ALICE Inner Tracking System Upgrade, in the proceedings of the 2018 IEEE Nuclear Science Symposium and Medical Imaging Conference, Sydney, NSW, Australia, 10–17 November 2018, p. 1–6 [DOI:10.1109/NSSMIC.2018.8824419].

[19] CMS collaboration, The versatile link, a common project for super-LHC, 2009 JINST 4 P12003.

[20] Two-wire bus-system comprising a clock wire and a data wire for interconnecting a number of stations, U.S. Patent, https://patents.google.com/patent/US4689740A/en.

[21] F. Reidt, Upgrade of the ALICE ITS detector, https://arxiv.org/abs/2111.08301 [DOI:10.48550/ARXIV.2111.08301].

[22] ALICE collaboration, Technical Design Report for the Muon Forward Tracker, CERN-LHCC-2015-001 (2015).

[23] THEALICE collaboration, Upgrade of the ALICE Time Projection Chamber, CERN-LHCC-2013-020 (2013).

[24] J. Alme et al., The ALICE TPC, a large 3-dimensional tracking device with fast readout for ultra-high multiplicity events, Nucl. Instrum. Meth. A 622 (2010) 316 [arXiv:1001.1950].

[25] F. Sauli, GEM: A new concept for electron amplification in gas detectors, Nucl. Instrum. Meth. A 386 (1997) 531.

[26] A. Deisting, C. Garabatos, A. Szabo and D. Vranic, Measurements of ion mobility in argon and neon based gas mixtures, Nucl. Instrum. Meth. A 845 (2017) 215 [arXiv:1603.07638].

[27] M. Villa et al., Progress on large area GEMs, Nucl. Instrum. Meth. A 628 (2011) 182 [arXiv:1007.1131].

[28] ALICE collaboration, Quality assurance of GEM foils for the upgrade of the ALICE TPC, 2017 JINST 12 C01081.

[29] ALICE TPC UPGRADE collaboration, The GEM QA Protocol of the ALICE TPC Upgrade Project, PoS MPGD2017 (2019) 073 [arXiv:1811.07043].
[30] E. Brücken and T. Hildén, GEM Foil Quality Assurance For The ALICE TPC Upgrade, EPJ Web Conf. 174 (2018) 03004 [arXiv:1704.06310].

[31] T.E. Hilden, J.E. Brucken, D. Varga and M. Vargyas, GEM foil gain prediction, PoS MPGD2017 (2019) 010.

[32] M. Capéans-Garrido et al., A GIF++ Gamma Irradiation Facility at the SPS H4 Beam Line, CERN-SPSC-2009-029, CERN, Geneva (2009).

[33] M.R. Jäkel et al., CERN-GIF++: a new irradiation facility to test large-area particle detectors for the high-luminosity LHC program, PoS TIPP2014 (2014) 102.

[34] P. Moreira et al., The GBT Project, in the proceedings of the Topical Workshop on Electronics for Particle Physics, Paris, France, 21–25 September 2009, pp. 342–346
[DOI:10.5170/CERN-2009-006.342].

[35] ALICE collaboration, New Fast Interaction Trigger for ALICE, Nucl. Instrum. Meth. A 845 (2017) 463.

[36] M. Bondila et al., ALICE T0 detector, IEEE Trans. Nucl. Sci. 52 (2005) 1705.

[37] ALICE collaboration, Performance of the ALICE VZERO system, 2013 JINST 8 P10016
[arXiv:1306.3130].

[38] M. Broz et al., Performance of ALICE AD modules in the CERN PS test beam, 2021 JINST 16 P01017
[arXiv:2006.14982].

[39] ALICE collaboration, Readout system of the ALICE Fast Interaction Trigger, 2020 JINST 15 C09005.

[40] ALICE collaboration, New ALICE detectors for Run 3 and 4 at the CERN LHC, Nucl. Instrum. Meth. A 958 (2020) 162116.

[41] ALICE collaboration, Performance of Planacon MCP-PMT photosensors under extreme working conditions, Nucl. Instrum. Meth. A 952 (2020) 161689.

[42] T. Sjöstrand, The PYTHIA Event Generator: Past, Present and Future, Comput. Phys. Commun. 246 (2020) 106910 [arXiv:1907.09874].

[43] R. Brun et al., GEANT3, CERN-DD-EE-84-1 (1987).

[44] V. Grabski, New fiber read-out design for the large area scintillator detectors: providing good amplitude and time resolutions, arXiv:1909.01184.

[45] S. Rojas, The Forward Diffractive Detector for ALICE, PoS LHCP2020 (2021) 221.

[46] ALICE collaboration, Front-end electronics for the RPCs of the ALICE dimuon trigger, IEEE Trans. Nucl. Sci. 52 (2005) 1176.

[47] P. Dupieux et al., Upgrade of the ALICE muon trigger electronics, 2014 JINST 9 C09013.

[48] M. Marchisone, Performance of a resistive plate chamber equipped with a new prototype of amplified front-end electronics in the ALICE detector, J. Phys. Conf. Ser. 889 (2017) 012011.

[49] A. Bianchi et al., Characterization of tetrafluoropropene-based gas mixtures for the Resistive Plate Chambers of the ALICE muon spectrometer, 2019 JINST 14 P11014 [arXiv:1907.03268].

[50] B. Joly et al., Production readiness review for the upgrade of the muon trigger front-end electronics, https://edms.cern.ch/document/1728246/1.

[51] G. Blanchard, P. Crochet and P. Dupieux, The local trigger electronics of the ALICE dimuon trigger, ALICE-EN-2003-010 (2003).
[52] C. Renard et al., Mid readout electronics documentation on the web, https://www-subatech.in2p3.fr/~electro/projets/alice/dimuon/trigger/upgrade, 2019.

[53] P. Dupieux et al., Mid data format and content, 2020.

[54] ALICE collaboration, The ALICE Transition Radiation Detector: construction, operation, and performance, Nucl. Instrum. Meth. A 881 (2018) 88 [arXiv:1709.02743].

[55] J. Jadlovsky et al., Communication architecture of the Detector Control System for the Inner Tracking System, in the proceedings of the 16th International Conference on Accelerator and Large Experimental Physics Control Systems, Barcelona, Spain, 8–13 October 2017, pp. 1930–1933 [DOI:10.18429/JACoW-ICALEPCS2017-THPHA208].

[56] ALICE collaboration, ALICE Time-Of-Flight system (TOF): Technical Design Report, CERN-LHCC-2000-012, CERN, Geneva (2000).

[57] ALICE collaboration, ALICE Time-Of Flight system (TOF): addendum to the Technical Design Report, CERN-LHCC-2002-016, CERN, Geneva (2002).

[58] A.N. Akindinov et al., Design aspects and prototype test of a very precise TDC system implemented for the multigap RPC of the ALICE-TOF, Nucl. Instrum. Meth. A 533 (2004) 178.

[59] ALICE collaboration, Upgrade of the ALICE Readout & Trigger System, CERN-LHCC-2013-019 (2013).

[60] D. Falchieri, DRM2: the readout board for the ALICE TOF upgrade, PoS TWEPP-17 (2018) 081.

[61] F. Carena et al., DDL, the ALICE data transmission protocol and its evolution from 2 to 6 Gb/s, 2015 JINST 10 C04008.

[62] D. Falchieri, Radiation tests and production test strategy for the ALICE TOF readout upgrade board, PoS TWEPP2018 (2019) 025.

[63] D. Falchieri et al., Design and Test of a GBTX-Based Board for the Upgrade of the ALICE TOF Readout Electronics, IEEE Trans. Nucl. Sci. 64 (2017) 1357.

[64] CAEN, CAENVME Library and VME-PCI optical bridges, https://www.caen.it/products/caenvmelib-library/.

[65] D. Falchieri et al., Readout board validation setup for the ALICE Time of Flight detector upgrade, in the proceedings of the 2019 IEEE Nuclear Science Symposium and Medical Imaging Conference, Manchester, U.K., 26 October–2 November 2019, p. 1–3 [DOI:10.1109/NSS/MIC42101.2019.9059832].

[66] G. Giacomelli, A.A. Faust and J.L. Pinfold, A search for highly ionizing particles and slow exotic decays at the LHC using the MOEDAL detectors: letter of intent, CERN-LHCC-98-05, CERN, Geneva (1998).

[67] ALICE collaboration, ALICE Electromagnetic Calorimeter Technical Design Report, CERN-LHCC-2008-014 (2008).

[68] J. Allen et al., ALICE DCal: An Addendum to the EMCal Technical Design Report Di-Jet and Hadron-Jet correlation measurements in ALICE, CERN-LHCC-2010-011 (2010).

[69] ALICE collaboration, Performance of the ALICE Electromagnetic Calorimeter, arXiv:2209.04216.

[70] ALICE collaboration, Configurable electronics with low noise and 14-bit dynamic range for photodiode-based photon detectors, Nucl. Instrum. Meth. A 565 (2006) 768.
[71] R. Esteve Bosch, A. Jimenez de Parga, B. Mota and L. Musa, *The ALTRO chip: A 16-channel A/D converter and digital processor for gas detectors*, *IEEE Trans. Nucl. Sci.* **50** (2003) 2460.

[72] F. Zhang et al., *Point-to-point readout for the ALICE EMCal detector*, *Nucl. Instrum. Meth. A* **735** (2014) 157.

[73] J. Christiansen, A. Marchioro, P. Moreira and A. Sancho, *Receiver ASIC for timing, trigger and control distribution in LHC experiments*, *IEEE Trans. Nucl. Sci.* **43** (1996) 1773.

[74] J. Kral et al., *L0 trigger for the EMCal detector of the ALICE experiment*, *Nucl. Instrum. Meth. A* **693** (2012) 261.

[75] ALICE EMCal collaboration, *The ALICE EMCal L1 trigger first year of operation experience*, 2013 *JINST* **8** C01013 [arXiv:1210.8078].

[76] ALICE collaboration, *ALICE technical design report of the photon spectrometer (PHOS)*, CERN-LHCC-99-004 (1999).

[77] H. Muller et al., *Front-end electronics for PWO-based PHOS calorimeter of ALICE*, *Nucl. Instrum. Meth. A* **567** (2006) 264.

[78] O. Bourrion et al., *Level-1 jet trigger hardware for the ALICE electromagnetic calorimeter at LHC*, 2010 *JINST* **5** C12048 [arXiv:1010.2670].

[79] ALICE PHOS CALORIMETER collaboration, *A high resolution electromagnetic calorimeter based on lead-tungstate crystals*, *Nucl. Instrum. Meth. A* **550** (2005) 169.

[80] ALICE collaboration, *ALICE: Physics Performance Report*, *J. Phys. G* **32** (2006) 1295.

[81] ALICE collaboration, *Calibration of the photon spectrometer PHOS of the ALICE experiment*, 2019 *JINST* **14** P05025 [arXiv:1902.06145].

[82] I.A. Pshenichnov et al., *Mutual heavy ion dissociation in peripheral collisions at ultrarelativistic energies*, *Phys. Rev. C* **64** (2001) 024903 [nucl-th/0101035].

[83] I.A. Pshenichnov, *Electromagnetic excitation and fragmentation of ultrarelativistic nuclei*, *Phys. Part. Nucl.* **42** (2011) 215.

[84] ALICE collaboration, *Measurement of the Cross Section for Electromagnetic Dissociation with Neutron Emission in Pb-Pb Collisions at $\sqrt{s_{NN}}$ = 2.76 TeV*, *Phys. Rev. Lett.* **109** (2012) 252302 [arXiv:1203.2436].

[85] IOxOS Technologies SA, *ADC_3112 — Four Channel 900 Msps 12-bit ADC*, https://www.ioxos.ch/wp-content/uploads/2018/02/ADC_3112_DS_A1.pdf, 2018.

[86] Texas Instruments Inc, *Dual Channel 12-Bit 900Msps Analog-to-Digital Converter*, ADS5409, https://www.ti.com/lit/ds/symlink/ads5409.pdf, 2014.

[87] IOxOS Technologies SA, *IFC_1211 Intelligent FPGA Controller*, https://www.ioxos.ch/wp-content/uploads/2018/02/IFC_1211_DS_A2.pdf.

[88] P. Buncic, M. Krzewicki and P. Vande Vyvre, *Technical Design Report for the Upgrade of the Online-Offline Computing System*, CERN-LHCC-2015-006 (2015).

[89] A.J. Peters, E.A. Sindrilaru and G. Adde, *EOS as the present and future solution for data storage at CERN*, *J. Phys. Conf. Ser.* **664** (2015) 042042.

[90] G. Eulisse et al., *Evolution of the ALICE software framework for Run 3*, *EPJ Web Conf.* **214** (2019) 05010.
[91] A. Rybalchenko, D. Klein, M. Al-Turany and T. Kollegger, *Shared Memory Transport for ALFA*, EPJ Web Conf. **214** (2019) 05029.

[92] M. Richter, M. Krzewicki and G. Eulisse, *Data handling in the ALICE $O^2$ event processing*, EPJ Web Conf. **214** (2019) 01035.

[93] R. Brun and F. Rademakers, *ROOT: An object oriented data analysis framework*, Nucl. Instrum. Meth. A **389** (1997) 81.

[94] A cross-language development platform for in-memory analytics, https://arrow.apache.org.

[95] M. Lettrich, *Fast and Efficient Entropy Compression of ALICE Data using ANS Coding*, EPJ Web Conf. **245** (2020) 01001.

[96] D. Rohr, *Usage of GPUs in ALICE Online and Offline processing during LHC Run 3*, EPJ Web Conf. **251** (2021) 04026 [arXiv:2106.03636].

[97] J. Duda, *Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding*, arXiv:1311.2540.

[98] F. Costa, S. Chapeland, K. Alexopoulos and U. Fuchs, *Assessment of the ALICE O2 readout servers*, EPJ Web Conf. **245** (2020) 01013.

[99] D. Eschweiler and V. Lindenstruth, *The portable driver architecture*, in the proceedings of the 16th Real-Time Linux Workshop, Dusseldorf, Germany, 12–13 October 2014.

[100] D. Eschweiler, *Efficient device drivers for supercomputers*, Ph.D. thesis, Goethe University Frankfurt, Frankfurt am Main, Germany (2015).

[101] ALICE collaboration, *The ReadoutCard Userspace Driver for the New Alice O2 Computing System*, IEEE Trans. Nucl. Sci. **68** (2021) 1876 [arXiv:2010.16327].

[102] P. Konopka and B. von Haller, *The ALICE O2 data quality control system*, EPJ Web Conf. **245** (2020) 01027.

[103] R. Brun and F. Rademakers, *ROOT: An object oriented data analysis framework*, Nucl. Instrum. Meth. A **389** (1997) 81.

[104] T. Mrnjavac, K. Alexopoulos, V. Chibante Barroso and G. Raduta, *AliECS: a New Experiment Control System for the ALICE Experiment*, EPJ Web Conf. **245** (2020) 01033.

[105] Apache mesos, http://mesos.apache.org/, 2020.

[106] M. Al-Turany et al., *ALFA: A framework for building distributed applications*, EPJ Web Conf. **245** (2020) 05021.

[107] HashiCorp, Consul, https://www.consul.io/, 2020.

[108] *The Go programming language*, https://golang.org/, 2020.

[109] G. Vino, V. Chibante Barroso, D. Elia and A. Wegrzynek, *A Monitoring System for the New ALICE O2 Farm*, in the proceedings of the 17th International Conference on Accelerator and Large Experimental Physics Control Systems, New York, NY, U.S.A., 5–11 October 2019, pp. 835–840 [DOI:10.18429/JACoW-ICALEPCS2019-TUDPP01].

[110] V. Chibante Barroso et al., *Towards the integrated ALICE Online-Offline ($O^2$) monitoring subsystem*, EPJ Web Conf. **214** (2019) 03043.

[111] Telegraf, https://www.influxdata.com/time-series-platform/telegraf/, 2020.

[112] Apache kafka, https://kafka.apache.org/, 2018.
[113] Influxdb — downsampling and data retention, https://docs.influxdata.com/influxdb/v1.8/guides/downsample_and_retain, 2020.

[114] Grafana — the open platform for analytics and monitoring, https://grafana.com/, 2020.

[115] ALICE collaboration, The ALICE DAQ infoLogger, J. Phys. Conf. Ser. 513 (2014) 012005.

[116] M. Teitsma, V.C. Barosso, P. Boeschoten and P. Hendriks, Jiskefet, a bookkeeping application for ALICE, EPJ Web Conf. 245 (2020) 04023 [arXiv:2003.05756].

[117] S. Wenzel, A scalable and asynchronous detector simulation system based on ALFA, EPJ Web Conf. 214 (2019) 02029.

[118] M. Al-Turany et al., The FairRoot framework, J. Phys. Conf. Ser. 396 (2012) 022001.

[119] J. Allison et al., Recent developments in Geant4, Nucl. Instrum. Meth. A 835 (2016) 186.

[120] F. Carminati and A. Morsch, Simulation in ALICE, eConf C0303241 (2003) TUMT004 [physics/0306092].

[121] I. Hrivnacova, D. Adamova, V. Berejnoi, R. Brun, F. Carminati, A. Fasso et al., The Virtual Monte Carlo, in the proceedings of the Computing in High Energy and Nuclear Physics Conference, La Jolla, CA, U.S.A., 24–28 March 2003, THJT006.

[122] G. Battistoni et al., Overview of the FLUKA code, Ann. Nucl. Energy 82 (2015) 10.

[123] B. Volkel et al., Using multiple engines in the Virtual Monte Carlo package, EPJ Web Conf. 245 (2020) 02008.

[124] A. Alkin et al., ALICE Run 3 Analysis Framework, EPJ Web Conf. 251 (2021) 03063.

[125] R. Quishpe, J.F. Grosse-Oetringhaus, R. Cruceru and C. Grigoras, Hyperloop – The ALICE analysis train system for Run 3, PoS LHCP2021 (2021) 250 [arXiv:2109.09594].

[126] W. McKinney, Data Structures for Statistical Computing in Python, in the proceedings of the 9th Python in Science Conference, Austin, TX, U.S.A., 28 June–3 July 2010, pp. 56–61 [DOI:10.25080/majora-92bf1922-00a].

[127] M. Zaharia et al., Apache Spark, Commun. ACM 59 (2016) 56.

[128] J. Kvapil et al., ALICE Central Trigger System for LHC Run 3, EPJ Web Conf. 251 (2021) 04022 [arXiv:2106.08353].

[129] C. Ghabrous Larrea et al., IPbus: a flexible Ethernet-based control system for xTCA hardware, 2015 JINST 10 C02019.

[130] A. Augustinus et al., The wonderland of operating the alice experiment, Conf. Proc. C111010 (2011) 1182.

[131] P. Chochula et al., Challenges of the ALICE Detector Control System for the LHC RUN3, in the proceedings of the 16th International Conference on Accelerator and Large Experimental Physics Control Systems, Barcelona, Spain, 8–13 October 2017, pp. 323–327 [DOI:10.18429/JACoW-ICALEPCS2017-TUMPL09].

[132] Wincc open architecture, https://winccoa.com/.

[133] H. Boterenbrood and B.I. Hallgren, The Development of Embedded Local Monitor Board (ELMB), in the proceedings of the 9th Workshop on Electronics for LHC Experiments, Amsterdam, The Netherland, 29 September–3 October 2003, pp. 331–334 http://cds.cern.ch/record/690030 [DOI:10.5170/CERN-2003-006.331].
[134] C. Gaspar, M. Dönszelmann and P. Charpentier, DIM, a portable, light weight package for information publishing, data transfer and inter-process communication, *Comput. Phys. Commun.* **140** (2001) 102.

[135] W. Salter et al., *LHC Data Interchange Protocol (DIP) Definition*, https://edms.cern.ch/file/457113/2/DIPDescription.doc.

[136] C. Gaspar and B. Franek, SMI++ - Object oriented framework for designing control systems for HEP experiments, *Comput. Phys. Commun.* **110** (1998) 87.

[137] https://www.oracle.com/database/data-guard.

[138] https://www.can-cia.org/can-knowledge.

[139] https://www.anagate.de/products/can-ethernet-gateways.php.

[140] https://opcfoundation.org/about/opc-technologies/opc-ua.

[141] M. Tkáčik et al., FRED—Flexible Framework for Frontend Electronics Control in ALICE Experiment at CERN, *Processes* **8** (2020) 565.

[142] J. Lång et al., ADAPOS: An architecture for publishing ALICE DCS conditions data, in the proceedings of the 16th International Conference on Accelerator and Large Experimental Physics Control Systems, Barcelona, Spain, 8–13 October 2017, pp. 482–485
[DOI:10.18429/JACoW-ICALEPCS2017-TUPHA042].

[143] ALICE collaboration, *Upgrade of the ALICE Experiment: Letter Of Intent*, *J. Phys. G* **41** (2014) 087001.

[144] Z. Citron et al., Report from Working Group 5: Future physics opportunities for high-density QCD at the LHC with heavy-ion and proton beams, *CERN Yellow Rep. Monogr.* **7** (2019) 1159 [arXiv:1812.06772].

[145] Y. Oh, C.M. Ko, S.H. Lee and S. Yasui, *Heavy baryon/meson ratios in relativistic heavy ion collisions*, *Phys. Rev. C* **79** (2009) 044905 [arXiv:0901.1382].

[146] S. Plumari et al., *Charmed Hadrons from Coalescence plus Fragmentation in relativistic nucleus-nucleus collisions at RHIC and LHC*, *Eur. Phys. J. C* **78** (2018) 348 [arXiv:1712.00730].

[147] M. He and R. Rapp, *Hadronization and Charm-Hadron Ratios in Heavy-Ion Collisions*, *Phys. Rev. Lett.* **124** (2020) 042301 [arXiv:1905.09216].

[148] R. Rapp, *Dilepton Spectroscopy of QCD Matter at Collider Energies*, *Adv. High Energy Phys.* **2013** (2013) 148253 [arXiv:1304.2309].

[149] L. Musa, *Letter of Intent for an ALICE ITS Upgrade in LS3*, CERN-LHCC-2019-018, CERN, Geneva (2019) [DOI:10.17181/CERN-LHCC-2019-018].

[150] A. Andronic, P. Braun-Munzinger, K. Redlich and J. Stachel, *Decoding the phase structure of QCD via particle production at high energy*, *Nature* **561** (2018) 321 [arXiv:1710.09425].

[151] X. Du and R. Rapp, *Sequential Regeneration of Charmonia in Heavy-Ion Collisions*, *Nucl. Phys. A* **943** (2015) 147 [arXiv:1504.00670].

[152] ALICE collaboration, *ALICE physics projections for a short oxygen-beam run at the LHC*, ALICE-PUBLIC-2021-004 (2021).

[153] ALICE collaboration, *Future high-energy pp programme with ALICE*, ALICE-PUBLIC-2020-005 (2020).
[154] ALICE collaboration, *Constraints on jet quenching in p-Pb collisions at $\sqrt{s_{NN}}$ = 5.02 TeV* measured by the event-activity dependence of semi-inclusive hadron-jet distributions, *Phys. Lett. B* **783** (2018) 95 [arXiv:1712.05603].

[155] ALICE collaboration, *Measurement of jet quenching with semi-inclusive hadron-jet distributions in central Pb-Pb collisions at $\sqrt{s_{NN}}$ = 2.76 TeV*, JHEP **09** (2015) 170 [arXiv:1506.03984].

[156] ALICE collaboration, *Letter of Intent: A Forward Calorimeter (FoCal) in the ALICE experiment*, CERN-LHCC-2020-009, CERN, Geneva (2020).

[157] ALICE ITS PROJECT collaboration, *First demonstration of in-beam performance of bent Monolithic Active Pixel Sensors*, Nucl. Instrum. Meth. A **1028** (2022) 166280 [arXiv:2105.13000].

[158] ALICE collaboration, *Letter of intent for ALICE 3: A next-generation heavy-ion experiment at the LHC*, arXiv:2211.02491.
The ALICE collaboration

S. Acharya ¹²⁷, R. Acosta Hernandez¹¹¹, D. Adamová ⁸⁶, A. Adler⁷⁰, J. Adolfsson ⁷⁵, D. Agguiaro⁵⁴, G. Aglieri Rinella ³², M. Agnello ²⁹, F. Agnese ¹²⁹, N. Agrawal ⁵¹, S. Aguilar Salazar⁶⁷, Z. Ahammed ¹³⁵, S. Ahmad¹⁵, M.U. Ahmed⁶⁴, S.U. Ahn ⁷¹, I. Ahuja ³⁷, S. Aiola ¹⁴⁰, A. Akindinov ¹⁴³, M. Al-Turany ⁹⁸, H.G. Alarcon Cubas¹¹¹, D. Aleksandrov ¹⁴³, B. Alessandro ⁵⁶, M. Alexis³², K. Alexopoulos³², H.M. Alfanda ⁶, R. Alfaro Molina ⁶⁷, G. Alfarone⁵⁶, B. Ali ¹⁵, A. Alici ²⁵, N. Alizadehvandchali ¹¹⁵, A. Alkin ³², J. Alme ²⁰, G. Alocco ⁵², T. Alt ⁶⁴, I. Altsybeev ¹⁴³, W. Amend⁶⁴, M.N. Anaam ⁶, F. Anastasopoulos⁹⁴, E.C. Anderssen⁷⁴, C. Andrei ⁴⁵, D. Andreou ⁸⁴, A. Andronic ¹³⁸, M.T. Angelsmark⁷⁵, V. Anguelov ⁹⁴, A. Anjam⁹⁴, F. Antinori ⁵⁴, P. Antonioli ⁵¹, N. Apadula ⁷⁴, L. Aphecetche ¹⁰⁴, H. Appelshäuser ⁶⁴, V. Aprodu⁴⁵, C. Arata ⁷³, M. Arba⁵², S. Arcelli ²⁵, M. Aresti ⁵², R. Arnaldi ⁵⁶, J.G.M.C.A. Arneiro ¹¹¹, O.W. Arnold⁹⁵, I.C. Arsene ¹⁹, M. Arslandok ¹⁴⁰, P. Atkinson⁸⁵, A. Augustinus ³², R. Averbeck ⁹⁸, A. Ayala Pabon¹¹¹, M.D. Azmi¹⁵, C. Azzan⁵⁷, R. Baccomi⁵⁷, A. Badalà ⁵³, J. Bae ¹⁰⁵, Y.W. Baek ⁴⁰, X. Bai ¹²⁰, R. Bailhache ⁶⁴, Y. Bailung ⁴⁸, D. Baitinger⁹⁴, A. Balbino ²⁹, C. Baldanza⁵¹, A. Baldisseri ¹³⁰, B. Balis ², M. Ball⁴², D. Banerjee ⁴, Z. Banoo ⁹¹, R. Barbera ²⁶, P. Barberis⁵⁶, F. Barile ³¹, L. Barioglio ⁹⁵, M. Barlou⁷⁸, G.G. Barnaföldi ¹³⁹, L.S. Barnby ⁸⁵, V. Barret ¹²⁷, L. Barreto ¹¹¹, C. Bartels ¹¹⁸, K. Barth ³², R.G.E. Barthel⁸⁴, E. Bartsch ⁶⁴, F. Baruffaldi ²⁷, N. Bastid ¹²⁷, S. Basu ⁷⁵, G. Batigne ¹⁰⁴, D. Battistini ⁹⁵, B. Batyunya ¹⁴⁴, D. Bauri⁴⁷, J.L. Bazo Alba ¹⁰², I.G. Bearden ⁸³, C. Beattie ¹⁴⁰, P. Becht ⁹⁸, D. Behera ⁴⁸, I. Belikov ¹²⁹, A.D.C. Bell Hechavarria ¹³⁸, F. Bellini ²⁵, R. Bellwied ¹¹⁵, S. Belokurova ¹⁴³, V. Belyaev ¹⁴³, A. Benato⁵⁴, G. Bencedi ¹³⁹, M. Benettoni ⁵⁴, J.L. Beney¹⁰⁴, F. Benotto⁵⁶, S. Beole ²⁴, Y. Berdnikov ¹⁴³, A. Berdnikova ⁹⁴, M.E. Berger⁹⁵, L. Bergmann ⁹⁴, D. Berzano ³², M.G. Besoiu ⁶³, L. Betev ³², N. Bez⁵⁴, P.P. Bhaduri ¹³⁵, A. Bhasin ⁹¹, M.A. Bhat ⁴, B. Bhattacharjee ⁴¹, A.S. Bhatti ¹³, M.F. Bhopal¹³, N. Bialas⁶⁴, P. Białas⁶⁴, L. Bianchi ²⁴, N. Bianchi ⁴⁹, J. Bielčík ³⁵, J. Bielčíková ⁸⁶, J. Biernat ¹⁰⁸, A.P. Bigot ¹²⁹, A. Bilandzic ⁹⁵, G. Biro ¹³⁹, S. Biswas ⁴, N. Bize ¹⁰⁴, J.T. Blair ¹⁰⁹, D. Blau ¹⁴³, M.B. Blidaru ⁹⁸, N. Bluhme³⁸, C. Blume ⁶⁴, G. Boca ²¹,⁵⁵, F. Bock ⁸⁷, T. Bodova ²⁰, A. Bogdanov¹⁴³, S. Boi ²², J. Bok ⁵⁸, L. Boldizsár ¹³⁹, M. Bombara ³⁷, P.M. Bond ³², A. Bonnevaux¹²⁸, G. Bonomi ¹³⁴,⁵⁵, M. Bonora³², H. Borel ¹³⁰, A. Borissov ¹⁴³, F. Borotto Dalla Vecchia⁵⁶, A.G. Borquez Carcamo ⁹⁴, M. Borri⁸⁵, V. Borshchov³, H. Bossi ¹⁴⁰, E. Botta ²⁴, S. Bouvier¹⁰⁴, Y.E.M. Bouziani ⁶⁴, L. Boynton¹¹⁸, L. Bratrud ⁶⁴, P. Braun-Munzinger ⁹⁸, M. Bregant ¹¹¹, C. Britton⁸⁷, G. Brouwer⁸⁴, M. Broz ³⁵, E.J. Brücken ⁴³, S. Brucker⁹⁴, G. Brulin¹³¹, E. Bruna ⁵⁶, O. Brunasso Cattarello⁵⁶, G.E. Bruno ⁹⁷,³¹, M.D. Buckland ²³, D. Budnikov ¹⁴³, H. Buesching ⁶⁴, S. Bufalino ²⁹, O. Bugnon¹⁰⁴, P. Buhler ¹⁰³, J.-M. Buhour¹,¹⁰⁴, P. Buncic³², N. Burmasov ¹⁴³, Z. Buthelezi ⁶⁸,¹²³, S.A. Bysiak¹⁰⁸, J.C. Cabanillas Noris ¹¹⁰, M. Cai ⁶, H. Caines ¹⁴⁰, A. Caliva ⁹⁸, E. Calvo Villar ¹⁰², J.M.M. Camacho ¹¹⁰, P. Camerini ²³, F.D.M. Canedo ¹¹¹, M. Carabas ¹²⁶, G. Caragheorgheopol⁴⁵, A.A. Carballo ³², W. Carena³², P. Cariola⁵⁰, F. Carnesecchi ³², R. Caron ¹²⁸, L.A.D. Carvalho ¹¹¹, G. Castelneau¹²⁸, J. Castillo Castellanos ¹³⁰, A.J. Castro¹²², F. Catalano ²⁴, B. Cavalcante De Souza Sanches¹¹¹, D. Cavazza⁵¹, C. Ceballos Sanchez ¹⁴⁴, I. Chakaberia ⁷⁴, P. Chakraborty ⁴⁷, S. Chandra ¹³⁵, S. Chapeland ³², M. Chartier ¹¹⁸, S. Chattopadhyay ¹³⁵, S. Chattopadhyay ¹⁰⁰, P. Chatzidaki ⁹⁴, T.G. Chavez ⁴⁴, T. Cheng ⁹⁸,⁶, C. Cheshkov ¹²⁸, B. Cheynis ¹²⁸, V. Chibante Barroso ³², D.D. Chinellato ¹¹², E.S. Chizzali ¹¹,⁹⁵, J. Cho ⁵⁸,
S. Cho ⁵⁸, P. Chochula ³², P. Christakoglou ⁸⁴, C.H. Christensen ⁸³, S.G. Christensen⁵,
P. Christiansen ⁷⁵, T. Chujo ¹²⁵, M. Ciacco ²⁹, C. Cicalo ⁵², F. Cindolo ⁵¹, M.R. Ciupek⁹⁸,
N.J. Clague⁸⁵, G. Clai¹¹¹,⁵¹, O.A. Clausse¹²⁹, L.G. Clonts⁸⁷, F. Colamaria ⁵⁰, J.S. Colburn¹⁰¹,
D. Colella ⁹⁷,³¹, S. Coli ⁵⁶, A. Collu⁷⁴, M. Colocci ²⁵, M. Concas ⁴⁵,⁵⁶, G. Conesa Balbastre ⁷³,
Z. Conesa del Valle ¹³¹, G. Contin ²³, J.G. Contreras ³⁵, M.L. Coquet ¹³⁰, T.M. Cormier¹,⁸⁷,
Y. Corrales Morales ⁵⁶, P. Cortese ¹³³,⁵⁶, M.R. Cosentino ¹¹³, F. Costa ³², S. Costanza ²¹,⁵⁵,
C. Cot ¹³¹, G. Cotto²⁴, J. Crkovská ⁹⁴, P. Crochet ¹²⁷, J.R. Crowley⁵, R. Cruz-Torres ⁷⁴,
E. Cuautle⁶⁵, P. Cui ⁶, R.W. Da Silva¹¹¹, A. Dainese ⁵⁴, J.B. Dainton¹¹⁸, E. Danè⁴⁹, M.C. Danisch ⁹⁴,
A. Danu ⁶³, A. Das ¹⁰⁰, D. Das¹⁰⁰, D. Das¹⁰⁰, P. Das ⁸⁰, P. Das ⁴, S. Das ⁴, A.R. Dash ¹³⁸,
S. Dash ⁴⁷, R.M.H. David⁴⁴, A. De Caro ²⁸, D. De Carvalho¹¹¹, G. de Cataldo ⁵⁰, L. De Cilladi ²⁴,
J. de Cuveland³⁸, A. De Falco ²², D. De Gruttola ²⁸, N. De Marco ⁵⁶, C. De Martin ²³, S. De
Pasquale ²⁸, P. De Remigis ⁵⁶, G. De Robertis ⁵⁰, R. Deb ¹³⁴, S. Deb ⁴⁸, R.J. Debski ²,
W. Degraw⁸², A. Deisting ⁹⁴, K.R. Deja¹³⁶, R. Del Grande ⁹⁵, G. Dellacasa⁵⁶, R.M. Della Negra¹²⁸,
L. Dello Stritto ²⁸, W. Deng ⁶, P. Dhankher ¹⁸, D. Di Bari ³¹, A. Di Mauro ³², R.A. Diaz ¹⁴⁴,⁷,
T. Dietel ¹¹⁴, Y. Ding ¹²⁸,⁶, S. Dittrich ⁶⁴, R. Divià ³², D.U. Dixit ¹⁸, Ø. Djuvsland²⁰,
U. Dmitrieva ¹⁴³, A.L. Do Couto¹¹¹, A. Dobrin ⁶³, C.M. Domingues Goncalves¹³¹, B. Dönigus ⁶⁴,
J.M. Dubinski ¹³⁶, A. Dubla ⁹⁸, S. Dudi ⁹⁰, F. Dumitrache⁵⁶, P. Dupieux ¹²⁷, M. Durkac¹⁰⁷,
V. Duta⁴⁵, N. Dzalaiova¹², T.M. Eder ¹³⁸, R.J. Ehlers ⁷⁴, V.N. Eikeland²⁰, F. Eisenhut ⁶⁴, D. Elia ⁵⁰,
M.J. Engel⁴², J.B. Eppler⁶⁴, B. Erazmus ¹⁰⁴, F. Ercolessi ²⁵, F. Erhardt ⁸⁹, M.N. Ericson⁸⁷,
M.R. Ersdal²⁰, B. Espagnon ¹³¹, G. Eulisse ³², D. Evans ¹⁰¹, S. Evdokimov ¹⁴³, N.D.B. Ezell ⁸⁷,
L. Fabbietti ⁹⁵, M. Faggin ²⁷, J. Faivre ⁷³, D. Falchieri ⁵¹, F. Fan ⁶, W. Fan ⁷⁴, A. Fantoni ⁴⁹,
M. Fasel ⁸⁷, P. Fecchio²⁹, A. Feliciello ⁵⁶, G. Feofilov ¹⁴³, J. Ferencei⁸⁶, A. Fernández Téllez ⁴⁴,
L. Ferrandi ¹¹¹, M.B. Ferrer ³², A. Ferrero ¹³⁰, C. Ferrero ⁵⁶, A. Ferretti ²⁴, A. Festanti ³²,
V.J.G. Feuillard ⁹⁴, F. Fichera²⁶, V. Filova ³⁵, D. Finogeev ¹⁴³, F.M. Fionda ⁵², G. Fiorenza³²,⁵⁰,
E. Flatland³², F. Flor ¹¹⁵, A.N. Flores ¹⁰⁹, C. Flouzat¹³⁰, S. Foertsch ⁶⁸, G. F"ohner⁹⁴, I. Fokin ⁹⁴,
S. Fokin ¹⁴³, E. Fragiacomo ⁵⁷, E. Frajna ¹³⁹, A. Franco ⁵⁰, U. Frankenfeld⁹⁸, J.P. Fransen⁸⁴,
U. Fuchs ³², N. Funicello ²⁸, C. Furget ⁷³, A. Furs ¹⁴³, T. Fusayasu ⁹⁹, E. Futo ¹³⁹,
J.J. Gaardhøje ⁸³, M. Gagliardi ²⁴, A.M. Gago ¹⁰², D. Gajanana ⁸⁴, A. Gal¹²⁹, A. Galdames
Perez³², S. Gallian⁵⁶, C.D. Galvan ¹¹⁰, D.R. Gangadharan ¹¹⁵, P. Ganoti ⁷⁸, C. Gao⁶,
C. Garabatos ⁹⁸, J.R.A. Garcia ⁴⁴, E. Garcia-Solis ⁹, K. Garg ¹⁰⁴, C. Gargiulo ³², L. Garizzo⁵⁴,
K. Garner¹³⁸, P. Gasik ⁹⁸, A. Gautam ¹¹⁷, M.B. Gay Ducati ⁶⁶, T. Geiger⁶⁴, A.L. Gera¹³⁹,
M. Germain ¹⁰⁴, M. Gheata³², A. Ghimouz¹²⁵, C. Ghosh¹³⁵, M. Giacalone ⁵¹,²⁵, P. Giubellino ⁹⁸,⁵⁶,
P. Giubilato ²⁷, A.M.C. Glaenzer ¹³⁰, P. Glässel ⁹⁴, E. Glimos ¹²², M. Goffe ¹²⁹, D.J.Q. Goh⁷⁶,
V. Gonzalez ¹³⁷, M. Gorgon ², S. Gotovac³³, A.M. Grabas¹³⁰, V. Grabski ⁶⁷, O.A. Grachov ¹³⁷,
L.K. Graczykowski ¹³⁶, A.F. Grant⁸⁵, E. Grecka ⁸⁶, A. Grein⁶⁴, L. Greiner ⁷⁴, A. Grelli ⁵⁹,
C. Grigoras ³², V. Grigoriev ¹⁴³, S. Grigoryan ¹⁴⁴,¹, A. Grimaldi²⁶, F. Grosa ³²,
J.F. Grosse-Oetringhaus ³², R. Grosso ⁹⁸, D. Grund ³⁵, A.E. Guard⁵, G.G. Guardiano ¹¹²,
R. Guernane ⁷³, M. Guilbaud ¹⁰⁴, M.J. Guillamet¹⁰⁴, F. Guilloux¹³⁰, M. Gul ⁹⁶, K. Gulbrandsen ⁸³,
T. Gündem ⁶⁴, T. Gunji ¹²⁴, W. Guo ⁶, C. Guo Hu ¹²⁹, A. Gupta ⁹¹, R. Gupta ⁹¹,
R. Gupta ⁴⁸, S.P. Guzman ⁴⁴, H. Guzzo Neves¹¹¹, L. Gyulai ¹³⁹, M.K. Habib⁹⁸, C. Hadjidakis ¹³¹,
F.U. Haider ⁹¹, H. Hamagaki ⁷⁶, A. Hamdi ⁷⁴, M. Hamid⁶, Y. Han ¹⁴¹, R. Hannigan ¹⁰⁹,
J.C. Hansen⁸³, M.R. Haque ¹³⁶, N. Hardi³², A. Harlenderova⁹⁸, J.W. Harris ¹⁴⁰, A. Harton ⁹,
H. Hassan ⁸⁷, S. Hassan ⁹⁶, D. Hatzifotiadou ⁵¹, P. Hauer ⁴², L.B. Havener ¹⁴⁰, S.T. Heckel ⁹⁵,
J.L. Hehner<sup>98</sup>, J. Heino<sup>43</sup>, E. Hellbär<sup>98</sup>, H. Helstrup<sup>34</sup>, M. Hemmer<sup>64</sup>, A. Herghelegiu<sup>45</sup>, T. Herman<sup>35</sup>, L. Hernandes da Costa Porto<sup>111</sup>, H.D. Hernandez Herrera<sup>111</sup>, T. Herold<sup>94</sup>, G. Herrera Corral<sup>8</sup>, F. Herrmann<sup>138</sup>, S. Herrmann<sup>128</sup>, K.F. Hetland<sup>34</sup>, B. Heybeck<sup>64</sup>, T.E. Hilden<sup>43</sup>, A. Hill<sup>85</sup>, H. Hillemanns<sup>32</sup>, C. Hills<sup>118</sup>, P. Hindley<sup>85</sup>, B. Hippolyte<sup>129</sup>, F.W. Hoffmann<sup>70</sup>, B. Hofman<sup>59</sup>, B. Hohlweger<sup>84</sup>, G.H. Hong<sup>141</sup>, S. Hornung<sup>98</sup>, M. Horst<sup>95</sup>, A. Horzyk<sup>2</sup>, Y. Hou<sup>6</sup>, P. Hristov<sup>32</sup>, I. Hřivnáčová<sup>131</sup>, G. Huang<sup>6</sup>, C. Hughes<sup>122</sup>, P. Huhn<sup>64</sup>, L.M. Huhta<sup>116</sup>, C.V. Hulse<sup>131</sup>, T.J. Humanic<sup>88</sup>, S. Hummel<sup>94</sup>, A. Hutson<sup>115</sup>, D. Hutter<sup>38</sup>, J.P. Iddon<sup>32,118</sup>, S. Igelkin<sup>143</sup>, P. Ijzermans<sup>32</sup>, R. Ilkaev<sup>143</sup>, H. Ilyas<sup>13</sup>, M.A. Imhoff<sup>129</sup>, M. Imre<sup>131</sup>, M. Inaba<sup>125</sup>, G.M. Innocenti<sup>32</sup>, M. Ippolitov<sup>143</sup>, A. Isakov<sup>86</sup>, T. Isidori<sup>117</sup>, M.S. Islam<sup>100</sup>, D. Ivanischchev<sup>143</sup>, M. Ivanov<sup>98</sup>, M. Ivanov<sup>12</sup>, V. Ivanov<sup>143</sup>, M. Jablonski<sup>2</sup>, B. Jacak<sup>74</sup>, N. Jacazio<sup>32</sup>, P.M. Jacobs<sup>74</sup>, S. Jadlovska<sup>107</sup>, J. Jadlovsky<sup>107</sup>, S. Jaelani<sup>82</sup>, L. Jaffe<sup>38</sup>, J.N. Jager<sup>64</sup>, C. Jahnke<sup>112</sup>, M.J. Jakubowska<sup>136</sup>, M.A. Janik<sup>136</sup>, T. Janson<sup>70</sup>, M. Jercic<sup>89</sup>, S. Jia<sup>10</sup>, A.A.P. Jimenez<sup>65</sup>, T. Johnson<sup>74</sup>, B. Joly<sup>127</sup>, F. Jonas<sup>87</sup>, F. Jouve<sup>127</sup>, J.M. Jowett<sup>32,98</sup>, J. Jung<sup>64</sup>, M. Jung<sup>64</sup>, A. Junique<sup>32</sup>, A. Jusko<sup>101</sup>, D. Just<sup>64</sup>, M.J. Kabus<sup>32,136</sup>, J. Kaewjai<sup>106</sup>, P. Kalinak<sup>60</sup>, A.S. Kalteyer<sup>98</sup>, A. Kalweit<sup>32</sup>, E. Kangasaho<sup>43</sup>, V. Kaplin<sup>143</sup>, A. Karasu Uysal<sup>72</sup>, D. Karatovic<sup>89</sup>, O. Karavichev<sup>143</sup>, T. Karavicheva<sup>143</sup>, L. Karayan<sup>98</sup>, P. Karczmarczyk<sup>136</sup>, E. Karpechev<sup>143</sup>, U. Kebschull<sup>70</sup>, R. Keidel<sup>142</sup>, D.L.D. Keijdener<sup>59</sup>, M. Keil<sup>32</sup>, B. Ketzer<sup>42</sup>, Z. Khabanova<sup>84</sup>, S.S. Khade<sup>48</sup>, A.M. Khan<sup>6</sup>, H. Khan<sup>13</sup>, S. Khan<sup>15</sup>, A. Khanzadeev<sup>143</sup>, Y. Kharlov<sup>143</sup>, A. Khatun<sup>117,15</sup>, A. Khuntia<sup>108</sup>, M.B. Kidson<sup>114</sup>, B. Kileng<sup>34</sup>, B. Kim<sup>105</sup>, C. Kim<sup>16</sup>, D.J. Kim<sup>116</sup>, E.J. Kim<sup>69</sup>, J. Kim<sup>141</sup>, J.S. Kim<sup>40</sup>, J. Kim<sup>69</sup>, M. Kim<sup>18,94</sup>, S. Kim<sup>17</sup>, T. Kim<sup>141</sup>, K. Kimura<sup>92</sup>, S. Kirsch<sup>64</sup>, I. Kisel<sup>38</sup>, S. Kiselev<sup>143</sup>, A. Kisiel<sup>136</sup>, J.P. Kitowski<sup>2</sup>, J.L. Klay<sup>5</sup>, J. Klein<sup>32</sup>, S. Klein<sup>74</sup>, C. Klein-Bösing<sup>138</sup>, M. Kleiner<sup>64</sup>, T. Klemenz<sup>95</sup>, S. Klewin<sup>94</sup>, A. Kluge<sup>32</sup>, A.G. Knospe<sup>115</sup>, C. Kobdaj<sup>106</sup>, T. Kollegger<sup>98</sup>, A. Kondratyev<sup>144</sup>, N. Kondratyeva<sup>143</sup>, E. Kondratyuk<sup>143</sup>, J. Konig<sup>64</sup>, S.A. Konigstorfer<sup>95</sup>, P.J. Konopka<sup>32</sup>, G. Kornakov<sup>136</sup>, M. Korwieser<sup>95</sup>, S.D. Koryciak<sup>2</sup>, E. Koskinen<sup>43</sup>, A. Kotliarov<sup>86</sup>, V. Kovalenko<sup>143</sup>, M. Kowalski<sup>108</sup>, V. Kozhuharov<sup>36</sup>, M.J. Kraan<sup>84</sup>, I. Králik<sup>60</sup>, A. Kravčáková<sup>37</sup>, L. Krcal<sup>32,38</sup>, L. Kreis<sup>98</sup>, M. Krivda<sup>101,60</sup>, F. Krizek<sup>86</sup>, K. Krizkova Gajdosova<sup>35</sup>, M. Kroesen<sup>94</sup>, M. Krüger<sup>64</sup>, D.M. Krupova<sup>35</sup>, E. Kryshen<sup>143</sup>, V. Kučera<sup>32</sup>, T. Kugathasan<sup>32</sup>, C. Kuhn<sup>129</sup>, P.G. Kuijer<sup>84</sup>, T. Kumaoka<sup>125</sup>, D. Kumar<sup>135</sup>, L. Kumar<sup>90</sup>, N. Kumar<sup>90</sup>, S. Kumar<sup>31</sup>, S. Kundu<sup>32</sup>, P. Kurashvili<sup>79</sup>, A. Kurepin<sup>143</sup>, A.B. Kurepin<sup>143</sup>, R.K. Kuriakose<sup>68</sup>, A. Kuryakin<sup>143</sup>, S. Kushpil<sup>86</sup>, J. Kvapil<sup>101</sup>, M.J. Kweon<sup>58</sup>, J.Y. Kwon<sup>58</sup>, Y. Kwon<sup>141</sup>, B.Y. Ky<sup>131</sup>, S.L. La Pointe<sup>38</sup>, P. La Rocca<sup>26</sup>, N. Lacalamita<sup>50</sup>, P. Lafarguette<sup>127</sup>, Y.S. Lai<sup>74</sup>, A. Lakrathok<sup>106</sup>, M. Lamanna<sup>32</sup>, R. Lang<sup>95</sup>, R. Langoy<sup>121</sup>, P. Larionov<sup>32</sup>, E. Laudi<sup>32</sup>, L. Lautner<sup>32,95</sup>, R. Lavicka<sup>103</sup>, T. Lazareva<sup>143</sup>, C. Le Galliard<sup>131</sup>, R. Lea<sup>134,55</sup>, A. Lebedev<sup>98</sup>, G. Ledey<sup>32</sup>, H. Lee<sup>105</sup>, T. Lee<sup>85</sup>, G. Legras<sup>138</sup>, J. Lehrbach<sup>38</sup>, T.M. Lelek<sup>2</sup>, R.C. Lemmon<sup>85</sup>, I. León Monzón<sup>110</sup>, M.M. Lesch<sup>95</sup>, Y. Lesenechal<sup>32</sup>, E.D. Lesser<sup>18</sup>, M. Lettrich<sup>32,95</sup>, P. Lévai<sup>139</sup>, X. Li<sup>10</sup>, X.L. Li<sup>6</sup>, F. Librizzi<sup>53</sup>, F. Liebske<sup>64</sup>, J. Lien<sup>121</sup>, R. Lietava<sup>101</sup>, I. Likmeta<sup>115</sup>, B. Lim<sup>24,16</sup>, S.H. Lim<sup>16</sup>, V. Lindenstruth<sup>38</sup>, A. Lindner<sup>45</sup>, S.W. Lindsay<sup>118</sup>, C. Lippmann<sup>98</sup>, V. Litichevskyi<sup>43</sup>, A. Liu<sup>18</sup>, D.H. Liu<sup>6</sup>, J. Liu<sup>118</sup>, H.M. Ljunggren<sup>75</sup>, W.J. Llope<sup>137</sup>, I.M. Lofnes<sup>20</sup>, C. Loizides<sup>87</sup>, S. Lokos<sup>108</sup>, A. Lombardi Campos<sup>111</sup>, L. Lombardo<sup>IV,56</sup>, J. Lomker<sup>59</sup>, P. Loncar<sup>33</sup>, J.A. Lopez<sup>94</sup>, X. Lopez<sup>127</sup>, E. López Torres<sup>7</sup>, P. Lu<sup>98,120</sup>, J.R. Luhder<sup>138</sup>, M. Lunardon<sup>27</sup>, G. Luparello<sup>57</sup>, M. Lupi<sup>32</sup>, Y.G. Ma<sup>39</sup>, A. Maevskaya<sup>143</sup>, M. Mager<sup>32</sup>, S.M. Mahmood<sup>19</sup>,
T. Mahmoud⁴², A. Maire ID 129, R.D. Majka¹,¹⁴⁰, M.V. Makariev ID 36, M. Malaev ID 143, G. Malfattore ID 25, N.M. Malik ID 91, Q.W. Malik¹⁹, S.K. Malik ID 91, L. Malinina ID VII,¹⁴⁴, D. Mal'Kevich ID 143, D. Mallick ID 80, N. Mallick ID 48, A. Manafov⁹⁸, G. Mandaglio ID 30,53, S.K. Mandal ID 79, S.P. Manen¹²⁷, V. Manko ID 143, F. Manso ID 127, V. Manzari ID 50, Y. Mao ID 6, M. Marchisone ID 128, G.V. Margagliotti ID 23, A. Margotti ID 51, A. Marín ID 98, C. Markert ID 109, G. Markey⁸⁵, D. Marras⁵², P. Martinengo ID 32, J.L. Martinez¹¹⁵, M.I. Martínez ID 44, S. Martinez¹⁰⁴, G. Martínez García ID 104, T.A. Martins¹¹¹, S. Masciocchi ID 98, M. Masera ID 24, A. Masoni ID 52, L. Massacrier ID 131, A. Mastroserio ID 132,50, A.M. Mathis ID 95, B.S. Mathon¹³¹, O. Matonoha ID 75, Y. Matsuyama⁷⁶, P.F.T. Matuoka¹¹¹, A. Matyja ID 108, C. Mayer ID 108, A.L. Mazuecos ID 32, G. Mazza⁵⁶, D. Mazzaro⁵⁴, F. Mazzaschi ID 24, M. Mazzilli ID 32, L. McAlpine³², J.E. Mdhluli ID 123, A.F. Mechler⁶⁴, Y. Melikyan ID 43,143, A. Menchaca-Rocha ID 67, E. Meninno ID 103,28, A.S. Menon ID 115, M. Meres ID 12, P. Mereu ID 56, S. Mhlanga¹¹⁴,68, Y. Miake¹²⁵, L. Micheletti ID 56, L.C. Migliorin¹²⁸, D.L. Mihaylov ID 95, K. Mikhaylov ID 144,143, N.J. Miller⁵, A.N. Mishra ID 139, D. Miśkowiec ID 98, T. Mittelstaedt⁹⁴, A. Modak ID 4, A.P. Mohanty ID 59, B. Mohanty⁸⁰, M. Mohisin KhanV,15, M.A. Molander ID 43, L.S. Montali¹¹¹, D.M. Moraes¹¹¹, J. Morant³², Z. Moravcova ID 83, C. Mordasini ID 95, D.A. Moreira De Godoy ID 138, F. Morel¹²⁹, T. Morhardt⁹⁸, I. Morozov ID 143, P. Morral⁸⁵, A. Morsch ID 32, T. Mrnjavac ID 32, V. Muccifora ID 49, S. Muhuri ID 135, S.O. Muley⁹⁴, J.D. Mulligan ID 74, A. Mulliri²², M.G. Munhoz ID 111, K. Münnen ID 42, R.H. Munzer ID 64, H. Murakami ID 124, M.R.M. Murray⁵, S. Murray ID 114, L. Musa ID 32, J. Musinsky ID 60, J.W. Myrcha ID 136, B. Naik ID 123, A.I. Nambrath ID 18, B.K. Nandi ID 47, R. Nania ID 51, E. Nappi ID 50, A.F. Nassirpour ID 75, H. Natal da Luz ID 111, A. Nath ID 94, C. Nattrass ID 122, M.N. Naydenov ID 36, A. Neagu¹⁹, R.A. Negrao De Oliveira⁶⁴, A. Negru¹²⁶, L. Nellen ID 65, S.V. Nesbo³⁴, G. Neskovic ID 38, D. Nesterov ID 143, B.S. Nielsen ID 83, E.G. Nielsen ID 83, S. Nikolaev ID 143, S. Nikulin ID 143, V. Nikulin ID 143, F. Noferini ID 51, S. Noh ID 11, P. Nomokonov ID 144, J. Norman ID 118, N. Novitzky ID 125, P. Nowakowski ID 136, A. Nyanin ID 143, J. Nystrand ID 20, M. Oberegger³², M. Ogino ID 76, A. Ohlson ID 75, V.A. Okorokov ID 143, J. Oleniacz ID 136, A.C. Oliveira Da Silva ID 122, T. Oliveira Weber¹¹¹, M.H. Oliver ID 140, A. Onnerstad ID 116, C. Oppedisano ID 56, A. Orlando⁴⁹, A. Ortiz Velasquez ID 65, A. Oskarsson⁷⁵, L. Österman⁷⁵, J. Ottnad⁴², J. Otwinowski ID 108, M. Oya⁹², K. Oyama ID 76, Y. Pachmayer ID 94, S. Padhan ID 47, D. Pagano ID 134,55, G. Paić ID 65, A. Palasciano ID 50, S. Panebianco ID 130, R. Panero⁵⁶, E. Paoletti⁴⁹, O. Parasole²⁶, H. Park ID 125, H. Park ID 105, J. Park ID 58, J.E. Parkkila ID 32, L. Passamonti⁴⁹, C. Pastore ID 50, S.P. Pathak¹¹⁵, R.N. Patra⁹¹, B. Paul ID 22, H. Pei ID 6, T. Peitzmann ID 59, F. Pellegrino³², X. Peng ID 6, M. Pennisi ID 24, A. Pepato ID 54, L.G. Pereira ID 66, D. Peresunko ID 143, G.M. Perez ID 7, S. Perrin ID 130, V. Peskov ID 64, Y. Pestov¹⁴³, V. Petráček ID 35, M. Petris⁴⁵, V. Petrov ID 143, M. Petrovici ID 45, C. Petta ID 26, R.P. Pezzi ID 104,66, S. Piano ID 57, P. Pichot¹⁰⁴, D. Pierluigi⁴⁹, M. Pikna ID 12, P. Pillot ID 104, O. Pinazza ID 51,32, L. Pinsky¹¹⁵, C. Pinto ID 95, S. Pisano ID 49, M. Płoskoń ID 74, M. Planinic⁸⁹, F. Pliquett⁶⁴, M.T. Poblocki³²,118, M.G. Poghosyan ID 87, B. Polichtchouk ID 143, S. Politano ID 29, N. Poljak ID 89, F. Pompei¹³⁷, A. Pop ID 45, S. Porteboeuf-Houssais ID 127, V. Pozdniakov ID 144, K.K. Pradhan ID 48, E. Prakasa ID 82, S.K. Prasad ID 4, S. Prasad ID 48, R. Preghenella ID 51, F. Prino ID 56, L. Prodan⁴⁵, M. Protsenko³, J.R. Pruitt⁵, C.A. Pruneau ID 137, I. Pshenichnov ID 143, M. Puccio ID 32, S. Pucillo ID 24, Z. Pugelova¹⁰⁷, C. Puggioni ID 52, E. Puleo²⁴, S. Qiu ID 84, L. Quaglia ID 24, R.E. Quishpe¹¹⁵, A. Rachevski ID 57, A.B. Radu⁴⁵, L. Radulescu⁴⁵, S. Ragoni ID 14, J. Rak¹¹⁶, A. Rakotozafindrabe ID 130, S. Rambeaud⁶⁴, L. Ramello ID 133,56, F. Rami ID 129, S.A.R. Ramirez ID 44, R. Ramirez Jimenez⁶⁷, T.A. Rancien⁷³, M. Rasa ID 26, S.S. Räsänen ID 43, J. Rasson⁷⁴, R. Rath ID 51, V. Ratza⁴², M.P. Rauch ID 20, I. Ravasenga ID 84, K.F. Read ID 87,122, C. Reckziegel ID 113,