where $\Delta R_{ij}^2 = (\phi_i - \phi_j)^2 + (\eta_i - \eta_j)^2$ and $R^2$ is a parameter of the algorithm that sets the size scale of the jet clusters. At each iteration of the algorithm, the minimum of the *Separation List* ($d_{ij}, d_{iB}$) is found. If the minimum corresponds to $d_{aB}$ then object *a* is moved to the *Jet List* and removed from the *Object* and *Separation* lists. Otherwise, if the minimum is $d_{ab}$ then objects *a* and *b* are merged to form a new object (*k*) which is added to the *Object* and *Separation* lists while objects *a* and *b* are removed. This process continues until there are no objects left in the *Object List*. Merging is performed in the so-called "E-Scheme" by adding the 3-momenta of objects *a* and *b* to define $p_T, \phi, \eta$ for the new object, *k*. At most, *N* iterations are required to finish the algorithm and after each iteration, the number of objects is reduced by one. Minimum-finding at each iteration scales like the number of objects present at that iteration squared. Thus, the simplest version of the anti-$k_t$ algorithm scales with the number of objects as $N^3$.

Implementation of the algorithm in an FPGA takes advantage of parallelism to reduce the latency required at each iteration. For example, a binary tree minimum finder run on an *N* object list requires $\log_2 N$ steps to complete if overhead and timing delays are ignored. Additionally, the full *Separation List* ($d_{ij}, d_{iB}$) is not calculated. Rather, separate lists of object $p_T^{-2}$ or $p_T$ (the $p_T$ List) and the distance between object *i* and all other objects ($\Delta R_{ij}^2$) (the *Distance Lists*) are used. Calculating the *N* elements $p_{T,i}^{-2}$ requires a LUT and is probably most efficiently done during the clustering stage, which would also save on latency. The distances, $\Delta R_{ij}^2$, can be calculated in parallel but require $N(N-1)$ multiply operations. For *N* approaching 100 this would saturate the DSP units available on the target Xilinx UltraScale+ FPGAs. These calculations therefore have to be done in stages, increasing latency.

Minimisation of the *Separation List* is done in two stages in this scheme to avoid large $O(N^2)$ binary minimisation trees. In the first stage, the minima of each the *Distance Lists* ($\Delta R_{ij}^2$) are found in parallel. Then, these distance minima are multiplied by the relevant $p_{T,i}^{-2}$ and the minimum of that list is found. This identifies the two objects to be merged, or the jet to be removed. A large LUT must be used to transform the merged object 3-momentum into $p_T^{-2}, \phi, \eta$, and $2(N-2)$ multipliers are required to calculate the distances $\Delta R_{kj}^2$ between the new object *k* and the remaining objects in the list. This algorithm therefore uses a significant amount of LUT and DSP resources for even moderate values of *N*. Assuming realistic synchronisation/timing requirements, the latency per iteration in this scheme is: $\log_2(N-1) + \log_2(N+1) + 11$. At a 320 MHz clock speed the total latency exceeds 2.5 µs for the very modest value of *N* ~ 32. Taken together with the significant resources required to implement the algorithm, this disqualifies the *E-merging* version of the *Exact anti-k_t* algorithm as a viable option.

Thus, the goal becomes identifying tradeoffs in precision that could balance the latency restriction. Several modifications to the basic anti-$k_t$ algorithm have been proposed to improve its computational efficiency. Two that are of particular relevance to the anti-$k_t$ implementation on the GEP are: the Winner-Take-All (WTA) merging scheme [9.6], and the use of Tiling [9.7]. In the WTA merging scheme computational resources are saved by simply
summing the $p_T$'s of the two objects and assigning the $\phi, \eta$ position of the initial object with the larger $p_T$ to the merged object. Tiling effectively reduces the number of objects to consider ($N$) by dividing the area over which jet-finding is performed into smaller, overlapping tiles. In the following, we explore the use of the WTA-merging scheme, though Tiling is not ruled out as a future option.

The fact that object positions are not changed in the WTA merging scheme makes this method much more amenable to implementation in the GEP than the E-scheme. This has three consequences. First, if the *Distance Lists* ($\Delta R_{ij}^2$) are sorted before the first iteration, these sorted lists can be used unchanged (simply disabling removed objects) for all iterations. Using a parallel sorting algorithm (e.g., bitonic mergesort), $(\log_2 N)(\log_2(N) + 1)/2$ stages + timing/synchronisation overhead are required to do each $N$-element sort, all of which can be done in parallel. This removes one of the two minimisation stages per iteration required in the *Exact anti-$k_t$ – E-merging* scheme. Second, the LUT needed to calculate $p_T^{-2}$ of a merged object is reduced in size compared to that needed in the *Exact anti-$k_t$ – E-merging* scheme. Finally, $\Delta R_{kj}^2$ does not have to be calculated for a new merged object at each iteration because the position of the merged object is the same as one of its constituents. Thus both latency per iteration and FPGA resources required are reduced significantly in this scheme with respect to the *Exact anti-$k_t$ – E-merging* scheme. An estimate of the latency per iteration in the *Exact anti-$k_t$ – WTA* scheme has been made using known latencies for each of its component blocks (minimisation, LUT, etc) and including conservative assumptions of additional delays required to meet timing and synchronisation requirements. The dependence of this latency on the number of input clusters ($N$) is found to be $\log_2 N + \log_2(N+1) + 8$ per iteration, and is illustrated in Fig. 9.12.

<figure><img src="image_176.png" /><figcaption>Figure 9.12: Latency of the prototype anti-$k_t$ algorithm variants in units of FPGA clock cycles (ticks) for a nominal 320 MHz reference clock as a function of the number of clusters being processed ($N$). The right plot is a zoomed out version of the left plot.</figcaption></figure>

Significant latency and resource savings can be realised in an *Approximate anti-$k_T$ scheme* where the criterion for clustering is modified from that specified for anti-$k_t$. In this case, the list of object momenta ($p_{T,i}$) as well as the *Distance Lists* are sorted before iterations start, and the closest object to the highest $p_T$ object is merged with it unless $\Delta R_{ij}^2 > R^2$ in which
case the highest $p_T$ object is moved to the *Jet List*. The WTA scheme is used for merging. At each iteration the sorted $p_T$ List is used to reference the *Distance List* corresponding to the maximum $p_T$ object remaining. The minimum element of that (sorted) *Distance List* is then used to define the object to merge with the highest $p_T$ object (or to promote the highest $p_T$ object to the *Jet List*). No minimisations are required, and $p_{T,i}^{-2}$ does not need to be calculated in this scheme, meaning that no LUTs are needed. The latency per iteration is therefore approximately independent of $N$. It is estimated to be $2 \text{ceil}[\log_2(\log_2 N)] + 7$ per iteration based on the mathematical and list-management operations that are required at each iteration.

An estimate of the latency per iteration in each of the above schemes has been made by summing the known latencies for individual element blocks in each scheme (minimisation, sorting, multiplies, look-ups, etc) and adding synchronisation/timing delays between blocks. A comparison of input and latency estimates for the various jet-finding schemes is summarised in Table 9.2. FPGA resource requirements, particular the number of LUTs and DSP units needed, decrease substantially when moving from the *Exact anti-k_t - E-merging* to the *Exact anti-k_t - WTA* to the *Approximate anti-k_t* implementations. Although precise numerical estimates of resource usage are not yet available, it is apparent that the *Exact anti-k_t - E-merging* variant is not a viable option due to latency considerations given the number of input clusters expected even in the most aggressive Tiling schemes. However, the *Approximate anti-k_t* implementation appears to be feasible in terms of latency and FPGA resources, and assuming Tiling schemes will allow a reduction of $< N >$ to less than roughly 100. Studies of further optimisations in jet finding algorithms and their physics performance are ongoing.

<table><caption>Table 9.2: Comparison of the parameters used and latencies (in FPGA clock cycles for a nominal 320 MHz reference clock) for the initial sorting and per iteration estimated for various jet-finding schemes as a function of the number of clusters ($N$) sent from the clustering step. Index $i$ refers to individual cluster objects and runs from 1 to $N$.</caption><thead><tr><th>Scheme</th><th>From clustering</th><th>Latency for initial sort</th><th>Latency per iteration</th></tr></thead><tbody><tr><td>Exact anti-k<sub>t</sub> - E-merging</td><td>p<sub>x,i</sub>, p<sub>y,i</sub>, p<sub>z,i</sub><br>p<sub>T,i</sub>, p<sub>T,i</sub><sup>-2</sup><br>φ<sub>i</sub>, η<sub>i</sub></td><td>-</td><td>log<sub>2</sub>(N - 1) + log<sub>2</sub>(N + 1) + 11</td></tr><tr><td>Exact anti-k<sub>t</sub> - WTA</td><td>p<sub>T,i</sub>, p<sub>T,i</sub><sup>-2</sup><br>φ<sub>i</sub>, η<sub>i</sub></td><td>log<sub>2</sub>[N(N + 1)/2] +ceil[log<sub>2</sub>(log<sub>2</sub> N)] + 5</td><td>log<sub>2</sub> N + log<sub>2</sub>(N + 1) + 8</td></tr><tr><td>Approximate anti-k<sub>t</sub></td><td>p<sub>T,i</sub><br>φ<sub>i</sub>, η<sub>i</sub></td><td>log<sub>2</sub>[N(N + 1)/2] + 5</td><td>2 ceil[log<sub>2</sub>(log<sub>2</sub> N)] + 7</td></tr></tbody></table>

The inputs to these algorithm are not yet considered by this study. The list of topoclusters described in the previous subsection is a natural choice. However, there are potentially many topoclusters in each event which has a significant impact on the overall latency as shown in Fig. 9.12. Several methods are being studied to reduce the overall multiplicity of topoclusters including implementation of a Tiling procedure, applying minimum $p_T$
<figure><img src="image_177.png" /><figcaption>Figure 9.13: Average topocluster number density $\langle \delta N_{clus} / \delta \eta \rangle$ as a function of $\eta_{clus}$, for clusters with $p_{T,clus} > p_{T,min}$, for various $p_{T,min}$ values. Results are obtained using QCD dijet samples with $< \mu > = 200$.</figcaption></figure>

<figure><img src="image_178.png" /><figcaption>Figure 9.14: The average number of topo-clusters $\langle N_{clus}^{jet} \rangle$ in anti-$k_t$ jets reconstructed with $R = 0.4$ within $30 < p_{T,jet} < 40$ GeV as a function of $\mu$ using QCD dijet samples with $< \mu > = 200$ (left), and when selecting topo-clusters by $E_{T,clus} > 1$ GeV inside jets (right).</figcaption></figure>

requirements to the topoclusters, only running jet algorithms on topoclusters within RoI defined by the jets identified in the jFEX or the gFEX, or by building towers from the topo-clusters (so called TopoTowers). All of these options will be studied as part of the optimisation of the algorithm performance and resource usage. As an initial indication, studies of the impact of a minimum $p_T$ requirement were carried out for the offline [9.2]. Even a moderate $p_T$ requirement of 1 or 2 GeV can reduce the overall number of topoclusters by a factor of 50 or more as shown in Fig. 9.13. Such cuts have benefit by reducing low-energy topoclusters that can increase sensitivity to pileup as shown, for example in Fig. 9.14.

Such low-energy topoclusters are typically removed during studies of the substructure of large-radius jets including those that contain the decays of highly Lorentz-boosted particles
<figure><img src="image_179.png" /><figcaption>Figure 9.15: The anti-$k_t$ $R = 1.0$ leading jet mass distribution before (top) and after (bottom) jet trimming and pileup subtraction in dijet events and $Z' \to t\bar{t}$ (right) events with mean number of interactions per bunch crossing ($\mu$) of 0 (black closed circles), 80 (red closed squares), 140 (red open squares), 200 (black open crosses), and 300 (black closed crosses). Jets with $|\eta| < 1.2$ and $0.5 < pT < 1$ TeV are used. In the bottom plots, jets are trimmed with parameters $f_{cut} > 5\%$ and $R_{subjet} = 0.3$ and are corrected for pileup using area-based pileup subtraction.</figcaption></figure>

such as W, Z, H bosons, top quarks, and various exotica. One popular pileup suppression technique in such jets is called *trimming* where low-$p_T$ topoclusters in small jets (subjet radius $R_{subjet} = 0.3$) that make up less than 5% of the energy of the large-radius jet ($f_{cut} > 5\%$) are removed. An example of the signal improvements from such a technique is shown in Fig. 9.15. On the left are results from QCD dijets and on the right signals from $Z' \to t\bar{t}$. The upper plots are without any trimming while the lower plots include it. Trimming low $p_T$ contributions is effective in removing the impact of pileup and improving the separation between signal and background. Further physics performance is discussed in Section 6.6.
## 9.4 Physical Realisation

### 9.4.1 Architecture

This section describes the design of the Global Trigger system, a multi-layer hardware scheme to multiplex, process, and demultiplex the input data and output trigger information. Use of a common hardware design throughout minimises the complexity of the firmware and eases long-term maintenance; this design being the Global Common Module (GCM). This section motivates the required number of GCMs at each layer of the global processing chain and describes the requirements to support the outside interfaces to the Global Trigger, as well as internal interfaces between GCMs, concluding with a description of a proof-of-principle implementation of the GCM design.

Each layer in the system consists of a number of processing nodes, where each node is based on a large FPGA with extensive input and output capabilities. Previous experience on ATLAS designing ATCA modules and on CMS designing μTCA modules, along with consideration of the power and cooling environment within the USA15 racks, suggests ATCA as a suitable form factor with a single ATCA Front Board supporting two independent nodes along with shared infrastructure. The GCM design is therefore an ATCA Front Board with two large FPGAs, where each FPGA can be configured as a MUX, GEP, or CTP Interface node as required, along with a central processing chip (such as a ZYNQ MPSoC) to allow for monitoring, control and readout. A schematic block-diagram view of the Global Trigger is given in Fig. 9.16. This figure demonstrates conceptually how the I/O interfaces, multiplexing, event processing, and demultiplexing can be achieved with a single-module design. Each step requires a significant number of optical links, onboard processing capabilities and, central control.

### 9.4.2 Required Number of Global Common Modules (GCMs)

There are two drivers on the number of nodes in the GCM design and its implementation in the Global Trigger: first, the number of MUX inputs (both the number of inputs per MUX and the number of MUX modules) and second, the number of required GEP modules, which defines the event pipeline. The number of real-time input fibres to the system add up to 2312 in the baseline design as summarised in Table 9.3. For symmetric input and output link speeds the number of input links on a MUX node should match the number of outputs, this latter being the number of GEP nodes. Balancing the number of MUX nodes and GEP nodes to keep the data flow in the multiplexing mesh network symmetric then gives 48 MUX nodes and GEP nodes to symmetrically distribute the 2312 input links.

High density optical connectors, a prerequisite for the design, target 12-fibre ribbons so there is good sense in keeping either the number of MUX nodes or GEP nodes a multiple
<figure><img src="image_180.png" /><figcaption>Figure 9.16: Schematic diagram of the Global Common Module physical design and interfaces.</figcaption></figure>

of 12. Allowing the possibility of additional inputs, most particularly to allow the evolution described in Chapter 14 leads to the baseline design having 48 GEP nodes (24 GEP modules). With a large fraction of the input links running at speeds lower than the internal multiplexing links, this results in the 47 MUX nodes (24 MUX modules) listed in Table 9.3, leaving 1 MUX node spare in the baseline fibre plant. Further inputs can be accommodated by increasing the number of MUX nodes above 48 and keeping the number of GEP nodes at 48. To this end the ribbons into the GEP nodes foresee a maximum 72 MUX nodes.

### 9.4.3 Required Number of Links on each GCM

The three use cases of the GCM, as MUX, GEP and as CTP Interface, each bring different requirements on the number of input and output links.

For the MUX the maximum input loading on a node in Table 9.3 is 64. Mapping this onto 12-fibre ribbons suggests up to 72 as the number of inputs per node. Each node also requires an output fibre to each GEP node, which would give 48 output fibres per node in the baseline design. With muon information being on the critical path for latency (see Section 9.5 below) there is mileage in supporting double links from the MUCTPI MUX nodes,
<table><caption>Table 9.3: Summary of the real-time optical fibre interfaces to the Global Trigger from input data sources and output to CTP.</caption><thead><tr><th>Input source</th><th>Link Speed (Gb/s)</th><th>Expected fibres</th><th>fibres/node</th><th>Nodes</th></tr></thead><tbody><tr><td>eFEX</td><td>11.2</td><td>384</td><td>64</td><td>6</td></tr><tr><td>jFEX</td><td>12.8</td><td>192</td><td>48</td><td>4</td></tr><tr><td>fFEX</td><td>12.8</td><td>48</td><td>48</td><td>1</td></tr><tr><td>gFEX</td><td>11.2</td><td>24</td><td>24</td><td>1</td></tr><tr><td>MUCTPI</td><td>9.6 - 12.8</td><td>96</td><td>48</td><td>2</td></tr><tr><td>LASP</td><td>25.8</td><td>1484</td><td>48</td><td>31</td></tr><tr><td>TPPr</td><td>11.2</td><td>64</td><td>64</td><td>1</td></tr><tr><td>ZDC</td><td>11.2</td><td>20</td><td>20</td><td>1</td></tr><tr><td></td><td>TOTAL</td><td>2312</td><td></td><td>47</td></tr><tr><td>Output destination</td><td></td><td></td><td></td><td></td></tr><tr><td>CTP</td><td>9.6</td><td>12</td><td>12</td><td>1</td></tr></tbody></table>

or alternatively sharing the MUCTPI links over more nodes to reduce the multiplexed data transfer time. The minimum number of outputs per node is then 48, with the possibility of 96 being desirable.

For the GEP the number of inputs on a node in the baseline design is 47. In the two-level Level-0/Level-1 system there will be additional inputs from L1Track and from the Level-0 CTP which will exhaust the number of links on 4 12-fibre ribbons. Future-proofing then gives 60 or 72 links as the minimum number of inputs. Outputs per node are minimal, in the baseline a single fibre to the CTP Interface, possibly two to reduce latency on the critical path. Even in the two-level Level-0/Level-1 system the number of outputs will still fit on a single 12-fibre ribbon.

For the CTP Interface there is a single input from each GEP node, so 48 in total. Again there could be mileage in having dual inputs from each GEP node to reduce latency, so minimum number of inputs is 48 with the possibility of 96 being desirable. Output is a single 12-fibre ribbon to the CTP. The second half of the CTP Interface module could be idle, or latency considerations might split this functionality across both nodes, but in any case in the two-level Level-0/Level-1 system this second node could drive the outputs to the Level-1 CTP.

In addition to the above inputs and outputs per node, each GCM module has inputs from the TTC system via FELIX and readout outputs to FELIX. This amounts to an additional input ribbon and output ribbon per module. The number of fibres per module is summarised in Table 9.4. The minimum number of inputs that the generic GCM module needs to support is therefore 144+12 and outputs 96+12, in both cases with 192+12 a desirable maximum. Whilst the GCM module would be built with all the necessary sockets for the optical I/O only those needed would be populated on each actual module.
The same type of fibre management as on the Phase-I FEX modules is proposed. This consists of ribbons on-board bundled together and routed out through optical backplane connectors in ATCA Zone 3, thereby affording an element of protection to the delicate fibre infrastructure. The connectors used on the FEX modules will allow up to 24 ribbons, 288 fibres, to be routed in this manner, more than enough for any of the combinations of inputs and outputs foreseen.

<table><caption>Table 9.4: Summary of the maximum number of optical fibre interfaces for the GCM modules.</caption><thead><tr><th>Module</th><th>Fibre</th><th>Size</th><th>Number</th></tr></thead><tbody><tr><td rowspan="3">Multiplexer</td><td>Input from Source</td><td>2 × 72</td><td>144</td></tr><tr><td>Output to GEP</td><td>2 × 48(96)</td><td>96(192)</td></tr><tr><td>FELIX</td><td>2 × 12</td><td>24</td></tr><tr><td rowspan="3">Event Processor</td><td>Input from MUX</td><td>2 × 60</td><td>120</td></tr><tr><td>Output to CTP Interface</td><td>2 × 12</td><td>24</td></tr><tr><td>FELIX I/O</td><td>2 × 12</td><td>24</td></tr><tr><td rowspan="3">CTP Interface</td><td>Input from GEP</td><td>2 × 48(96)</td><td>96(192)</td></tr><tr><td>Output to CTP</td><td>1 × 12</td><td>12</td></tr><tr><td>FELIX I/O</td><td>2 × 12</td><td>24</td></tr></tbody></table>

### 9.4.4 GCM Requirements and Preliminary Design

Figure 9.17 contains a preliminary block diagram for the GCM. There are two primary features. First, each board would host two large FPGAs that are capable of providing a sufficient number of MGTs to support the optical interfaces as well as the required event processing/multiplexing demands. And second, each board would host a central processing chip (such as a ZYNQ MPSoC) to allow for monitoring, control and readout. In addition to the internal Random Access Memory (RAM) each of these chips would be interfaced with large RAM storage to allow for long-latency buffering of the input and output data. The strategy, as shown in Fig. 9.16, is that each FPGA would serve as one MUX or one GEP node, or as the CTP Interface. This choice defines the requirements for FPGAs to be utilised.

The proof of concept architecture sketched in Fig. 9.17 assumes 2017 technology to allow for realistic costing and design strategy. To satisfy the required number of optical links and on-board MGT interfaces, two candidate FPGAs were identified: the Xilinx Ultrascale+ VU9P and VU13P (each provides up to 104 MGTs capable of 30+ Gb/s serial transmission in the C2104 variant). The candidate for the control/readout chip was a Xilinx UltraScale+ ZYNQ (e.g. ZU17EG). The GCM module would require maximally 204 optical inputs and
204 optical outputs. These requirements can be achieved using 12-channel optical modules (e.g. Avago MiniPODs), which would require the board to host 34 optical modules. This figure further illustrates that an optional mezzanine could be included, taking advantage of 28 MGT links from the ZYNQ chip and allowing additional processing or readout capacity in specialised use cases. However, in the baseline design there is no use case for such a mezzanine.

<figure><img src="image_181.png" /><figcaption>Figure 9.17: Block diagram for the Global Common Module board.</figcaption></figure>

The GCM design is a natural evolution from the Phase-I gFEX hardware, and a proof of principle layout is shown in Fig. 9.18. This example design is based on a standard ATCA form factor and includes realistic footprints for the ATCA power entry and conversion, UltraScale+ and ZYNQ chips, IPMC mezzanine, miniPODs, GbE interfaces and other required components. Each of the chips are highlighted along with their primary board resources. This exercise demonstrates the basic feasibility of such a design within the ATCA form factor.

Building on the experience with the Phase-I eFEX and gFEX hardware the MPV power estimate is 350 W for the MUX variant and 375 W for the GEP variant, well within the 400 W slot limit (see Section 4.2). Experience with the eFEX cooling suggests that maintaining sensible FPGA temperatures in a single slot will not be difficult with suitable airflow management and heatsink design.

## 9.4.5 Fibre Plant

The same modular scheme as already successfully implemented in the CMS trigger, shown in Fig. 9.19, is proposed for the fibre plant. The 48-fibre bundle output from a MUX module is split out into 4 12-fibre ribbons and each ribbon is plugged into a separate 12 × 12 fibre
<figure><img src="image_182.png" /><figcaption>Figure 9.18: Example layout for the GCM board using the standard ATCA form factor.</figcaption></figure>

ribbon remapping module. The 72-fibre bundle input to a GEP module is similarly split into 6 12-fibre ribbons (2 additional ribbons to allow expansion in number of MUX nodes above 48) and plugged into the outputs from the appropriate remapping modules, with 4 × 4 remapping modules providing the full 48 × 48 mesh between MUX and GEP systems. A separate custom module can provide the concentrator mapping between GEP and CTP Interface.

9.4.6 Production Firmware Deployment Module

Modern firmware simulation tools are essential but not sufficient to guarantee fail-safe per-
formance of the firmware on the target hardware system. Therefore, in order to evaluate
various firmware blocks of the Global Trigger system a dedicated hardware platform is re-
quired by each group involved in the firmware development. For this purpose a number
of Production Firmware Deployment Modules (PFM) will be produced and distributed to
firmware developers. The PFM represents a slice of the GCM, which includes a processing
unit, a control FPGA and a number of optical modules. Such a configuration of the PFM
<figure><img src="image_183.png" /><figcaption>Figure 9.19: Dual 12 × 12 fibre ribbon remapping modules in a 2U enclosure installed in a standard LHC rack in Point-5. 8 such dual modules will be required on ATLAS.</figcaption></figure>

allows testing and debugging algorithm and infrastructure firmware together with the corresponding software, thus minimising the risk of firmware failures on the final system. In order to allow for early firmware development and evaluation, the PFM needs to be available well before the production GCM. With the production system in place, the PFM system will continue to be maintained for pre-commissioning of regular firmware upgrades during future operations.

## 9.5 Latency

The latency of the Global Trigger system has two primary components: the latency for the delivery of data inputs to the MUX system and that of processing and transmission of the Global Trigger decisions. As the arrival time of the muon information is significantly later than all other inputs this can be further factorised into the latency path for trigger signatures not depending on muons and those involving muon information. For the former these can be broken down as:

* arrival of data at MUX

* sorting and transmission of data from MUX to GEP

* contemporaneous spatially pipelined processing of incoming data, including topoclustering

* processing of trigger signatures which do not depend on jets or muons

* asynchronous/iterative jet algorithms

* processing of jet-based trigger signatures which do not depend on muons
In addition preprocessing can occur for those signatures which do require muon information, ready for merging with the muon information on its arrival, an obvious candidate being $E_T^{miss}$ where the bulk of the vector sums can be formed in advance.

The latency critical path involving muons through to transmission to the CTP can be similarly broken down as:

* arrival of muon information at MUX

* sorting and transmission from MUX to GEP

* contemporaneous processing of incoming data, such as muon isolation

* completion of trigger signatures depending on muons

* assemble Global Trigger TIP

* serial transmission of Global Trigger TIP from GEP to CTP Interface

* serial transmission of Global Trigger TIP from CTP Interface to CTP

The best estimates of each of these components is summarised in Table 9.5, illustrating both the individual components and also the total latency for each step following the same prescription for CBE and MPV as in Section 5.2.8 and Table 5.5.

### 9.5.1 Data arrival time at Multiplexer Processor

The LAr [9.4] and Tile [9.8] full-granularity inputs are anticipated to arrive in 1.425 – 1.7 µs CBE to MPV, followed by the L0Calo inputs (2.075 – 2.575 µs, see Section 7) and the L0Muon inputs at 4.46 – 5.26 µs (see Section 8). The difference in arrival times between the full-granularity calorimeter inputs and the L0Muon inputs provides a significant period for calorimeter pre-processing in the shadow of the muon latency.

### 9.5.2 Processing latency and budget

A key driver in the Global Trigger latency is the time taken to transfer data from the MUX to the GEP. In the case of the LAr LASP the link bandwidths are matched, so the transfer time is 48 BC (1.25 µs) in addition to the latency of transmission and encoding. In the case of the muon information the input bandwidth is significantly less, so this transfer time is just over 0.5 µs.

The contemporaneous processing of incoming data adds 0.1 to 0.2 µs in each case. The time for processing of trigger signatures which do not depend on jets or muons, along with the time for iterative or asynchronous jet processing, remains hidden in the shadow of the muon latency.

A window of 1.25 µs CBE (2.50 µs MPV) on jet processing processing leaves a safety margin of around 1 µs on the processing of trigger signatures which do depend on jets before there is any impact on overall latency, so the overall latency is always dictated by the muon
arrival time. Here there is an overall margin of 0.9 µs MPV in the Level-0 latency as shown in Table 5.5.

<table><caption>Table 9.5: Global Trigger latency estimate, which begins with the latency of input signals and steps through processing to the delivery of TIP to the CTP.</caption><thead><tr><th>Category</th><th>Description</th><th>CBE (µs)</th><th>∫ CBE</th><th>MPV (µs)</th><th>∫ MPV</th></tr></thead><tbody><tr><td rowspan="3">Inputs</td><td>Calorimeter</td><td>1.425</td><td></td><td>1.70</td><td></td></tr><tr><td>L0Calo</td><td>2.08</td><td></td><td>2.58</td><td></td></tr><tr><td>L0Muon</td><td>4.46</td><td></td><td>5.26</td><td></td></tr><tr><td rowspan="6">Calo path</td><td>signals present</td><td>1.425</td><td>-</td><td>1.70</td><td>-</td></tr><tr><td>MUX</td><td>1.400</td><td>2.83</td><td>1.500</td><td>3.20</td></tr><tr><td>pipelined processing</td><td>0.100</td><td>2.93</td><td>0.200</td><td>3.40</td></tr><tr><td>non-jet triggers</td><td>0.200</td><td>3.13</td><td>0.250</td><td>3.65</td></tr><tr><td>jet processing (in parallel)</td><td>1.250</td><td>4.18</td><td>2.500</td><td>5.90</td></tr><tr><td>non-muon triggers</td><td>0.200</td><td>4.38</td><td>0.250</td><td>6.15</td></tr><tr><td rowspan="4">Muon path</td><td>signals present</td><td>4.46</td><td>-</td><td>5.26</td><td>-</td></tr><tr><td>MUX</td><td>0.650</td><td>5.11</td><td>1.250</td><td>6.51</td></tr><tr><td>pipelined processing</td><td>0.100</td><td>5.21</td><td>0.200</td><td>6.71</td></tr><tr><td>muon triggers</td><td>0.200</td><td>5.41</td><td>0.250</td><td>6.96</td></tr><tr><td rowspan="3">TIP</td><td>TIP assembly</td><td>0.050</td><td>5.46</td><td>0.100</td><td>7.06</td></tr><tr><td>serial transmission</td><td>0.250</td><td>5.71</td><td>0.300</td><td>7.36</td></tr><tr><td>link to CTP</td><td>0.150</td><td>5.86</td><td>0.200</td><td>7.56</td></tr></tbody></table>

## 9.6 Firmware

As described above in this Chapter, Global Trigger system will consist of a GCM being used in three distinct instances: as a multiplexing module, as an event processing module and as a demultiplexing module. These three instances will require dedicated firmware to enact the required functionality. Figure 9.2 illustrates a conceptual view of the overall data flow, firmware blocks and firmware interfaces for the Global Trigger. Each of the three use case instances require common firmware for the basic low-level frame, command/control and monitoring. Each component will also require an individual configuration interface and management system. In addition, the ZYNQ command/control chip will require specialised firmware and software to support board control and readout.
### 9.6.1 Firmware Blocks

There are several features of the Global Trigger system firmware architecture that naturally align to firmware blocks, and are described herein below.

**Common Module Firmware:** There are several common firmware aspects required for the Global Trigger.

1. Each GCM requires a low-level frame firmware that provides a base environment for the management of FPGA interfaces: reference clocks, I/O interfaces, and general infrastructure. This firmware would be common to all FPGAs and ZYNQ, but would require a dedicated synthesis for each case.

2. The ZYNQ chip on each board will provide command, control, configuration and synchronisation signals for each processing FPGA. A firmware communication block must be developed to support these functions.

3. The ZYNQ chip will provide monitoring access for the data sent from the processing FPGAs. Thus firmware to provide an interface to the monitoring data is required to accept data from the FPGA/ZYNQ interface, process the data and transfer data to the ZYNQ's software applications.

4. The ZYNQ chip will also be required to manage and monitor the GCM board resources. Thus, firmware to manage board-level functions will be required.

5. All FELIX and monitoring interfaces for the GCM are handled on the ZYNQ chip, so a data flow interface between each processor FPGA and the ZYNQ is required. This interface provides data transmission from the FPGAs (Tx) to the ZYNQ (Rx) using common protocol, which must be adapted for each GCM use case.

6. The GCM will have significant internal and external RAM resources to allow buffering in the multiplexing, data handling and readout steps. The interface to this RAM will be common for each GCM use case, though the details of how data will be utilised is specific to each use case. Thus the firmware for the RAM interface will be developed as a common module.

**Data Aggregation and Multiplexing Firmware:** The MUX module will require firmware to receive input data from the detector and trigger sources, unpack the data, perform the time multiplexing and to transmit the multiplexed data to the GEP modules. The GEP will further require a version of the MUX/GEP interface to receive and decode the data.

**Trigger Signature Firmware:** The Global Trigger TIP will be derived from a series of trigger algorithms and hypotheses processed by the GEP. As described in Section 9.3, these algorithms will be firmware-based with performance requirements that allow a series of algorithms to run simultaneously in parallel. The trigger firmware will be divided into two classes: algorithms and hypotheses. The algorithms will process input data to refine trigger objects and signatures. The hypotheses will group these objects and signatures to be tested against pre-defined thresholds.
**Trigger Framework Firmware:** The trigger signature firmware will require a coordinated scheme to deliver input data, sequence algorithms, collect and package algorithm outputs, and collate hypothesis results. These needs will be provided via a trigger framework that also incorporates a memory handling block. The memory handler must interface with the MUX, the core trigger framework and the ZYNQ firmware. Furthermore, the trigger framework must provide trigger information to the GEP/CTP interface firmware.

**Central Trigger Interface Firmware:** The CTP Interface must receive trigger information from the GEP, time-demultiplex the data and transmit the demultiplexed results to the CTP. Firmware to support these functions will be implemented on the CTP Interface and the GEP, which requires firmware to package and transmit the TIP information from the GEP.

**FELIX Interface Firmware:** The communications with FELIX are handled by the ZYNQ chip. The FELIX interface must provide firmware to transmit formatted trigger data to FE-LIX and to receive trigger control (TTC/LTI) information to be transmitted to all Global Trigger system components. The FELIX interface will also require firmware to unpack the trigger control information. Finally, there must be a firmware block that provides readout formatting, interfaces with data I/O (FELIX Rx/Tx, monitoring, FPGA input, and the RAM).

**On-Board Software:** The ZYNQ chip will be responsible for communications to the outside general network, which will allow for monitoring and command/control processes. Thus there will be software required for IPbus network communication, management and distribution of command/control signals, and monitoring applications.

## 9.6.2 FPGA Resource Usage

The critical FPGA resource usage is on the GEP. As illustrated in Table 9.1 the resource usage for topoclustering will be well below 1% of the current Xilinx VU13P, with a similar number for the $e/\gamma$ and $\tau$ spatially pipelined algorithms. The resource usage will therefore be dominated by the jet finding and trigger algorithms and hypotheses.

An estimate for the latter can be made from the existing L1Topo usage, where the worst case usage for each of the 4 Virtex7 FPGA is between 36.5% and 68.3%. Allowing for double counting of common logic (each L1Topo FPGA performs the same sorting) and scaling for the different size of the FPGA gives an estimate of some 30% usage for this logic in the Xilinx VU13P device in the proof of concept architecture.

This choice of FPGA would therefore allow some 20% of the device for the jet finding algorithm, a realistic target. It is reasonable to expect that a new FPGA generation will be available at the time of the final board design, where historically the maximum FPGA logic availability doubles between generations, allowing significantly more resource for the jet finding.
### 9.6.3 Firmware Development

Preliminary studies of the algorithm and framework firmware have begun (e.g., Section 9.3) and will continue during the system design process. Initial development and offline testing of new algorithms will take place on commercial development boards and the Production Firmware Deployment Module described in Section 9.4.6. Alongside the firmware development will be both algorithmic and bitwise simulation.

Initial validation will be in the Surface Test Facility with final validation in Point-1. It will be possible to deploy additional GEP modules receiving duplicates of events in the live system to allow parasitic testing of new firmware alongside the production system. Evolution of the firmware will be controlled by a rigorous change control mechanism, such as that already used for the Run 2 L1Topo [9.9].

Steering of thresholds and selection of algorithms within each firmware build for the Global Trigger will be defined in the trigger menu and the change control will be defined within that context.

## References

[9.1] A. Zabi et al., The CMS Level-1 Calorimeter Trigger for the LHC Run II, Journal of Instrumentation 12 (2017) no. 01, C01065.
http://stacks.iop.org/1748-0221/12/i=01/a=C01065.

[9.2] ATLAS Collaboration, G. Aad et al., Topological cell clustering in the ATLAS calorimeters and its performance in LHC Run 1, Eur. Phys. J. C77 (2017) 490, arXiv:1603.02934 [hep-ex].

[9.3] ATLAS Collaboration, G. Aad et al., Jet energy measurement with the ATLAS detector in proton-proton collisions at $\sqrt{s} = 7$ TeV, Eur. Phys. J. C73 (2013) no. 3, 2304, arXiv:1112.6426 [hep-ex].

[9.4] ATLAS Collaboration, Technical Design Report for the Phase-II Upgrade of the ATLAS LAr Calorimeter, Tech. Rep. CERN-LHCC-2017-018. ATLAS-TDR-027, CERN, Geneva, Sep, 2017. https://cds.cern.ch/record/2285582.

[9.5] M. Cacciari, G. P. Salam, and G. Soyez, The Anti-k(t) jet clustering algorithm, JHEP 04 (2008) 063, arXiv:0802.1189 [hep-ph].

[9.6] A. J. Larkoski and J. Thaler, Aspects of jets at 100 TeV, Phys. Rev. D90 (2014) no. 3, 034010, arXiv:1406.7011 [hep-ph].

[9.7] M. Cacciari, G. P. Salam, and G. Soyez, FastJet User Manual, Eur. Phys. J. C72 (2012) 1896, arXiv:1111.6097 [hep-ph].
[9.8] ATLAS Collaboration, Technical Design Report for the Phase-II Upgrade of the ATLAS Tile Calorimeter, Tech. Rep. CERN-LHCC-2017-019. ATLAS-TDR-028, CERN, Geneva, Sep, 2017. http://cds.cern.ch/record/2285583.

[9.9] K. Nagano and H. J. Stelzer, L1Topo Menu Update Procedure, Tech. Rep. ATL-COM-DAQ-2017-100, CERN, Geneva, Aug, 2017. https://cds.cern.ch/record/2280269.
<figure><img src="image_184.png" /></figure>
# 10 Central Trigger System

The Central Trigger System consists of the Central Trigger Processor (CTP), the Muon-to-CTP-Interface (MUCTPI), and the distribution of Trigger, Timing, and Controls signals (TTC) to the sub-detector readout systems, of which the Local Trigger Interface modules (LTI) are an integral part. Figure 10.1 illustrates its various components in context. While

<figure><img src="image_185.png" /><figcaption>Figure 10.1: Context diagram showing the Level-0 CTP system interfacing to the Global Trigger and FELIX systems (via the LTI modules). Only the trigger path is depicted; control, monitoring, and readout are not shown.</figcaption></figure>

the CTP and LTI hardware is newly developed and built for the Phase-II upgrade, the baseline for the MUCTPI is to use the Phase-I MUCTPI hardware [10.1] with upgraded firmware which takes into account the new requirements for Phase-II. Two MUCTPI modules may be used, compared to one in the Phase-I system, to handle increased input bandwidth requirements from the Phase-II L0Muon Sector Logic.

The CTP needs to be capable of supporting bigger trigger menus than in Phase-I, exploiting the ability of the Global Trigger to evaluate a large set of trigger criteria to address the physics challenges of the HL-LHC. This is reflected in increased input bandwidth and more logic to evaluate complex selection criteria.

The TTC system, incorporating the LTI modules, makes use of modern high-bandwidth optical-link technologies, replacing the original optical broadcast system that was developed
in the 1990s and incorporating the detector BUSY collection function that is currently per-
formed electrically. The proposed TTC system for Phase-II builds on the experience that
ATLAS has built up in Run 1 and Run 2, including both the central TTC system and related
detector-specific electronics. The LTI together with FELIX will incorporate functionality
that could not previously be supported in a common way for the detector systems, e.g. in
connection with calibration procedures.

The main functions of the Central Trigger System are described in the following.

The CTP will have optical serial inputs to receive trigger information from the Global Trig-
ger, the Phase-I legacy system and the MUCTPI. While the Phase-I CTP can form 512 trigger
items from 512 single-bit inputs, the Phase-II CTP will allow 1024 trigger items based on
1024 usable single-bit inputs. The CTP combines trigger inputs from the Global Trigger,
the MUCTPI and the Phase-I legacy L1Calo trigger systems, in order to form trigger items.
The CTP combines trigger inputs from the Global Trigger – flags indicating which trigger
criteria have been satisfied or multiplicities of trigger objects – with multiplicities and flags
coming from the MUCTPI and Phase-I legacy L1Calo trigger systems, in order to form trigger
items. The trigger items may be simple requirements, such as an inclusive selection
of objects (e.g. muons) above a given pT threshold, or complex requirements combining
multiple objects. For each item, trigger algorithm flags from the Global Trigger, and con-
ditions on the MUCTPI and the Phase-I legacy inputs may be combined in coincidence or
anti-coincidence. The CTP forms the overall trigger decision based on the trigger items that
are active, applying additional requirements, such as limiting triggers to a subset of bunch
crossings in the LHC train, deadtime, and pre-scaling.

The Central Trigger System includes extensive monitoring of trigger rates and of deadtime.
This is used in assessing the luminosity and beam conditions, as well as for operational
monitoring and optimisation of trigger conditions.

The Central Trigger System also distributes the LHC timing signals and the trigger decision, together with associated data such as a Trigger Type word to LTI modules associated to the sub-detectors. The LTI will reformat the received TTC information, with the possibility of adding asynchronous and synchronous sub-detector data and commands via a programmable interface, and send it to the sub-detector FELIX systems or other sub-detector-specific electronics. BUSY signals, i.e. backpressure signals from the subsystem readout, are collected via the LTIs and sent back to the CTP, where they are combined and used to apply deadtime when necessary. The LTI modules provide facilities that can be used by the detector subsystems for various functions, including standalone calibration running and synchronous distribution of detector-specific signals and data. It is assumed that the TTC-PON (Passive Optical Network) technology [10.2] is used for the link between the LTI and FELIX I/O cards.

The Level-0 MUCTPI is the component of the Level-0 Central Trigger system which aggreg-
ates and merges the trigger information from the barrel muon and endcap muon systems
before passing it on to the Global Trigger and to the CTP. It provides detailed information
on the muon candidates to the Global Trigger and sends muon multiplicity information
and trigger flags directly to the CTP. It is also responsible for resolving cases where a muon
gives rise to multiple muon candidates, e.g. when it passed through detectors in more than
one sector of the muon spectrometer.

Care is being taken to use processing FPGAs in the MUCTPI and CTP with some reserve
in resources and speed, beyond the minimum requirements for the algorithms that are cur-
rently foreseen. It is justified by the central role played by these systems and cost effective
due to the very small number of MUCTPI and CTPCORE units used. This reserve will give
considerable flexibility to address new requirements that may emerge in the coming years.
An example that we have already considered is a trigger that includes delayed signatures
combined with prompt ones. This could be implemented by adding trigger items based on
special or duplicated inputs to the CTP with adjusted time-alignment parameters.

## 10.1 Level-0 CTP

The CTP is the last stage of the processing chain of the Level-0 trigger system, making the final Level-0 accept decision (L0A). It receives digital trigger inputs from the Global Trigger, the MUCTPI, the legacy Phase-I trigger system, and various forward detectors and sub-detector calibration systems in the agreed form of simple flags and multi-bit multiplicities of trigger objects. While the majority of trigger inputs are received via the Global Trigger, some muon-only inputs such as muon multiplicities and muon-only topological triggers may be produced by the MUCTPI as in Phase-I. During the commissioning phase, the legacy Phase-I trigger system can provide well understood trigger inputs that can be used while the new Global Trigger is being commissioned. This includes inputs from the Phase-I L1Calo system and the muon multiplicities from the MUCTPI (that may continue to be used once the Global Trigger system is commissioned).

The trigger inputs arriving at the CTP from different sources will not generally be aligned
in time, i.e. data from the same bunch crossing but from different sources will arrive at the
CTP at different times. The CTP therefore includes programmable elements to delay the
inputs that arrive first, to align them in time with those that arrive last.

The algorithm used by the CTP to combine the different trigger inputs allows events to be
selected on the basis of a programmable trigger menu. An event is selected if it satisfies
the criteria of one or more menu items. Each menu item consists of the logical combination
of a number of criteria, typically multiplicity requirements for muons, electrons/photons,
hadrons/taus and jets, threshold requirements on missing or total transverse energy, and
flags indicating events passing the criteria in a Global Trigger algorithm. Menu items may
be enabled or disabled for certain bunches in the LHC, for example be enabled for collid-
ing bunch pairs only, or be disabled during the gaps in the bunch train when calibration
pulsers may be fired. The CTP makes provision for individually prescaling any trigger
menu item, and is responsible for introducing deadtime as required by the detector front-end and readout systems. Deadtime is normally generated internally using algorithms that are described in detail later, and taking into account the BUSY backpressure signals from the sub-detector readout systems.

After the application of deadtime, the L0A signal is formed as the logical OR of the enabled trigger items. Along with associated trigger information and timing signals, the L0A signal is distributed to the sub-detectors, via optical point-to-point links to sub-detector LTI modules, and subsequently via TTC-PON links to the FELIX systems.

The CTP will be implemented as a single ATCA shelf [10.3] with the following custom-built blades:

*   **CTPMI**: the LHC Machine interface, to receive the beam timing signals from the LHC, most notably the 40 MHz bunch clock and the ORBIT signal. The backplane will be used to distribute the timing signals to all the blades.

*   **CTPIN**: an input board to receive non-latency-critical optical and electrical trigger signals, i.e. optical trigger inputs from the legacy Phase-I trigger processors and from the MUCTPI, and electrical trigger inputs from various forward detectors and sub-detector calibration systems.

*   **CTPCORE**: the board that implements the core functions of the CTP, the realtime trigger logic path, pre-scaling, deadtime, readout, monitoring, and the TTC interface to the sub-detectors. There will be one instance (L0CTP) to be used for the Level-0 trigger system. For the evolved Level-0/Level-1 system, a second CTPCORE board with different firmware would be used as the L1CTP.

Figure 10.2 shows the basic architecture of the CTP, with its main components, its inputs and outputs, and the main flow of information.

### 10.1.1 CTPMI: Machine Interface

In order for ATLAS to be synchronous with the LHC beam, the CTP will need to provide two signals to the ATLAS sub-detectors via the aforementioned TTC distribution system: the 40 MHz bunch clock and the ORBIT signal. The bunch clock is a 40 MHz clock derived from the LHC RF system and hence synchronous with the arrival of bunches in ATLAS. The ORBIT signal is one pulse per LHC revolution and is synchronous with the LHC bunch train structure. As for Run 1 and Run 2, the LHC will provide the following signals: BC1, BC2, ORBIT1, ORBIT2; i.e. 40 MHz bunch clock and ORBIT signals separate for beam 1 and beam 2 of the LHC. At flat top of the LHC magnet cycle, the two beams will be frequency-locked onto each other, so that BC1 and BC2 become identical.

As no LHC-wide decision has been made for a common receiver for these four signals, the requirements need to be collected in order to understand how these signals are received.
<figure><img src="image_186.png" /><figcaption>Figure 10.2: Basic architecture of the CTP, showing its main components, inputs, outputs, and the main information flow. Shown in green are additional elements only present in the evolved Level-0/Level-1 architecture.</figcaption></figure>

Sufficient flexibility on the CTPMI will be provided to facilitate later adaptations. The CT-PMI will be equipped with an FPGA (e.g. Xilinx Kintex Ultrascale+ KU15P [10.4]) to perform a variety of functions and to provide the required flexibility. Typical functions include synchronisation of the ORBIT signals, fine-tuning of the phase of the clock signals, switching between clocks, providing an internal 40 MHz clock, and jitter cleaning. Depending on whether or not they will be already implemented on a possible LHC-wide common receiver module, these functions could be provided by the CTPMI. The timing signals (bunch clock, ORBIT) are made available on the ATCA backplane for distribution to other modules (CTPCORE, CTPIN).

### 10.1.2 CTPIN: Trigger Inputs

The CTPIN board provides electrical and optical inputs for non-latency critical inputs, such as auxiliary electrical trigger signals from forward detectors and calibration systems, as well as optical trigger inputs from the legacy Phase-I trigger system and the MUCTPI. Up to 24 electrical NIM standard trigger signals can be received via LEMO connectors. Two 12-way ribbon fibre receiver modules [10.5] allow one to connect up to 24 serial optical links, each
with line rates of up to 9.6 Gb/s. The two receiver modules simplify the physical interface to the MUCTPI and the legacy Phase-I trigger system and give additional flexibility via sufficient headroom in the number of fibres and bandwidth. The inputs are connected to an FPGA (e.g. Xilinx Kintex Ultrascale+ KU15P [10.4]) where the signals are synchronised and aligned in time. The rates are monitored in detail, including per bunch, and up to 512 input bits can be selected to be routed to the output. A 12-way ribbon fibre transmitter module [10.5] allows one to send the 512 selected trigger signals to the CTPCORE module.

It is possible to use more than one CTPIN module, e.g. in case more than 24 electrical trigger signals need to be included.

### 10.1.3 CTPCORE

<figure><img src="image_187.png" /><figcaption>Figure 10.3: Block diagram of the CTPCORE architecture.</figcaption></figure>

Figure 10.3 shows the proposed implementation of the CTPCORE and its interfaces to other components of the trigger system. It will be a single electronics board based on the ATCA standard.
The implementation will use FPGAs with a large number of on-chip multi-gigabit transceivers/receivers (MGTs). An existing candidate FPGA from the Xilinx Virtex Ultrascale+ family [10.4] is the VU13P, which has 128 on-chip MGTs, a sufficient number of logic cells and sufficient memory to implement the required logic. There will be one such FPGA for the real-time trigger path, and a second one for the event readout, monitoring, and dead-time generation. The FPGAs will be chosen such that a significant fraction of their resources remain free as margin for the implementation of future functionality. High speed-grade devices will be used to minimise the contribution of the CTP to the trigger latency. An additional FPGA with an on-chip processor (System-on-Chip SoC) will be used to control the CTPCORE blade.

**Trigger inputs** The trigger inputs are implemented as 24 optical serial links, grouped into two 12-way ribbon fibre receiver modules [10.5]. These links will be operated at 9.6 Gb/s synchronously with the bunch crossing (BC) frequency with 8b/10b encoding, i.e. 4600 bits/BC, thus providing a lot of headroom in the required input bandwidth and connectivity.

The first 12 links are reserved for the Global Trigger, from which it is planned to receive 1024 trigger bits. The remaining 12 links are to be used for trigger inputs from one or more CTPIN modules, with up to 512 trigger bits from the MUCTPI, the legacy Phase-I trigger system, and from forward detectors and sub-detector calibration trigger signals.

**Trigger formation** The realtime path of the trigger formation in the CTPCORE is shown in Fig. 10.4. Out of the 1536 received trigger inputs, at any given time, the CTPCORE is able to use 1024 in the subsequent L0A trigger formation process. The 1536 incoming trigger inputs will be synchronised to the BC clock and passed through a delay pipeline of programmable length in order to align the data to the correct bunch crossing. Programmable and self-generated trigger signals such as random triggers and bunch pattern flags will be added at this stage. A switch matrix allows a selection of 1024 trigger inputs (TIP) to be mapped to the subsequent Trigger Logic block. The Trigger Logic block offers programmable logical functions to combine the 1024 inputs to form 1024 trigger items according to a defined trigger menu. The block will be organised as an array of Look-Up Tables (LUTs), in combination with one or more large ternary Content-Addressable Memories (CAM). The LUT/CAM scheme builds on the extensive experience gained during Run 1 [10.6] and Run 2, where this scheme was successfully used. It offers a lot of flexibility in the logical combinations of the trigger signals and allows one to change the trigger menu without modifying the firmware, by simply loading different configuration parameters and memory contents into the CTPCORE.

The selection of the FPGA for the CTPCORE board is driven by the number of on-chip high-speed serial links as well as by the required logic and memory resources. The required resources are dominated by the LUT/CAM structure used to implement the trigger menu, in particular the CAM resources scale with the square of the number of trigger items. Doubling the number of trigger items with respect to the Run 2 system would require an FPGA
<figure><img src="image_188.png" /><figcaption>Figure 10.4: Realtime path of the trigger formation in the CTPCORE.</figcaption></figure>

with at least four times the capacity of the Run 2 system to implement the Phase-II CTP-CORE trigger menu. The foreseen FPGA device (Xilinx Virtex Ultrascale+ VU13P [10.4]) has more than five times the number of logic look-up tables as the device used on the current Run 2 CTPCORE (Xilinx Virtex-7 7VX485T [10.4]) and should therefore have sufficient spare resources to also implement additional Phase-II functionality. Thus, we are confident that we can implement the aforementioned logic for 1024 inputs and 1024 trigger items in a single FPGA device.

The 1024 trigger items from the Trigger Logic Block are called "Trigger items before Bunch Groups" (TBG). They are each gated with an individually programmable combination of so-called bunch group masks. The bunch group masks implement fully programmable bunch patterns repeated at the LHC revolution frequency. The resulting 1024 so-called "Trigger items Before Pre-scale" (TBP) are pre-scaled with individually programmable pseudo-
random scalers to form 1024 so-called "Trigger items After Pre-scale" (TAP). The 1024 TAP
signals are gated with the VETO signal, whose formation is described below. A so-called
"trigger mask" is also applied to enable and disable individual trigger items. The resulting
1024 so-called "Trigger items After Veto" (TAV) are then combined in logic OR to form the
L0A signal. Along with the L0A signal, a 16-bit Trigger Type word is calculated, based on
trigger type masks programmed for each trigger item. The resulting Trigger Type word
is the logic OR of the trigger type masks of all trigger items active after veto and trigger
mask.

The Deadtime block uses the time history of the L0A and the logic OR of all sub-detector BUSY signals to generate the VETO signal that is used to gate the TAP signals. The VETO signal is formed as the OR of the combined sub-detector BUSY signals and the result of an array of complex deadtime algorithms which offer protection against trigger bursts for sub-detector front-end buffers. The complex deadtime algorithms are typically leaky-bucket or sliding window algorithms, with programmable parameters and fed by the L0A signal of previous bunch crossings. The FPGA allows sufficient flexibility to address specific sub-detector requirements and needs based on experience gained during commissioning and early running.

**TTC output** The L0A signal, along with the bunch clock and additional information is sent to the sub-detector Local Trigger Interface (LTI) modules via high-speed serial optical point-to-point links. Four 12-way ribbon fibre transmitter modules [10.5] are foreseen to interface, through an optical patch-panel, with up to 48 sub-detector LTI modules. These links will be operated at 9.6 Gb/s synchronously with the BC frequency, with 8b/10b encoding, which allows the transmission of 192 bits per bunch crossing. Since it is implemented in firmware, the exact data format can be adapted to the final requirements when they are understood in detail. However, the signals and information will include:

* BC

* Bunch Counter Reset

* BCID

* Turn counter value

* L0A

* Trigger Type

* L0A counter value

* Control commands, resets, etc.

**BUSY collection** The BUSY signals from the up to 48 sub-detector LTI modules are received using 48 optical serial links, grouped via an optical patch-panel into four 12-way ribbon fibre receiver modules. While only the BUSY signal needs to be transmitted by the LTIs, there is bandwidth available to send additional information, if needed. The BUSY signals will be received in the Readout and Monitoring FPGA of the CTPCORE. They can be individually masked and they are combined in logical OR to form the combined sub-
detector BUSY signal, which is subsequently used in the Deadtime generation block. The BUSY signals can be constantly monitored in order to identify the sources of deadtime.

**Event readout** For each L0A event, the CTP will send an event fragment to its FELIX system. A 12-way ribbon fibre transmitter module [10.5] is foreseen for this purpose, to be used as four or more 9.6 Gb/s FELIX GBT-FPGA links operating in "full mode". Along with standard header and trailer information, bit arrays of the trigger inputs and items at the various stages of trigger processing will be sent as main payload: the trigger input bit pattern TIP, the trigger item bit pattern before prescaling (TBP), after prescaling (TAP), and after veto (TAV), as well as the bit pattern of all trigger inputs contributing to the formation of the L0A (TIC). It will be possible to send parts of this information in a programmable window around the bunch crossing which contains the L0A. For illustration, without any data-compression scheme, one can estimate about 5 kbits of data per bunch crossing being read out, and the additional information, including header and trailers will likely not surpass 1 kbit. Hence, four links at 9.6 Gb/s line rate and 7.5 Gb/s available bandwidth in full mode, will allow, at 1 MHz L0A rate, to read out with a readout window of up to 5 BC.

**Monitoring** The CTP features monitoring counters for all trigger input signals, trigger items before and after pre-scaling and veto (TIP, TBG, TBP, TAP, TAV), L0A, incoming sub-detector BUSY signals, and generated deadtime. They allow continuous counting with intermediate readout accompanied by a 25 ns precision time interval for accurate rate normalisation. Special counter arrays ("per-bunch histograms") will allow trigger input rates, item rates, and deadtime to be monitored for each individual bunch-crossing identifier.

**Hooks for the evolved Level-0/Level-1 architecture** Two 12-way ribbon fibre transmitter modules [10.5] are foreseen to send Level-0 trigger information to the Level-1 trigger system in the evolved Level-0/Level-1 architecture. See Section 14.4.5 for more details.

**Partitioning** It will be possible to partition the CTP outputs and assign each individual output to one of several partitions. This will allow one to run several data-taking partitions concurrently, which is useful during testing and calibration periods. In this scheme, the trigger path is shared among the three partitions up to and including the pre-scaling. There will be several instances of the Veto/Mask, Trigger Decision, and Deadtime blocks, one per partition. CTP event readout fragments can be sent to FELIX for each partition.

## 10.2 Distribution of Timing, Trigger, and Control Signals (TTC)

Timing, Trigger, and Control (TTC) signals need to be distributed to the sub-detector FELIX systems and other sub-detector-specific electronics. The timing signals include the 40 MHz beam-synchronous clock derived from the LHC, which is used to clock the sub-detector and trigger electronics, as well as the ORBIT signal that is synchronous with the LHC turns. The clock needs to be distributed with high quality and low jitter. Trigger signals include the L0A signal, which has to be sent with a fixed latency with respect to the corresponding
collision, along with associated information, such as a L0A counter, a bunch-crossing iden-
tifier, information on the type of trigger decision, etc. In addition, there will be control
signals and data, sent synchronously or asynchronously with the beam, for sub-detector-
specific uses. The final contents and format of the TTC information will be defined taking
into account detailed requirements from TDAQ and the detector systems. More details can
be found in [10.7].

The distribution of the TTC signals is organised as a tree-like network, driven by a TTC master. During physics data-taking, the CTP assumes the role of the TTC master, while during sub-detector tests and calibration runs, an LTI module associated with the sub-detector can take over this role. In the opposite (“upstream”) direction, BUSY signals are sent from the sub-detector FELIX systems or sub-detector-specific back-end electronics. They are collected by the TTC master and used to throttle the generation of L0A signals.

The CTP will have 9.6 Gb/s point-to-point optical serial links to up to 48 LTI modules, which define the sub-detector partitioning and are associated with the sub-detector back-end or FELIX systems. For calibration and test runs, the LTI modules can replace the CTP and act as TTC master. The LTI distributes the TTC signals to a large number of FELIX I/O cards. The baseline implementation uses the 9.6 Gb/s TTC-PON optical link [10.2], which can be split by a ratio of 1:32 to be broadcasted to up to 32 destinations. A solution in which the LTI has up to 8 TTC-PON links is described here; hence each LTI can drive up to $8 \times 32 = 256$ FELIX I/O cards.

The aforementioned TTC-PON 9.6 Gb/s link has a downlink bandwidth of 200 bits per BC and uses time-division multiplexing for the uplink. Each FELIX I/O card can send up to 56 bits every 4 µs (in the maximum case of 32 FELIX I/O modules per TTC-PON link). This solution provides a lot of additional flexibility. For instance, instead of transmitting only a single BUSY flag from one FELIX I/O card, the available upstream bandwidth would allow one to send additional information indicating the source of the BUSY.

10.2.1 Local Trigger Interface

The Local Trigger Interface (LTI) is an integral part of the TTC distribution in USA15. It provides a point-to-point interface to the CTP and a TTC-PON interface to the sub-detector FELIX systems. For subdetector standalone running, it provides the basic CTP trigger functions.

The LTI will be implemented as a single ATCA blade, to be housed either in existing sub-detector ATCA shelves, or in a dedicated ATCA mini-chassis installed close to the sub-detectors' FELIX systems. Figure 10.5 shows the basic architecture of the LTI module. It features an FPGA (e.g. Xilinx Kintex Ultrascale+ KU15P [10.4]) which has connections to all the optical and electrical inputs and outputs and thus gives a lot of flexibility to implement the required functions. The blade is controlled by a System-on-Chip FPGA [10.4] connected to a gigabit ethernet network. Two 10 Gb/s SFP+ transceivers [10.8] allow the
<figure><img src="image_189.png" /><figcaption>Figure 10.5: Block diagram of the LTI architecture.</figcaption></figure>

optical connection to the L0CTP and possibly the L1CTP (see below), through which the TTC signals are received and the BUSY signal is sent. Up to 8 TTC-PON OLT (Optical Line Terminal) pluggable modules allow one to send TTC information to up to 256 TTC-PON destinations and receive corresponding BUSY information. The module features an additional TTC-PON ONU (Optical Network Unit) pluggable module to be used as a TTC analyser for debugging and monitoring purposes. Additional electrical inputs are foreseen, in order to connect trigger, control, and BUSY signals from external sub-detector electronics. A few electrical outputs will allow one to monitor selected signals with an oscilloscope.

The FPGA is used to implement in firmware the main functions of the LTI: the translation of the incoming TTC signals between the CTP and the TTC-PON format, a programmable interface for sending sub-detector data and commands, and the standalone CTP-like functions. In normal physics running, the LTI receives the main TTC signals optically from the CTP via the 9.6 Gb/s link.
It will be possible to add sub-detector specific information to the TTC stream and merge
it with the TTC information from the CTP. Typical use-cases include detector calibration
information and on-the-fly periodic re-configuration to mitigate SEU effects. A separate in-
terface is provided for each TTC-PON output with programmable mechanisms for sending
the data in asynchronous or synchronous ways. The information can be inserted in the TTC
stream in various ways, e.g. synchronously with the LHC turn, during bunch crossings
without L0A, during longer LHC bunch train gaps, during the time when an externally
connected signal is active, etc. The combined TTC information is then sent via the TTC-
PON outputs to up to 32 destinations per output. Each TTC-PON also receives the BUSY
information from all of its up to 32 destinations, via the TTC-PON up-link that follows a
round-robin time-division multiplexing mechanism. The BUSY signals of all TTC-PONs
are combined in logical OR and sent to the CTP. In addition to the BUSY state of the partic-
ular FELIX I/O card, the up-link could carry additional information, such as information
on the BUSY source to be used for detailed BUSY monitoring, or information in the context
of detector-specific calibration or configuration procedures.

For sub-detector standalone runs, the LTI plays the role of the TTC master. For this purpose, it includes CTP-like functions: essential trigger functionalities to be able to perform standalone sub-detector data-taking, using external electrical or internally generated signals, with realistic patterns e.g. for high-rate tests. Incoming electrical trigger signals can be synchronised and aligned in time. Additional trigger signals can be generated via a pattern generator and a pseudo-random generator. Trigger logic is provided via a programmable look-up table. The resulting trigger items can be combined with bunch group patterns and can be pre-scaled if necessary. They are gated with a VETO signal before the L0A signal is formed as the logical OR of all selected trigger items. The VETO signal is generated as the logical OR of the BUSY signals received via the TTC-PON links, an electrical BUSY input, and realistically generated deadtime using the same deadtime algorithms as in the CTP.

Monitoring counters, including some per-bunch monitoring facilities, are provided for the
trigger signals at the input, output, and intermediate stages, as well as for BUSY and dead-
time signals.

A pattern memory is also provided, which can be used to play back user-defined trigger
sequences or to capture incoming signals for debugging purposes.

The FPGA will be chosen such that it allows enough flexibility and available resources to
be able to embrace new requirements based on future user experience.

10.3 Level-0 MUCTPI

The MUCTPI processes and combines muon trigger information from the barrel and end-
cap trigger system and passes it on to the Global Trigger and the CTP for combination with
other signatures. It resolves cases where a single physical muon gives rise to more than one
muon candidate. It counts such multiple objects only once for the multiplicity calculation (e.g. by only retaining the candidate with the highest transverse momentum, $p_T$) and is able to filter duplicate candidates passed on for further processing.

After the synchronisation, time alignment, and overlap removal stage, the muon information is processed in two parallel paths: the Global Trigger and the CTP path. In the Global Trigger path, the muon candidates are sent to the Global Trigger for further processing, with the possibility to limit the number of candidates if necessary. In the CTP path, muon multiplicities are calculated for the various $p_T$ thresholds, taking into account all muon sectors; they are transmitted directly to the CTP.

The MUCTPI is currently being built for the ATLAS TDAQ Phase-I Upgrade [10.9]. It is designed to be forward compatible with the Phase-II upgrade, where is it planned that the Phase-I MUCTPI hardware will be re-used with new firmware, possibly using two hardware modules instead of one, as discussed below.

<figure><img src="image_190.png" /><figcaption>Figure 10.6: Block diagram of the MUCTPI board.</figcaption></figure>

Figure 10.6 shows a block diagram of the architecture of the Phase-I MUCTPI board. The implementation of the MUCTPI is a single electronics board following the ATCA standard. The module is based on FPGAs with a large number of on-chip high-speed serial links, as well as high-density parallel fibre optics receiver and transmitter modules. The implementation uses a pair of Muon Sector Processor FPGAs, each handling half of the inputs from the L0Muon Sector Logic modules. They perform the synchronisation, time alignment, overlap removal and further processing tasks on the incoming muon information. A smaller FPGA
interfaces to the CTP, TTC, and FELIX. The FPGA for the Muon Sector Processor is chosen
such that a significant fraction of its resources remain free as a reserve for the implement-
ation of the Phase-II functionality described here. Each of the two Muon Sector Processor
FPGAs receives and processes the output from up to 104 links from L0Muon Sector Logic
modules, via nine 12-channel parallel fibre optic receiver modules [10.5].

The results from each Muon Sector Processor FPGA are sent to the Global Trigger via up to
two 12-channel parallel fibre optic transmitter modules [10.5]. The optical links operate at
9.6 Gb/s synchronously with the LHC clock and use 8B10B encoding. Assuming 64 bits per
muon candidate, this allows each Muon Sector Processor FPGA to send up to 72 candidates
per BC to the Global Trigger, well in excess of the largest multiplicity relevant for the trigger.
The large available bandwidth gives the possibility to send the candidates for a window of
several BCs for use in evaluating delayed signals for exotic particles in the Global Trigger.
The exact information content and data format of the optical links will be defined based
on the available information from the L0Muon Sector Logic and the requirements of the
topological algorithms that will be implemented in the Global Trigger.

A third FPGA is used to merge the results from the two Muon Sector Processor FPGAs,
and to send multiplicities for each p<sub>T</sub> threshold to the CTP through a single optical link.
The merger FPGA also implements the MUCTPI readout driver functionality, including the
gathering and sending of the full muon trigger information to FELIX upon LOA reception.

The MUCTPI will interface directly to the TTC system via an LTI module. It will receive the
optical TTC fibre via a TTC-PON ONU, and the firmware will be adapted to the new TTC
distribution system foreseen for the Phase-II upgrade.

The Phase-I MUCTPI hardware aims to be forward-compatible for Phase-II. However,
based on the final design of the Phase-II muon trigger system (the design of which has
already evolved substantially since the Phase-I MUCTPI was specified) and based on ex-
perience from Run 3 operations, new requirements may emerge which exceed the input
bandwidth or muon sector processing FPGA resources of the Phase-I MUCTPI hardware
module and call for a hardware upgrade of the MUCTPI. For this reason, the baseline is to
operate the Phase-II MUCTPI with two Phase-I MUCTPI modules, one per detector hemi-
sphere (i.e. η < 0 or η>0), essentially doubling the input bandwidth and muon sector
processing resources. Low-latency LVDS links between the two Muon Sector Processor FP-
GAs in each MUCTPI module are used to share information needed in the overlap-removal
stage. There is no geometrical overlap between the muon detectors covered by the two
MUCTPI modules.

10.4 Monitoring, Control, Configuration, Offline Software

Each ATCA board of the Level-0 Central Trigger system will feature extensive environ-
mental monitoring. Parameters being monitored include voltages and currents of all crit-
ical power rails, as well as temperatures of FPGAs, fibre optic transmitter/receiver modules and power converters. A small subset of parameters critical to the health of the board will be collected by the on-board IPMC, which can take actions to protect the board. It also provides those parameters to the shelf manager of the ATCA which publishes them to DCS. The DCS path through the shelf manager will also be used to power-cycle the board and to reset the System-on-Chip (SoC). In addition, the SoC has access to the full set of environmental parameters, which will be sent to further processing using Ethernet. If required, they can also be published to DCS. The SoC will also be used for configuration, control, and monitoring of the module through a Gigabit Ethernet interface.

New software will be developed, which includes low-level software for interfacing with the ATCA blades, tools for diagnostics and tests of the hardware, online software for configuration, control, and monitoring within the ATLAS run control framework, as well as offline software for simulation, reconstruction, and event monitoring.

## 10.5 Latency

Table 10.1 summarises the estimated latency contributions of the critical path through the central trigger system. The latency of the CTP is for the latency critical path of inputs from

<table><caption>Table 10.1: Summary of the expected latency of the CTP and LTI system.</caption><thead><tr><th>Item</th><th>Partial Current Best<br>Estimate (μs)</th><th>Uncertainty/<br>Contingency (μs)</th></tr></thead><tbody><tr><td>CTP Processing</td><td>0.275</td><td>0.100</td></tr><tr><td>Link to LTI (5 BC + 30 m fibre)</td><td>0.275</td><td>0.025</td></tr><tr><td>LTI Internal processing</td><td>0.100</td><td>0.025</td></tr><tr><td>Link to FELIX (5 BC + 10 m fibre)</td><td>0.175</td><td>0.050</td></tr></tbody></table>

the Global Trigger to the CTP. The de-serialisation at the serial link input is accounted for elsewhere. The quoted latency number of 275 ns for CTP processing includes the re-synchronisation to the local BC clock, the selection and switching of trigger signals, and the complete trigger path including trigger logic, bunch groups, pre-scaling, gating, and final OR to form the L0A signal and associated Trigger Type word.

For the link to the LTI, 275 ns is estimated, including the serialisation in the CTP, a 30 m long fibre, and the de-serialisation in the LTI. The internal processing of the LTI is estimated to be 100 ns, and the link to FELIX is estimated to be 175 ns, including serialisation in the LTI, a 10 m long fibre to a close-by FELIX system, and the de-serialisation step in FELIX. The uncertainty is taken into account as contingency and summarised in Table 10.1.

Table 10.2 summarises the latency of the critical path through the MUCTPI. For the Phase-II
<table><caption>Table 10.2: Summary of the expected latency of the MUCTPI.</caption><thead><tr><th>Item</th><th>Partial Current Best Estimate (μs)</th><th>Uncertainty/ Contingency (μs)</th></tr></thead><tbody><tr><td>MUCTPI Processing</td><td>0.250</td><td>0.100</td></tr><tr><td>Link to MUX of Global Trigger (5 BC + 15 m fibre)</td><td>0.200</td><td>0.025</td></tr></tbody></table>

system, the firmware of the MUCTPI has to be completely re-written in order to process potentially twice as many candidates with a spatial resolution better by a factor of 16. The latency of MUCTPI processing is estimated to be 10 BC, i.e. 0.250 µs, taking into account the sharing of trigger information between the two Muon Sector Processor FPGAs via the low-latency LVDS links. The latency of the link to the MUX of the Global Trigger is estimated to be 8 BC, i.e. 0.200 µs, which is the sum of the serialisation in the MUCTPI, an assumed fibre length of 15 m, and the de-serialisation in the MUX of the Global Trigger.

## 10.6 Project Planning

A tentative schedule for the Phase-II Central Trigger upgrades is shown in chapter 19, for the CTP, the MUCTPI, and the TTC system.

It has been split into a number of activities, some of them common to the three systems. After capturing the requirements and designing the respective system in a conceptual way, the system needs to be specified in detail. For the CTP and TTC, this leads to the Preliminary Design Review. In parallel, hardware evaluations take place, e.g. concerning the optical links and FPGAs, using off-the-shelf evaluation tool-kits. For the CTP boards and the LTI, it is planned to start with a first prototype. After an evaluation and test phase, a Final Design Review is planned. Depending on the outcome of the review, a pre-production version may be necessary. For the LTI, the pre-production version could be produced including a small number of boards aimed to be distributed to the subdetectors for detailed assessment. After a Production Readiness Review, the final modules will be produced.

Concerning the MUCTPI, the requirements capture and conceptual design phase will last until one year after the start of Run 3 of the LHC, allowing one to benefit from experience with the Muon New Small Wheel. In a review of the requirements and specifications, a decision is expected for the final Phase-II MUCTPI architecture: whether the Phase-I single-blade MUCTPI hardware is adequate or whether a two-blade architecture using two Phase-I MUCTPI blades is needed. A redesign and construction of a new set of MUCTPI modules based on more modern link and FPGA technologies is considered as a risk mitigation option. For the baseline architecture with two Phase-I MUCTPI blades, one would have to launch the production of the additional Phase-I MUCTPI blades.
The evaluation of the prototypes will include interface tests with upstream and downstream systems, such as the Global Trigger, MUCTPI and LTI for the CTP, FELIX for the LTI, and the L0Muon Sector Logic and the Global Trigger for the MUCTPI.

For all three systems, firmware, low-level software, test software and high-level software needs to be developed. The firmware implementation starts with board infrastructure firmware and firmware to test and evaluate the hardware. It is followed by the firmware to control the board via the network and firmware for the high-speed links. Some of these firmware blocks are common for the various boards and may be shared. There is also firmware specific for the CTPIN, CTPMI, CTPCORE, MUCTPI, and LTI boards, implementing the main functions of the respective board. The CTPCORE, for example, will have several firmware blocks that implement the various functions of the CTPCORE: the realtime trigger, the event readout, the monitoring, and the TTC interface.

Low-level software needs to be developed, including software for basic communication with the board and the board infrastructure. This part will be similar for the various boards and can be shared. It is followed by a low-level library to communicate with the specific registers of each board and a low-level test suite which includes diagnostic tools as well as software to perform full system tests. For the CTPCORE and the MUCTPI there will be specific configuration software which allows the boards to be configured via a higher-level description of the trigger menu for the CTPCORE and of the overlap handling for the MUCTPI.

The software development concludes with the high-level software, which comprises control and monitoring software, which is integrated in the ATLAS TDAQ run control software framework. In addition, offline software for simulation, reconstruction, data quality, and analysis is required.

All hardware components will be first commissioned in the laboratory, based on trigger patterns generated at the hardware inputs and the comparison of readout and monitoring data with a full system simulation. Once fully commissioned in the lab, the systems will be ready for installation in USA15 during Long Shutdown 3 and final commissioning with the ATLAS subsystems can begin.

## References

[10.1] R. Spiwoks et al., The ATLAS Muon-to-Central Trigger Processor Interface (MUCTPI) Upgrade, Tech. Rep. ATL-DAQ-PROC-2017-013, CERN, Geneva, Jun, 2017. https://cds.cern.ch/record/2270141.

[10.2] D. Kolotouros, S. Baron, C. Soos, and F. Vasey, A TTC upgrade proposal using bidirectional 10G-PON FTTH technology, Journal of Instrumentation 10 (2015) no. 04, C04001. http://stacks.iop.org/1748-0221/10/i=04/a=C04001.
[10.3] *AdvancedTCA Base Specification*, Tech. Rep. PICMG-3.0-R3.0, PCI Industrial Computer Manufacturers Group, 2008.

[10.4] *UltraScale Architecture and Product Data Sheet: Overview*, Tech. Rep. DS890 v3.0, Xilinx, October, 2017.

[10.5] *MiniPOD AFBR-814VxyZ, AFBR-824VxyZ, 14 Gbps/Channel Twelve Channel, Parallel Fiber Optics Modules*, Tech. Rep. AV02-4039EN, Avago Technologies, 2013.

[10.6] *ATLAS Collaboration, Performance of the ATLAS Trigger System in 2010*, The European Physical Journal C 72 (2012) no. 1, 1–61.
http://dx.doi.org/10.1140/epjc/s10052-011-1849-1.

[10.7] F. Lanni et al., *Interfaces with Detector Front-End Systems. Requirement Document for HL-LHC*, Tech. Rep. ATL-DA-ES-0051 v.1, EDMS Id 1563801 v.1, CERN, Geneva, March, 2016. https://edms.cern.ch/document/1563801.

[10.8] *Specifications for Enhanced Small Form Factor Pluggable Module SFP+, Revision 4.1*, Tech. Rep. SFF-8431, SFF Committee, July, 2009.

[10.9] *ATLAS Collaboration, Technical Design Report for the Phase-I Upgrade of the ATLAS TDAQ System*, Tech. Rep. CERN-LHCC-2013-018. ATLAS-TDR-023, CERN, Geneva, Sep, 2013. https://cds.cern.ch/record/1602235.
<figure><img src="image_191.png" /></figure>
# 11 Data Acquisition

## 11.1 Introduction

The Data Acquisition (DAQ) acts to facilitate the final stage of the ATLAS online event selection process before permanent storage for offline reconstruction and analysis. The DAQ system transports and stores data accepted by the Level-0 trigger for further processing. The data are then provided on demand to the Event Filter (EF) system, which is then responsible for performing event selection. The DAQ system will change substantially for the Phase-II upgrade in response to evolving functional and performance requirements, driven in large part by increasing trigger rates and data volumes. The upgraded system will benefit from improvements in multiple related technologies. The work will continue the process, started during Phase-I, of replacing obsolete custom hardware and software with implementations built on commodity components.

This chapter describes the overall DAQ system architecture before detailing the design of each functional area, as well as the network which will facilitate all inter-component communication and data transport. Initial focus is given to the DAQ components themselves, moving then to the online software which underpins operations. Finally, a detailed discussion of the Event Filter component is presented, which draws on aspects of both the revised DAQ system design and new online software infrastructure. The EF system itself is described in more detail in Chapter 12.

The baseline architecture is designed to service a single-level hardware trigger running at 1 MHz, with hardware tracking as a co-processor to the EF. The system will support a maximum output rate of 10 kHz of full events for transfer to CERN permanent storage. This does not include rates for partial event building, which are not expected to drive bandwidth or processing requirements. All baseline figures will therefore only refer to the full event building rate.

Given the potential evolution to one featuring a split-level hardware trigger, a discussion of the implications of such a change for the DAQ system is presented at the end of the chapter. Such a change requires both modifications to the design and flexibility to be built into the revised baseline to facilitate future changes.
## 11.2 Overview of System Functionality

The primary functional blocks of the DAQ architecture in Phase-II are shown in Fig. 11.1. The detector Readout system in Phase-II will be made up of Front-End Link eXchange (FELIX) systems receiving event data from detector front-end (FE) links relayed across a commodity multi-gigabit network to software-based *Data Handler* applications running on commodity servers. The Data Handlers will host detector-specific processing such as fragment aggregation and formatting, while also facilitating monitoring operations. FELIX will also relay Timing, Trigger and Control (TTC) information from the Phase-II TTC system to on-detector electronics. Additionally, FELIX will provide a link between the Detector Control System (DCS), Online SW System and on-detector electronics. Through this channel commands and data will be relayed to detector FE components, with data also being distributed from the detector FE to DCS to facilitate routine operations such as control, configuration, monitoring and calibration. DCS will also receive monitoring data from TDAQ hardware throughout the system.

The Data Handlers will relay all event data to the Dataflow system for storage in a large buffer system known as the *Storage Handler*. Event building functionality will be implemented as part of the interface between the Data Handler and the Storage Handler. The Storage Handler will then serve data as needed to the EF, which will be implemented on a commodity server farm. Accepted events will be sent to the *Event Aggregator* system for packaging and transfer to the CERN computer centre for further processing. Underpinning all DAQ system activity will be a common online software framework; responsible for control, configuration and monitoring of the entire ATLAS data taking process. For development and commissioning, all DAQ system components will be designed to support standalone data taking either as a complete or partial detector slice as needed by detector operations.

## 11.3 Event Data Format

Detector data will be formatted by the *Data Handlers* after detector specific processing as needed by the *Dataflow* and *Event Filter* systems. Logical organisation of the event data will be overseen by the *Event Builder* to optimise throughput of the system and minimise bandwidth transferred to the *Event Filter*. The accepted events will be formatted as required by offline reconstruction. The use of data compression is not currently planned, but this could be reviewed in future if compression is shown to be advantageous.
<figure><img src="image_192.png" /><figcaption>Figure 11.1: The main functional blocks of the DAQ architecture for Phase-II, with focus on interfaces with other systems. Dataflow is shown in yellow, Readout in Green. The EF components are shown for reference in Red. The arrows linked to Online Software indicate the path for monitoring, control and configuration operations, denoting the integration of all components with the software infrastructure.</figcaption></figure>

# 11.4 Detector Readout

## 11.4.1 FELIX

As illustrated in Fig. 11.2, FELIX is the Readout system component which implements the interface to all detector-specific electronics via custom point-to-point serial-links. FELIX also acts as the interface to the Data Handlers, monitoring, control & configuration and DCS via a commodity multi-gigabit network. TTC information will be received by FELIX from the Local Trigger Interface (LTI) and relayed to detector FE electronics systems in a format satisfying system-specific requirements. The main idea behind the FELIX concept is the development of a modular system which makes it possible to independently upgrade or modify aspects of the system such as computing and buffering resources, network technology or supported point-to-point (or PON) serial-link protocols. The ability to evolve through further upgrades is a key feature of the Readout System when one considers the performance requirements and long development cycle leading to Phase-II, as well as the long lifetime of the ATLAS experiment beyond this period.

The ATLAS choice of optical link technology connecting to on-detector electronics is the Versatile Link [11.1] for Phase-I upgrades and the Versatile Link PLUS [11.2] for Phase-II
<figure><img src="image_193.png" /><figcaption>Figure 11.2: Possible Readout system architecture for Phase-II. The location of detector specific processing within the chain will evolve as requirements become better understood and tested.</figcaption></figure>

upgrades. The latter is specified to operate at up to 10 Gb/s for uplinks and at 2.5 Gb/s for downlinks, and therefore is lpGBT compatible. FELIX will interface to these links with standard transceivers (e.g. miniPODs). Connections within the ATLAS computing cavern between off-detector electronics and FELIX will be implemented with standard optical links. FELIX will support numerous uplink protocols. These include lpGBT, FULL mode [11.3] and GBTx mode [11.4] (as already used in Phase-I) and any other detector specific protocols which will develop over the course of the upgrade, such as the protocol used for readout of the pixel detector. Downlinks will always make use of the GBT or lpGBT protocol to convey information from the TDAQ and DCS systems to FE electronics. Examples of these data include: TTC information (e.g. the bunch crossing clock), configuration and control signals for calibration procedures and DCS traffic. FELIX must therefore also receive packets from a commodity multi-gigabit network and route them to the relevant serial-link. The connections to DCS places a requirement on FELIX reliability and uptime beyond that of regular data taking operations.

FELIX functions as a router between custom serial links and a commodity multi-gigabit network. It is detector agnostic and encapsulates common functionality. There is no requirement for FELIX to decode or process the received data beyond what is required to determine its destination. In the baseline architecture, the detector FE electronics do not require RoI information. The RoI list (along with L0CTP output information) will thus be sent to FELIX as data fragments from the trigger systems and follow the normal Readout
and Dataflow path. This information will be used by EF processors, which will decode the
list and retrieve the corresponding data.

FELIX will be implemented with a commodity server hosting custom FPGA I/O cards. Each FELIX card receives trigger and synchronous commands from one optical fibre (PON), which connects the FELIX card to one LTI module. The same fibre is used by the FELIX card to propagate back BUSY information. The FELIX card will receive complete information for each L0A, will pass on this information to the Data Handler(s), which are in charge of inserting it into the data fragments, and will send a subset of the information to the detector front-ends formatted according to the needs of the front-end electronics. FELIX will also allow front-end data sources to send data not only to the Data Handler but also to other network end points. For example, hardware trigger processors may sample 'bunch crossings of interest' that are not L0A or accumulate data for histograms of input and output data characteristics. These can be transferred to FELIX on a dedicated logical link different from L0A data and routed to a dedicated monitoring process on the network.

The LTI module will also be able to send FELIX additional user information, or specific
trigger patterns (e.g. for calibration), as agreed with the detectors. The communications
between the LTI and FELIX I/O card will be implemented in firmware at both ends. The
essential TTC and BUSY functionality that exists in the Phase-I system will be complemen-
ted by enhanced features, including additional event identification counters and facilities
useful for detector calibration, configuration and control. The detailed requirements will
be captured in discussion with the detector communities, and the implementation will be
optimised and refined in detailed design and prototype-evaluation work.

To meet the requirement of forward compatibility with Phase-II, the Phase-I upgrades of
the New Small Wheel, MUCTPI, Liquid Argon calorimeter trigger electronics and Level-1
calorimeter and muon trigger systems [11.5] will use FELIX-based readout. These upgrades
only require a subset of the functionality described, but FELIX will be designed to be ar-
chitecturally compatible across upgrade phases. For Phase-II the design of FELIX will be
expanded to address Phase-II specific requirements, such as: support for the upgraded
TTC system, more complex configuration and control scenarios, vertical and/or horizontal
aggregation of data across links before routing data fragments to the Data Handlers, expan-
ded data routing capabilities and handling of a higher number of potentially higher speed
links. Any Phase-I FELIX systems will be replaced with the upgraded Phase-II versions as
needed.

As illustrated in Fig. 11.2, the baseline implementation envisages that the FE links and
GBT/lpGBT/FULL mode protocols will be managed on a custom PCIe interface board (the
FELIX I/O card) hosting a custom mezzanine that will implement the TTC interface (i.e.
the connection to the LTI). It is expected that each FELIX interface board will be able to sup-
port up to forty-eight links, with two such boards hosted in a commodity server. However,
the number of links that will be handled by each PCIe card largely depends on the required
<table><caption>Table 11.1: Overall Phase-II Readout System Size. The precise number of FELIX systems will vary depending on the partitioning model which detector groups choose for their Readout slices. The number of links will increase due to the fact that fibres are usually handled in bundles of 12. The ratio between FELIX servers and Data Handler servers will be optimised for I/O and CPU load balancing.</caption><thead><tr><td>Component</td><td>Total Number</td></tr></thead><tbody><tr><td>Total links from detectors</td><td>17093</td></tr><tr><td>FELIX I/O card</td><td>545</td></tr><tr><td>FELIX servers</td><td>279</td></tr><tr><td>Data Handler PCs</td><td>545</td></tr></tbody></table>

link speeds, link utilisation and type of protocol as well as on the evolution of PCIe technology or any other equivalent high-speed bus that may be selected. The PCIe Gen4 (16 GT/s <fnref n="1" />) standard has already been released, offering double the bandwidth of PCIe Gen3 (in common use at the time of writing). The first devices compatible with PCIe Gen4 started appearing in the market in 2017. Based on this model of implementation, and assuming the choice of a bus at least as performant as PCIe Gen4 x16, the size of the Phase-II FELIX system is summarised in Table 11.1. A full breakdown of the expected number of links per subdetector is presented in Table 11.2. In places where the use of 10 Gb/s IpGBT with high-link utilisation (> 50%) is expected, 24 links from the front-end have been assumed for each board. In the case of lower link speeds (< 5 Gb/s or low utilisation), and less FPGA resource intensive link protocols, such as for the ITk Pixel detector, 48 links per board are envisaged.

The Phase-I FELIX has been demonstrated to be able to drive 24 GBT links and 12 FULL mode links. In the case of GBT the limiting factor is the number of available FPGA lookup tables, while in FULL mode the limitation comes from the maximum bandwidth of the PCIe Gen3 bus in use. More information can be found in the presentation prepared for the FELIX Phase-I progress review [11.6]. For Phase-II, FELIX systems will need to support 24 links, with the exception for the ITk Pixel FELIX systems which will need to support 48. This goal is achievable, since the ITk Pixel uplinks will resemble FULL mode links, which require far fewer FPGA resources to implement than GBT links. By 2019 PCIe Gen4 development systems are expected to be available. These will be used to start prototyping and identifying any limitations to be overcome.

### 11.4.2 Data Handler

The Data Handlers will receive data from FELIX via a commodity multi-gigabit network. The system will be implemented as a series of applications running on commodity serv-

<footnote n="1">¹ GigaTransfers per second, a common metric for bus performance.</footnote>
<table><caption>Table 11.2: Summary of Phase-II Detector Readout Link and Bandwidth Requirements. Downlink refers to data travelling toward the front-end electronics, and uplink to data travelling from the front-end toward the rest of the DAQ system. Detectors with existing FELIX installations from Phase-I will be updated with new hardware as required.</caption><thead><tr><th>Detector</th><th>Number of FELIX boards</th><th>Number of Links</th><th>Bandwidth (Gb/s)</th></tr></thead><tbody><tr><td>ITk Pixel downlink</td><td rowspan="2">224</td><td>1285</td><td>2.5</td></tr><tr><td>ITk Pixel uplink</td><td>10596</td><td>5</td></tr><tr><td>ITk Strips downlink</td><td rowspan="2">76</td><td>1552</td><td>2.5</td></tr><tr><td>ITk Strips uplink</td><td>1824</td><td>10</td></tr><tr><td>LAr LASP downlink</td><td rowspan="2">36</td><td>100</td><td>2.5</td></tr><tr><td>LAr LASP uplink</td><td>770</td><td>10</td></tr><tr><td>LAr LDPB downlink</td><td rowspan="2">8</td><td>31</td><td>2.5</td></tr><tr><td>LAr LDPB uplink</td><td>155</td><td>10</td></tr><tr><td>L0Calo downlink</td><td rowspan="2">8</td><td>16</td><td>2.5</td></tr><tr><td>L0Calo uplink</td><td>120</td><td>10</td></tr><tr><td>NSW downlink</td><td rowspan="3">96</td><td>864</td><td>5</td></tr><tr><td>NSW uplink</td><td>1440</td><td>5</td></tr><tr><td>NSW Trigger Processor</td><td>64</td><td>5 or 10</td></tr><tr><td>Global Trigger downlinks</td><td rowspan="2">4</td><td>48</td><td>2.5</td></tr><tr><td>Global Trigger uplinks</td><td>96</td><td>10</td></tr><tr><td>Tile</td><td>8</td><td>160</td><td>10</td></tr><tr><td>TGC</td><td>8</td><td>192</td><td>10</td></tr><tr><td>MDT</td><td>64</td><td>1536</td><td>10</td></tr><tr><td>RPC</td><td>4</td><td>64</td><td>10</td></tr><tr><td>CTP</td><td>1</td><td>12</td><td>10</td></tr><tr><td>MUCTPI</td><td>1</td><td>4</td><td>10</td></tr><tr><td>LUCID uplink</td><td>1</td><td>24</td><td>5</td></tr><tr><td>ZDC uplink</td><td>1</td><td>24</td><td>5</td></tr><tr><td>AFP uplink</td><td>1</td><td>12</td><td>5</td></tr></tbody></table>
ers which facilitate detector-specific processing, e.g. formatting and/or monitoring, within
a common DAQ infrastructure. After completion of detector-specific actions all data will
then be passed to the Dataflow system for further processing by the Event Filter (see Sec-
tion 11.5). To meet the requirements of detector-specific trigger-aware monitoring, auto-
mated recoveries and book keeping, the Data Handler will also receive Level-0 trigger in-
formation via FELIX.

<figure><img src="image_194.png" /><figcaption>Figure 11.3: *Main components of the Data Handler infrastructure.*</figcaption></figure>

The detector-specific data processing in the Data Handler, shown in Figure 11.3, as well as the configuration/calibration, control, and monitoring functions shown in Figure 11.2, will be implemented using common software tools. Raw data processing, which up to this point has been implemented in firmware on-board the Readout Driver (ROD) hardware, will now be implemented within customisable Data Handler software applications. On average, it is expected that a data handler will be capable of sustaining a bi-directional I/O capacity of order 100 Gb/s. The ratio between the number of FELIX boxes and Data Handlers for each detector is not yet well-defined. The number of Data Handlers, and the number of links each handle, can be easily adjusted given the flexibility of the network slice design. The final ratio will depend on I/O parameters such as bandwidth, i.e. how heavily subscribed each FELIX input link is and the data size for L0A fragments, and transaction rate, i.e. how many readout links per FELIX box will receive data for each L0A and whether all front-end sources report for every accept. The ratio will also depend on the total CPU load (within the Data Handlers) requested for DAQ data formatting and for detector specific needs, such as calibration and monitoring operations. For modelling this the assumption can be made that a few thousand operations per incoming fragment are possible.

11.4.3 R&D leading to the final design and implementation of the Readout
System

The rapid technology evolution in the area of high-speed interconnects makes it premature
to freeze the design of the Readout System at this point. There is nevertheless a baseline
architecture, in-line with the implementation of FELIX for Phase-I (hardware shown in Fig-
ure 11.4), relying on PCIe, multicore commodity servers and commodity high-speed net-
work technologies. The FELIX firmware designed for the Phase-I is in an advanced stage
of development [11.7]. The design provides a good estimate of FPGA resource utilisation,
from which it is possible to make extrapolations for FELIX in Phase-II. The system size
presented in this chapter is based on these extrapolations. By basing prototype implement-
ations on this architecture, it will be possible for TDAQ and detector communities to ex-
plore how the required Phase-II functionality and performance could be achieved. As an
example, the Phase-I FELIX is being used to demonstrate the capability to perform complex
calibration procedures for the ITk in the context of the ITk Pixel demonstrator programme.
Results are expected in the course of 2018 and will make it possible to tune both the fu-
ture design and implementation based on first-hand experience and collaboration between
TDAQ and detector communities. A similar programme of R&D will occur for DCS, in or-
der to inform final decisions as to the location of DCS-specific processing operations (from
the network and software layer to the level of the FELIX firmware itself).

<figure><img src="image_195.png" /><figcaption>Figure 11.4: Phase-I FELIX hardware platform, known as the FLX-711. Final prototype shown on the left, block diagram on the right.</figcaption></figure>

Similarly, by evolving the Phase-I FELIX firmware and Readout software it will be pos-
sible to study the optimisation of FPGA resources in order to deal with a higher number
of FE Links. It will also be possible to develop and evaluate more complex data routing
algorithms based on the Level-0 IDentifier (LOID)/BCID, as well as the implementation
of 'vertical or horizontal' data aggregation before forwarding the data to the Data Hand-
lers. In this context vertical aggregation refers to the grouping of multiple event fragments
from the same logical detector link before forwarding the data to a Data Handler; whereas
horizontal aggregation refers to the aggregation of data fragments from multiple logical
detector links with the same LOID/BCID before forwarding them to a Data Handler. FELIX
data-routing capabilities are an important ingredient not only for the design of the Data
Handler, but also for the DAQ system as a whole. For instance, being able to route data
based on the LOID/BCID opens the possibility of partitioning the Dataflow/EF system into
identical subdetector-wide slices. Each slice would be made of a subset of Data Handlers,
each connected to all Readout paths from a subdetector, each seeing all of the subdetector's
output for any event for a fraction of the LOA rate.
In addition to these areas of study, R&D will continue on the matter of high-speed server
buses, either evolutions of PCIe or any alternatives that may appear in the coming years.
Work will also focus on high-performance I/O software. Studies will be performed in the
areas of data compression, which may be useful in minimising network bandwidth as well
as Dataflow storage requirements. Last but not least, alternative form factors for the FELIX
and Data Handler may be considered, assuming they retain the modularity of the FELIX
concept in terms of independent upgradability of computing resources, network technolo-
gies and serial-link handling.

11.5 Dataflow

The Dataflow system buffers, transports, aggregates and compresses event data. It is re-
sponsible for the transport of data from the output of the detector Readout to CERN per-
manent storage. The main functional elements of the Dataflow system are the Event Builder,
the Storage Handler and the Event Aggregator. The Event Builder is the logical interface
with the Readout System. It receives event data from Data Handler systems and assembles
event records, associating each incoming event fragment with its parent. The Event Builder
also manages the storage volume of the Storage Handler system. The Storage Handler is a
high-throughput large-volume storage system which buffers event data before and during
processing by the Event Filter. For events accepted by the Event Filter, the Event Aggreg-
ator collects, formats, compresses, and transfers the output to CERN permanent storage.
Fig. 11.1 shows these main functional blocks as part of the Phase-II Dataflow system.

The Dataflow system receives events from the Readout System’s Data Handlers at 1 MHz and buffers them in the Storage Handler with total writing (i.e. input) traffic of 5.2 TB/s. The size of this traffic is driven by the overall event size. This is shown in comparison to Phase-I for the input stage of the system in Table 11.3. The Storage Handler provides the Event Filter with access to Level-0 trigger information, and approximately 10% of data, at 1 MHz and full event access at 400 kHz for a total reading (i.e. output) traffic of 2.6 TB/s. The Event Aggregator receives the full event record at the output rate of 10 kHz for a total throughput of 60 GB/s out to CERN permanent storage. The traffic requirements for the different components of the Dataflow system are summarised in Table 11.4.<fnref n="2" />

The next sections describe the different components of the system in more detail and provide
information independent from the implementation. The last section presents and discusses
the implementation-dependent aspects.

<footnote n="2">² Not including contributions from the *High Granularity Timing Detector* or small contributions from other subsystems.</footnote>
<table><caption>Table 11.3: Average Event Size (before HLT selection) as extrapolated from Run 2 data to the Phase-I conditions(including the new detector systems introduced by Phase-I upgrade), and from estimated by the detector systems in Run 4 after the Phase-II upgrades. Forward detectors are not listed since the associated event size is negligible.</caption><thead><tr><th rowspan="2">Detector System</th><th colspan="2">Extrapolated Data Size [MB]</th></tr><tr><th>Phase-I</th><th>Phase-II</th></tr></thead><tbody><tr><td>Pixel</td><td>0.3</td><td>2.4</td></tr><tr><td>Strip</td><td>0.2</td><td>0.5</td></tr><tr><td>TRT</td><td>0.5</td><td>-</td></tr><tr><td>LAr</td><td>0.7</td><td>0.7</td></tr><tr><td>Tile</td><td>0.1</td><td>0.2</td></tr><tr><td>Muon</td><td>0.5</td><td>0.8</td></tr><tr><td>TDAQ</td><td>0.6</td><td>0.6</td></tr></tbody><tfoot><tr><td><strong>Total</strong></td><td><strong>2.9</strong></td><td><strong>5.2</strong></td></tr></tfoot></table>

<table><caption>Table 11.4: Phase-II Dataflow traffic requirements.</caption><thead><tr><th colspan="2">Component Connection</th><th>Traffic</th></tr></thead><tbody><tr><td colspan="2">Detector Front-ends to FELIX</td><td>5.2 TB/s</td></tr><tr><td colspan="2">FELIX to Data Handlers</td><td>5.2 TB/s</td></tr><tr><td colspan="2">Data Handlers to Event Builder/Storage Handler</td><td>5.2 TB/s</td></tr><tr><td colspan="2">Storage Handler to Event Filter</td><td>2.6 TB/s</td></tr><tr><td rowspan="2">Event Filter to HTTIF</td><td>Event Filter to rHTT</td><td>175 GB/s</td></tr><tr><td>Event Filter to gHTT</td><td>560 GB/s</td></tr><tr><td colspan="2">Event Filter to Event Aggregator and Permanent Storage</td><td>60 GB/s</td></tr></tbody></table>

### 11.5.1 Event Builder

The Event Builder is the logical interface between the Dataflow and Readout systems and acts as the primary intelligence managing event data across the Dataflow system. It is also the mechanism by which backpressure from the Dataflow and downstream systems are communicated to the Readout system.

The Event Builder will be implemented as a software interface to the Storage Handler. The Data Handlers will write data to the logical storage volume via this interface. The interface will be responsible for associating event data with their parent L0A as they are received. Across the Event Builder system events must be tracked at the L0A rate of 1 MHz. In the case of an operational issue the Event Builder interface will throttle Data Handler traffic to reduce the rate of written data, by asserting back pressure to the Readout system.
### 11.5.2 Storage Handler

The Storage Handler buffers data received from the Readout system through the Event Builder to decouple the Readout and Event Filter. The large storage volume needed to achieve the required 7.8 TB/s aggregate I/O traffic allows for increased event processing time, the potential of processing deferment, and an increase in operational robustness in the EF.

The Readout system accesses the Storage Handler through the Event Builder software layer. Event Filter processors are then notified of the presence of new data within the storage volume and commence processing as permitted by their current load. Accepted events are then made available to the Event Aggregator for transfer to permanent storage.

### 11.5.3 Event Aggregator

The Event Aggregator system receives events from the Event Filter, performs compression and prepares output files for transfer to permanent storage. The Event Aggregator receives the 60 GB/s output from the Event Filter farm. In terms of implementation the system will make use of a common storage hardware platform shared with the Storage Handler.

The Event Aggregator provides a buffer area capable of storing up to 48 hours of accepted event data as required by CERN-IT. This decouples online data taking operations from non-ATLAS systems and enables the Dataflow system to cope with disruptions or malfunctions of the data transfer to the permanent storage systems outside of ATLAS experimental control.

The Event Aggregator will be responsible for ensuring that events associated with different output streams are sent to Tier-0 as part of the requested stream. The decision as to which stream an event should be associated with will be taken by the Event Filter.

The Event Aggregator will support partial event building based on the result of Event Filter processing.

Finally, the Event Aggregator will be responsible for any necessary communication with the Tier-0 processing centre if anything beyond data transfer is required.

### 11.5.4 Storage System Sizing & Technology

Looking at currently available technology, and extrapolating based on industry predictions, it is possible to produce an estimated scale for the Phase-II storage system. Current commercially available data centre solutions provide storage space within performance-tiered clusters using a variety of different media types with different throughput and storage capabilities including: HDD, SSD, Non-Volatile Memory Express (NVMe) and random-access
memory (RAM). Adopting the same policy, with the use of different media types according to role, could allow the system to meet its high-throughput requirements while using the most appropriate and cost-effective storage type for the component in question. For example, on the Data Handler side, very high-throughput media such as RAM could provide the necessary performance, while more reliable media such as HDDs could provide the stability needed for the Event Aggregator side. For the Storage Handler, the throughput requires that Solid State Drives (SSD) be the dominant technology. The system is sized according to this assumption.

Over the past ten years HDD capacity grew by a factor of 10 and the trend is expected to continue through to 2025. The same improvement in capacity can be reasonably expected for SSD technologies. Today the highest capacity SSD is of order 4 TB. Today's SSD technologies provide 2 GB/s of aggregated throughput; this throughput scales with capacity up to the extent that the connectivity can support. With PCIe Gen4 connected SSDs, we assume 5 GB/s achievable throughput as a baseline for future PCIe connected SSDs which will have a capacity of 20 TB.

The Storage Handler system needs to provide 5.2 TB/s of writing and 2.6 TB/s of reading traffic. The full 7.8 TB/s I/O throughput can be provided with 1800 SSDs providing 36 PB of total storage. This represents more than an hour of event buffering at the full Level-0 rate before and during Event Filter processing.

To meet the requirements of CERN-IT the Event Aggregator must buffer 48 hours of full event data at 10 kHz with an output event size of 6 MB, giving a storage volume requirement of 10 PB. This capacity would require 500 SSDs which are naturally included as part of the Storage System needed for the Storage Handler. The required output throughput of 60 GB/s is negligible compared to the total 7.8 TB/s of the Storage Handler, and a common storage system is designed for the input and output buffer to more efficiently use resources and provide greater system flexibility.

As a whole, the storage volume supporting the Dataflow system could consist of 1800 SSDs supporting 5.2 TB/s of input traffic, 2.6 TB/s of output traffic and 36 PB of storage capacity, among which up to 10 PB will be used by the Event Aggregator.

### 11.5.5 Factors Affecting Implementation

Due to the high-throughput requirements, the design of the Dataflow system is closely coupled with that of the network system which supports communications (see Fig. 11.8). The ideal case, where the network can be considered an infinite resource linking every system to every other system (i.e. all-to-all) regardless of functionality or physical location is not feasible from a cost or logistical perspective given the required bandwidths. The Dataflow system design and implementation must take into account the constraints emerging from the supportable network topology and bandwidth provided between clients.
From the Dataflow point of view the network design provides higher-bandwidth and lower-latency connections between subsets of Data Handlers and subsets of Storage Handler units by connecting them all via the same switch hardware within a subset. Every Data Handler has a set of storage units for which it will be much more efficient to write to. Connectivity between a specific Data Handler and the Storage Handler units not directly connected to its switch is provided by interconnecting switches through an intermediate router. The local network areas are referred to as Dataflow *slices* as shown in Fig. 11.5. Intra-slice communication will be considered efficient and fast while inter-slice communication will be considered limited.

<figure><img src="image_196.png" /><figcaption>Figure 11.5: Logical flow of the Dataflow system showing the slices of network traffic from Data Handlers to preferred Storage Handler units.</figcaption></figure>

11.5.6 System Workflow

Fig. 11.6 shows a high-level description of the communication between the Dataflow com-
ponents. It describes a comprehensive understanding of the life cycle of an event within
the Dataflow system and remains true regardless of the final implementation.

The Data Handlers write data to the storage system using the Event Builder interface. The
location of the data is irrelevant to the Data Handlers and the Event Builder guarantees that
written data is optimally placed within the Dataflow slice and will eventually be accessible
to the Event Filter. This communication step is used by the Event Builder to prevent Data
Handlers from writing new events, and therefore asserting back pressure to the Readout
<figure><img src="image_197.png" /><figcaption>Figure 11.6: Logical communications between different components of the Dataflow system. The red boxes show the communications involving inter-slice traffic, the dashed Vertical lines indicate the logical component responsible for each action. The arrows represent all communication between components, including the initial message (indicated by arrow direction) as well as any response and follow up high bandwidth traffic if present. The Event Builder serves as an interface for both the Data Handler to write data to the storage volume and the Event Filter Processing Units reading data from the storage volume.</figcaption></figure>

system. The main task of the Event Builder is to gather the data from the different Data Handlers into a single event entity. The event entity can be a physically contiguous file or a logical aggregation of independent blocks distributed across the storage cluster. By tracking the event entities the Event Builder is responsible for the maintenance of the Storage Handler volume. This event building step necessarily involves inter-slice communication. In the logical event building case, maintaining the state of the distributed event fragments accounts for far less network traffic than the event data itself needed for physical event building. For optimal use of storage media, as well as network resources, sets of built events could be aggregated into larger files for access by the Event Filter. The exact number of events in each file would then be dependent on the technologies in use.

Once an event is built it is published by the Event Builder to the Event Filter and can be requested by or pushed to a given Event Filter node. The Event Filter sees the event as part of a single file whether or not it is physically built. From the storage system and network point of view, if the file is contiguous or distributed in blocks implies very different access
patterns. In either case there is many-to-one communication at some point in the chain that requires special consideration in the design of the Dataflow and network systems.

The two different forms of event data (distributed and contiguous) may coexist in the sys-
tem. The Event Builder could, for example, initiate a background physical event building
process for long-lived events when computing, storage and network resources are avail-
able. The Event Builder may also use a timeout policy to move event data to different me-
dia types throughout their lifetime. The Event Builder guarantees that these background
activities are not visible to the Event Filter or the Event Aggregator.

If the event is accepted by the Event Filter it writes the result of the Event Filter processing to a designated location within the event file, and the Event Aggregator is informed of the file and can access it for aggregation and external transfer. The result written to the Event Aggregator can include information governing output stream allocation and partial event building. When the transfer is complete the file is deleted to release storage resources. If the event is rejected, the file is similarly deleted from the storage system.

<figure><img src="image_198.png" /><figcaption>Figure 11.7: The measured throughput of GlusterFS DFS normalised to the performance of a single local process. Measurements are shown for sequential reading and writing as well as random reading and writing as a function of IO block size. The distributed read and write processes are coordinated across multiple hosts allowing the performance to potentially be greater than the local process when throughput is limited by host CPU, for example sequential write of small block sizes.</figcaption></figure>
**A DFS-based Implementation** The rapid evolution of both hardware storage technologies and software storage solutions makes the definite choice of an implementation premature, but from what is known so far, a proposed solution has been developed based on linking the different Dataflow system components through a distributed file-system (DFS). In a DFS-based implementation, all system components (i.e. Data Handlers, Storage Handler, Event Filter, Event Aggregator) host part of the file-system on their local storage media. The location of all data on the system is tracked by metadata managed by the DFS. All Dataflow operations can then be enacted by modifying file metadata to schedule transfer to another storage volume, and the synchronisation of metadata between hosts. In this model, the event data files for processing exist in a global namespace, which is visible to all Dataflow system components.

The first step in the processing of event data is for the Data Handlers to open a file, located in the Storage Handler volume, and start writing into it. Local metadata servers then implement event building functionality by ensuring that the locations of event data are correctly tracked across the system. In this way the Event Builder component of the system is implicitly implemented without the need for more dedicated technology. The existence of the file in the global namespace will make it visible and accessible to the Event Filter and the Event Aggregator. Processing or accepting an event can then be achieved by moving the file to a specific directory monitored by the Event Filter or Event Aggregator, therefore making it a metadata-only operation. Event rejection can itself be implemented as a simple file deletion which is also a metadata-only operation. In the proposed solution the Storage Handler system provides its own storage units and devices but may also take opportunistic advantage of other systems' storage media to aggregate storage space where possible and pertinent.

From a networking perspective, DFSes like CephFS [11.8], GlusterFS [11.9], or OctopusFS [11.10] provide a network topology-aware placement of file blocks. With this it is possible to ensure each Data Handler will only write event fragments to parts of the file within its Dataflow slice. There is therefore no need for large scale data transfer between the storage system slices, only the transfer of the metadata needed to keep track of the contents of the file in each slice. The DFS metadata contains the storage block-to-file level information as well as the usual file system metadata (permissions, timestamps, etc.). At present, no DFS supports the opening of the same file from several hundred writers, but CephFS and OctopusFS do offer distributed metadata management that is required for sliced event building.

Storage tier-aware DFSes like OctopusFS or ScaleIO [11.11] take care of automatically managing the different capabilities of mixed media storage to balance throughput and reliability. Most DFSes (HDFS [11.12][11.13], GPFS [11.14], GlusterFS, OctopusFS, etc.) have built-in mechanisms to manage data replication to provide device, unit and rack fault tolerance. These mechanisms can be disabled to save storage space and storage and network bandwidth, and potentially overloaded to implement an automated Event Filter processing timeout mechanism.
In Fig. 11.7 the performance impact of a DFS is compared with the throughput that can be achieved by a single local process as a function of application-to-storage communication block size. Shown for GlusterFS, the writing performance is at or above the local throughput for I/O block sizes up to the ATLAS event size. The distributed read and write processes are coordinated across multiple hosts allowing the performance to potentially be greater than the local process when throughput is limited by host CPU, for example sequential write of small block sizes. Random read access shows a degradation of less than 10%. The CPU overhead associated with reading data from a DFS relative to the local storage media has been measured in CephFS, ScaleIO and GlusterFS to be negligible at less than 1-5% of the system CPU. Any performance impact associated with a DFS implementation is assumed to be within the uncertainty of the functional requirements of the Dataflow system.

The implementation that has been described in this section is based on the known functionality of third-party DFSes used as points of reference, with third-party solutions in general seeming the optimal choice for any final implementation. Such solutions may require additional customisation, the level of which depends on the evolution of the market. The final form of the system will favour commodity solutions where possible, with the desired features having been highlighted. In the event that the development of a custom solution provides a more efficient and cost-efficient approach it will be adopted. Market survey, experimentation and specific studies will be necessary to maintain an up-to-date knowledge of the available third-party solutions' capabilities and limitations. DFSes that minimise inter-slice communication to metadata-only traffic, trading off extra latency to achieve the required throughput, will be considered the top candidates. Automatic mechanisms internal to DFSes (e.g. load balancing, data replication, tiered-cache management) will provide a smarter and more upgradeable system supported by a potentially large developer community.

## 11.6 Network

The Readout and Dataflow components discussed so far rely on a high-throughput network for all communication and data transport. In this section the architecture of such a network will be discussed, focusing not just on data transport, but also on reliability and redundancy.

### 11.6.1 Network Domain Description

The ATLAS DAQ system consists of two well-separated network domains. The first domain, known as the Readout network (or FELIX network), is a high-throughput system connecting the FELIX servers and the Data Handlers with additional requirements for DCS, control and monitoring infrastructure interconnection. The high-level architecture for this
component is envisaged to be an extension of the Phase-I Readout network [11.15] comprising all the ATLAS subdetectors.

The second network domain, known as the Dataflow network, provides connectivity between all the Dataflow components and the EF farm. Its architecture has to be completely revised with respect to the Phase-I system as requirements will change significantly.

## 11.6.2 Network Technology Choice

There are currently several network technologies with promising future performance projections in terms of bandwidth and maximum latency that can be considered for the Phase-II networks. These are Ethernet, InfiniBand and OmniPath. While Ethernet is the market-leading technology for a wide range of applications (including data centre interconnection), InfiniBand and OmniPath are technologies developed specifically for High Performance Computing applications. It is not currently possible to make firm conclusions about the most suitable technology for Phase-II, as many factors are currently unknown and difficult to predict. For example: price per port and per Gigabit, lossless operation of Ethernet, effectiveness of remote direct memory access (RDMA) for high-speed Ethernet, medium-range link range, etc.

The current baseline is a system completely based on Ethernet. However, the use of two different network technologies in the two network domains presented is not excluded. While to first order this may appear inefficient, there might be some requirements only attainable with a specific technology (e.g. ultra-low latency communication in the Readout network) for each use case.

In terms of technology assumptions which can be made at this point, the first one is that by the time of Phase-II it will be possible to purchase 100/400 GbE devices, 400 Gb/s'Next Data Rate' (NDR) Infiniband or its equivalent in OmniPath. This assumption is supported by the latest official roadmaps [11.16][11.17]. The second assumption is that Ethernet NICs, InfiniBand host channel adapters (HCAs) and OmniPath host fabric adapters (HFAs) will have enough bandwidth towards the system CPU and memory to run at the required speed [11.18].

In order to select the best suitable technology the R&D strategy has been divided into three steps: technology tracking, technology evaluation and network slice construction. The technology tracking will be performed with Mellanox (for InfiniBand), with Intel (for OmniPath) and with the typical Ethernet vendors that regularly visit LHC experiments. Following the Phase-I R&D model, early stages of technology evaluation should be performed in High Performance Computing (HPC) clusters owned by third parties. This was proven to be enough to learn the basics of a given technology. Finally, when the candidate technology is selected a testbed setup should be built to perform an in-depth evaluation with a real DAQ system slice.
### 11.6.3 Network Architecture

As mentioned above, the architecture for the Readout network is similar to the Phase-I FELIX network. The design is based on organising the system in slices and connecting all the nodes in a given slice to the same high-throughput device. Each slice contains a certain number of FELIX servers and the set of Data Handlers they need to communicate with. All the network slices are connected together with a pair of routers on a higher hierarchical level allowing all-to-all communication for DCS, control and monitoring purposes.

The redundancy of the described setup could be improved by connecting the FELIX server links to a pair of switches working in active-active mode. This solves the problem of handling individual link failures (the most common type of failure) making use of the full capacity installed in normal operation. On the other hand, the service would be degraded, but not stopped, in case of a switch failure.

For the Dataflow network, the proposed architecture is based on aggregating Data Handler servers with pizza-box switches and connecting the Storage Handler units to the same network switch. A set of Data Handler nodes will then always write to the same set of Storage Handler units, maximising the throughput and reducing the latency. The switches need high-throughput uplinks to the network core routers, so as to allow all-to-all communication between the Storage Handler and the rest of the system.

For Event Filtering connectivity, the servers are stored in racks and connected to Top-of-Rack (ToR) switches. Every ToR switch is connected to the core network with enough uplink capacity to ensure the required throughput. Finally, events aggregated by the Event Aggregator need to be sent to permanent storage. For this, high-throughput Long-Range links are needed between the core routers at Point 1 and the CERN data centre.

A baseline network implementation is depicted in Figure 11.8. The shape of the core router cluster is an implementation detail and different solutions can be envisaged: from a pair of chassis routers to a distributed leaf-spine switch topology [11.19]. The advantage of using a router cluster is that it can provide a basic level of network redundancy simply by connecting pairs of uplinks to different devices.

The described architecture is suitable for Ethernet networks. However, if an HPC technology were to be used (where all the devices have the same link speed), it could be possible that link aggregation could be achieved by installing many links of the same speed. In this case, the number of ports available on the network devices for connecting endpoint hosts would be reduced and even more devices would be needed. In this situation, the architecture of the network would certainly need to be revised and tailored to a given technology.
<figure><img src="image_199.png" /><figcaption>Figure 11.8: Baseline network implementation using Ethernet technology. Network redundancy scenarios are not depicted.</figcaption></figure>

11.6.4 Control Network

The amount of control traffic during a data taking session is not expected to grow sig-
nificantly in the new system. Therefore, the need for a dedicated network is not con-
sidered necessary for control and monitoring traffic. Instead, new virtual networks can
be defined on top of the available high-throughput devices and make use of traffic en-
gineering and Quality-of-Service techniques to guarantee the non-interference of different
flows. Moreover, the control traffic can make use of all the installed network capacity dur-
ing software infrastructure initialisation and configuration steps. On the other hand, a flat
non-redundant 1 Gbps Ethernet network may still be needed for server management traffic
via Intelligent Platform Management Interface (IPMI).
### 11.6.5 Network Installation

In the case where the final network design is Ethernet based, the available Medium-Range interconnection technology will have an impact in the network installation. There are currently two industry standards used in big data centres (not yet standardised by IEEE) for 100 GbE: PSM4 and CWDM4. PSM4 stands for "Parallel Single Mode 4 lane" and can reach 500 m. CWDDM4 stands for "Coarse Wavelength Division Multiplex" and can reach 2 km. For 400 GbE similar solutions are expected to cover similar distances for reasonable prices. However, depending on the price ratio between Medium-Range 100 and Medium-Range 400 GbE the installation of an aggregation layer could be performed in the ATLAS service cavern (USA15) or in the surface data centre (installation of many 100 GbE links versus installation of fewer but more expensive 400 GbE links).

For practical purposes, patch-panel based solutions are preferred over active-fibre based ones for USA15 to surface links [11.20]. The reason is that with active cables we need to account for enough spares to deal with all the transceiver failures that might arise during operational periods. Laying new active cables is only possible during scheduled maintenance periods such as the yearly winter shutdown. On the other hand, with a patch panel solution we only need to account for spare fibres, which are passive components with low probability of failure, and account for few spare transceivers that can easily be replaced when necessary.

## 11.7 Online Software

So far in this chapter the focus has mainly been on architecture and connectivity of individual components of the DAQ system. Another core part of the design is the software infrastructure which will manage and operate the components of the system, and integrate them with the wider ATLAS data taking environment. The Phase-II upgrades provide an opportunity for significant re-designs and improvements of various parts of the system to make use of new technologies and standards. In this next section a detailed description of the proposed upgraded online software infrastructure will be presented, alongside an overview of the studies currently in progress in order to better understand and explore all aspects the design. Most of the software elements within the online setup will be maintained and upgraded adiabatically in the next several years by the ATLAS DAQ Operation group. However, some of the core software infrastructure may be significantly redesigned to benefit from the availability of new technologies

### 11.7.1 System Overview

The online software infrastructure used to facilitate ATLAS data taking is comprised of a small set of common libraries on which the DAQ and offline software projects depend. The
software stack is built on top of a large number of external libraries, which are provided in a dedicated release by the LHC Computing Grid (LCG) project. Most of the online software is written in C++, although some of the DAQ software uses Java, and Python is the standard for more complex scripting tasks. With the recent move to git for version control and CMake [11.21] as a build system, most of the software infrastructure is now using standard open-source tools for these tasks, as well as for continuous integration and testing.

The DAQ software contains components responsible for configuration, control and monitoring of the full ATLAS system during operations. Few fundamental changes are expected in terms of the responsibilities of this part of the system. The main exception is the increased use of ATCA over VME for Phase-II. The current software provides support for Single-Board Computers inside VME crates, which control much of the front-end hardware and Readout. The transition to ATCA for Phase-I seems to go hand-in-hand with solutions where a processor is integrated directly into an FPGA hosted on custom hardware. The processors are often ARM processors instead of Intel, and might run a different Linux distribution than the rest of ATLAS. Effort will be needed to provide coherent support for configuration and control of these systems across ATLAS.

The data acquisition part of the online software is expected to need to be completely rewritten compared to Run 2 and Run 3. This begins with the Data Handler component, which is the interface to FELIX, and includes the Event Builder, Storage Handler and Event Aggregator.

Another theme of the software upgrade will be the move to containerised solutions. There is an ongoing effort to reduce the dependencies between online and offline software as part of the Phase-I upgrade. This will make it possible to run Event Filter processes in their own environment inside a container as described in the next section. The same approach can be used for monitoring based on the offline Athena framework [11.22]. Monitoring in general will be a major component of the upgrade effort. As discussed below a number of changes will be required to the current monitoring infrastructure to support the proposed new system architecture.

Furthermore, control and monitoring software currently runs on dedicated machines with some limited support for failures of critical services. Moving as many of these services as possible to containers and running them on a common infrastructure will open new possibilities for fault tolerance, reliability and scalability.

Underpinning the operation of all upgraded systems will be the role of system administration, stretching from process monitoring to infrastructure management and facilitation. Such matters will be covered throughout this section, culminating in a more dedicated discussion once all the areas of interest have been introduced.
11.7.2 EF Farm Management

The EF computing farm hosts both the processing units (PUs) and all the supporting ser-
vices needed to implement the last step of the event selection component of the TDAQ
system. The farm is made up of several thousand commodity servers, each executing one
or more processes. A robust and reliable mechanism for the management of all processes
running in the EF farm is a requirement to guarantee stable and efficient execution of the EF
service. In modern software architectures, the management of large clusters of computing
nodes is delegated to so-called ‘Cluster Orchestrator’ services. In a system like the TDAQ
system, a Cluster Orchestrator will fulfill a series of well-defined basic requirements:

* It will support different types of application lifecycle (i.e., always-running, run-to-completion and cron-like services);

* It will allow both elastic and static allocation of processes to computing nodes;

* It will be able to dynamically handle cluster resources (i.e., enabling/disabling computation units at runtime, efficient exploitation of the available CPU power and memory);

* It will scale to thousands of hosts;

* It will make it possible to control (i.e., starting, stopping) and monitor the status of all active processes;

* It will make it possible to completely describe the requirements for all processes needing to be started, including the definition of command-line parameters and environment variables to be passed to the executable.

A survey of the offers currently available on the open-source market (based on the require-
ments above) highlighted Kubernetes [11.23] to be a perfect candidate as an orchestrator
for the EF computing farm.

Kubernetes was announced by Google to the open-source community in 2014 [11.24] and is
based on 15 years of experience at Google in managing and orchestrating large clusters.
Since its first release, the Kubernetes open-source community has experienced steady -
growth, reaching more than 1500 commits per month and more than 150 contributors per
month in February 2017 [11.25]. Today, Kubernetes is a mature product contributed to by
several technology partners like RedHat, CoreOS and Intel.

Kubernetes can be described as 'a system for automating deployment, scaling and man-
agement of containerised applications'. Among several supported features, Kubernetes
provides a set of services facilitating easy and effective management of applications in a
cluster:

* Scheduling of applications based on required resources and other constraints;

* Automatic re-scheduling of applications when the application itself fails or the node where the application is running dies;

* Built-in support for service discovery and load-balancing;
* Management of several storage back-ends, allowing transparent mounting of both local and network storage volumes;

* Easy (via command line tools or UIs) and automated (based on CPU usage) application scaling.

Kubernetes requires applications to be packed into software containers. Containers exploit virtualisation at the level of the operating system and are lightweight and simpler to build than Virtual Machines, which instead exploit hardware virtualisation. Packing an application into a container makes it possible to create immutable images disentangling the application itself from the host operating system. In such a way, containers do not only provide strong resource isolation but also make the development, integration and deployment cycle easier, thus simplifying software portability and distribution.

Kubernetes supports Docker [11.26] containers. Docker is currently the market-leading container platform.

### EF Processing Units in Software Containers
As a proof of concept, a small Kubernetes cluster (4 nodes) was set up using the CERN IT Virtual Infrastructure [11.27], with the goal of running EFPU instances in software containers. The PUs themselves were emulated with the offline version of today's HLT software (i.e., AthenaHLT), using a realistic trigger menu. A Docker container image was created starting from a base SLC6 [11.28] image and adding a few additional packages. The EF software was retrieved directly from the CVMFS [11.29] installation repository. Kubernetes was able to transparently mount the CVMFS volume, making it possible to keep the size of the Docker image to a few hundred megabytes. In order to better simulate data processing activity, two additional mount points were added to the container: an *input* directory with data files containing real events, and an *output* directory receiving the results of event selection algorithms. The AthenaHLT image was distributed to and executed by the Kubernetes cluster. Events were correctly retrieved from the input data files and processed by the PUs, with selection results stored in the output directory.

### Performance and Scaling
In the most recent release (1.6) available at the time of writing, Kubernetes is able to handle computing clusters with up to 5000 nodes populated with up to 150,000 containers [11.30]. To satisfy this requirement Kubernetes satisfies two performance goals for the reported cluster size: 99% of the calls to its backend (e.g., the calls to inspect the state of a container) return in less than 1 s and containers (with pre-pulled images) are able to start within 5 s with a probability of 99%<fnref n="3" />. It is worth noting that the

<footnote n="3">³ Performance of a big cluster is sensitive to the size of the Kubernetes 'master' node (i.e., the cluster control plane). The Kubernetes documentation indicates that the reported performance figures were achieved running tests on the Google Compute Engine using a n1-standard-32 [11.31] virtual machine for the master node.</footnote>
performance goals were achieved with the cluster being fully populated; giving also an estimation of the time needed to restart a container in case of failures. Larger clusters are also supported, but with degraded performance.

The time needed to completely fill the cluster represents another crucial performance and scaling figure, particularly important for system operations. Since Kubernetes does not provide any official result in that respect, some specific and dedicated experiments were performed on a cluster made up of about 1000 virtual cores. The cluster was organised in the following way: one Kubernetes master node (32 CPU cores and 60 GB of RAM) and 240 slave nodes (4 CPU cores and 8 GB of RAM each). All the nodes were equipped with CPUs of the Intel Broadwell family running at 2.2 GHz. The latest Kubernetes version (1.5) available on the CERN virtual infrastructure was used. The tests aimed at measuring the time needed to scale an application to a certain number of replicas (from one to five instances per host). In order to minimise the impact of the started applications on the measurement (i.e., they may consume CPU cycles competing with the Kubernetes system), a pause container was used and its image was pre-pulled into the cluster. Such a container sleeps for an undefined period of time after being started, with very minimal resource usage.

Fig. 11.9 shows the time needed to scale the *pause* container up to five replicas per node as a function of the cluster size, for a total of 1200 started containers in a 240 host cluster. The number of replicas was chosen to match the number of applications executed on each HLT host during Run 2. The size of the cluster could be easily changed by enabling or disabling the corresponding hosts in the Kubernetes scheduler. The measurements are reported for different values of the Kubernetes *Query Per Second (QPS)* configuration parameter set. The *QPS* set is used to set a limit on the maximum number of requests the different Kubernetes components can handle. In such a way it is possible to avoid overloading the system, resulting in a possible denial of service. The default *QPS* values are quite conservative and defined to safely allow Kubernetes to run on a wide range of hardware platforms. Tests were executed increasing *QPS* values to up to four times their defaults. Kubernetes performance proved to be strongly dependent on the *QPS* configuration. As an example, the time to fully scale the *pause* container to 1200 replicas in a 240 hosts cluster decreased from about 74 seconds down to 27 seconds with the most aggressive *QPS* configuration.

<table><caption>Table 11.5: Cumulative time to start the pause containers at the 25th, 50th, 75th, 95th and 100th percentile (240 nodes and 5 replicas per node).</caption><thead><tr><th rowspan="2"></th><th colspan="5">Percentile</th></tr><tr><th>25<sup>th</sup></th><th>50<sup>th</sup></th><th>75<sup>th</sup></th><th>95<sup>th</sup></th><th>100<sup>th</sup></th></tr></thead><tbody><tr><td>QPS x1</td><td>27 s</td><td>42 s</td><td>56 s</td><td>69 s</td><td>74 s</td></tr><tr><td>QPS x2</td><td>16 s</td><td>24 s</td><td>31 s</td><td>37 s</td><td>45 s</td></tr><tr><td>QPS x3</td><td>12 s</td><td>17 s</td><td>23 s</td><td>26 s</td><td>35 s</td></tr><tr><td>QPS x4</td><td>10 s</td><td>14 s</td><td>18 s</td><td>21 s</td><td>27 s</td></tr></tbody></table>
<figure><img src="image_200.png" /><figcaption>Figure 11.9: Time to scale the pause container up to five replicas per node as a function of the cluster size. The plot reports the measured times for four different values of the QPS parameter set.</figcaption></figure>

The impact of QPS settings can be also seen in Fig. 11.10, showing the number of started containers as a function of time for different QPS values. The QPS does not only strongly impact the rate at which Kubernetes manages to deploy applications, but it also has an impact on the time needed to start the first container (from 11 seconds down to 5 seconds for QPS values four times higher than the default configuration). At the same time it is evident how the time needed to have all the containers up and running is dominated by a few outlier instances beyond the 95th percentile (see Table 11.5).

In general, the executed tests demonstrated that:

* The capability for Kubernetes to deploy containers in a cluster scales linearly with the size of the cluster itself. Assuming no higher order effects with larger clusters (Kubernetes officially supports 5000 hosts clusters), an EFPU service instance can be fully deployed on each node of a 2500 host cluster in about 30 seconds;

* Kubernetes performance is highly dominated by its QPS configuration. QPS values four times higher than their defaults make it possible to reach a sustained deployment rate of almost 70 containers per second (to be compared to about 20 containers per second with the out-of-the-box configuration). The Kubernetes development road map aims to reach a rate of 100 containers per second on a 5000 host cluster in upcoming releases.
<figure><img src="image_201.png" /><figcaption>Figure 11.10: *Number of started pause containers as a function of time for a cluster of 240 hosts and five replicas per host. Time is counted from the moment the command to deploy the containers is sent to Kubernetes. Measured times are reported for four different values of the QPS parameter set.*</figcaption></figure>

Overall Kubernetes performance has proven to be sufficient for its usage as an orchestrator of the EF computing farm. Even so, it will be worthwhile to keep monitoring upcoming Kubernetes releases in order to track and verify evolving performance figures.

### 11.7.3 Operational Monitoring of the Online System

The status and health of every host in the online computing system (from the current HLT farm to the control nodes) must be constantly monitored to ensure the correct and reliable operation of the whole online system. The monitoring system is not critical for data taking, but it is the first line of defence: it has to promptly provide alerts in case of failure and warn of impending issues whenever possible. The reason for this is to minimise the risk of downtime, which is particularly important for mission critical systems (e.g. DCS systems) which are also a single point of failure.

The currently implemented monitoring system is based on Icinga 2 [11.32], for active checks and alerting, and Ganglia [11.33] for data storage and to provide performance data for debugging and to complement Icinga 2 information. At the host level, the system is complemented by IPMI [11.34] and Simple Network Management Protocol (SNMP) [11.35] that are used to retrieve additional data directly from the host, for hardware health and system information respectively.
This implementation has knowledge of how the physical hosts are performing, and is able to monitor the status of various services. The monitoring system relies on ping requests, custom-built service checks, etc. to determine the state of the machine. These checks can generally ensure that the needed services and applications are running, but not necessarily how well they are running.

The monitoring system has proven its reliability and effectiveness in production, providing alerting and advanced performance monitoring. Recent improvements include the automation and improvement of checks and notifications, and the generation of Icinga 2 configuration files in order to simplify system maintenance and guarantee complete coverage of the online system.

The next evolution in operational monitoring would require a change in perspective. The implementation of business logic rules would allow the monitoring system to move away from checking specific aspects of a node (for example, network speed and disk status) to a more holistic, global overview of all the services that are being provided.

This would imply focusing on the services that the servers are collectively responsible for providing and to react to a service failure (rather than a machine failure). Of course, the information provided by hardware checks should not be neglected as it would still be necessary to maintain the overall physical health of the nodes.

With this perspective, the prioritisation and schedule of the alerts and notifications would have to be reviewed and adjusted to focus on what has to be immediately addressed and what can be addressed during the next scheduled repair opportunity (such as a technical stop). By applying these aspects of business logic, it would be possible to more effectively triage and react to the various issues that arise during day-to-day operations while still maintaining acceptable levels of service quality.

Looking ahead, a variety of new monitoring solutions are now available, with further developments expected in the coming years. With the Phase-II upgrade, there is an opportunity to re-evaluate the current implementation and investigate the possibility of either improving the current design or upgrading the monitoring system with new tools.

Following a survey of the open-source market, Prometheus was identified as a possible improvement to the current implementation. Prometheus is a system and service monitoring platform capable of collecting a number of metrics from any number of configured targets. It is able to evaluate rule expressions and can trigger alerts if some arbitrary condition is met. A key factor to consider is the scalability of the implementation – an aspect that Prometheus has been shown to handle very well. The largest publicly-documented Prometheus deployment monitors ~20 million nodes, each with ~13 exporters across 13 geographic regions using ~80 Prometheus servers [11.36]. However, caution should be taken when considering these new technologies, as many of them are intended to be used in cloud-based infrastructure, and is therefore not necessarily applicable to TDAQ servers.
Prometheus can also be integrated with the TDAQ software infrastructure (through the use of exporters and client libraries) in order to provide more in-depth health monitoring (known as 'white box' monitoring). Combining these checks with physical host and service monitoring (for example, the number of database queries per second) will allow a more complete overview to be built. By adopting this approach, it would be possible to monitor the actual performance of the various applications and components in order to detect issues sooner.

### 11.7.4 Physics Monitoring

**Motivation** The Online Physics Monitoring facility is one of the key tools for maximising the efficiency of the experiment. The system can be used to verify the quality of data being taken, as well as the trigger efficiency at all levels using several complimentary approaches. The primary mechanisms used are based on event sampling and the generation of monitoring histograms. In Phase-II the majority of the components of the Run 2 and Run 3 DAQ system, as well as the current High Level Trigger, will be replaced with new designs. As such, the monitoring infrastructure must adapt to service the new designs. In this section the requirements for the new system will be discussed.

**Event Sampling** The quality of the data being taken can be affected at any time by a malfunction of detector equipment or Readout electronics. Such an issue will equally affect all physics events, so this can be monitored based on statistical samples of the full event stream. The sampling of partial or full events can be done at different levels of the Dataflow chain allowing optimisation of the resources used for different types of monitoring.

A sampling-based approach could be used in Phase-II to assess, for example, the selection performance of the Level-0 trigger. Such an assessment cannot be done using a full event stream due to the high trigger rates. Conversely, Event Filter selection performance can be monitored with the analysis of a full event stream due to the distributed nature of the EF farm. EF monitoring is based on the analysis of dedicated trigger efficiency histograms, which are built by every individual EF processing unit and periodically gathered to obtain full EF statistics.

The changes in the Dataflow system architecture which are planned for the Phase-II upgrade will affect the implementation of Physics Monitoring, in particular the Event Sampling sub-system, as most of the DAQ components which supported event sampling for Run 2 and Run 3 will be removed or replaced for Run 4. This implies that a new sampling implementation will have to be provided for the new Dataflow system components. At the same time, requirements for event sampling will be reviewed in order to improve performance and usability of the Monitoring system based on experience obtained during Run 2 and Run 3. In particular, the new Event Sampling facility will be required to:

* Sample events from a specific physics stream;
* Sample events according to an arbitrary set of properties from event headers;

* Support unbiased sampling of events before the EF decision;

* Support sampling of full events as well as fragments of events which correspond to a particular slice of the detector or Readout electronics;

* Facilitate sharing of sampled events to independent monitoring tasks;

The new Event Sampling facility will need to interact with the new Dataflow system components, namely Data Handlers and Storage Handler, in order to fulfill the sampling requirements mentioned above. The design and the implementation of these components will impact the spectrum of technologies considered for the Event Sampling facility. Depending on the specifics of Dataflow component implementation it may be possible to use third party software solutions for event sampling, or it may be necessary to proceed with building and maintenance of a custom in-house software service for that purpose.

## Event Filter Histogram Management

The implementation of the Online Histograming service, which provides histogram publication and gathering, will need to be adapted to the new Event Filter component. The monitoring of histograms produced by the EF will be vital to track the operational state of event selection code running in the new architecture. The transition to the new system will begin with a requirements review and a redesign of the existing facility. In particular, the new Histograming service will be required to support the following operations:

* Publication of an arbitrary set of histograms by every EF processing unit

* Merging of identical histograms produced by EF processing units

* Display of a pre-selected subset of produced histograms

* Application of pre-defined Data Quality assessment algorithms to a pre-selected subset of the produced histograms

* Archiving of a pre-selected subset of produced histograms, both periodically as well as at the end of a data taking session

While some of these requirements are already addressed by the original implementation of the online monitoring, those related to communication with the new Event Filter component would need to be thoroughly analysed and a new implementation of an appropriate interface will need to be provided.

## Physics Monitoring High Level Design

In order to fulfill the requirements the Physics Monitoring (PM) infrastructure will need to communicate with the other sub-systems and applications as shown in Figure 11.11. The input information for the Physics Monitoring will be provided by the Dataflow sub-system, which will supply samples of raw physics events to be analysed by dedicated Monitoring Tasks, which will fill histograms using these data. The PM infrastructure will be responsible for shipping raw events from the Dataflow
sub-system to these monitoring applications as well as for storing the histograms they pro-
duce. The PM infrastructure will also expose the stored histograms to the other applica-
tions, which can use them for data quality assessment, archive them for off-line analysis or
display to the experts and the shift crew.

<figure><img src="image_202.png" /><figcaption>Figure 11.11: Physics Monitoring interactions with external sub-systems and applications</figcaption></figure>

In order to simplify development and maintenance of the PM infrastructure it will be
broken down into 4 independent sub-systems which deal with different types of monit-
oring data:

* Event Sampling sub-system: responsible for transportation of physics events from the Dataflow sub-system to event data analysis applications

* Information Sharing: works as a medium for sharing histograms produced by data analysis applications with the other sub-systems and applications

* Information Gathering sub-system: sums up information objects, e.g. histograms, produced by homogeneous applications, e.g. monitoring tasks or EF processing units, in order to get complete statistics for a given system component, e.g the EF farm

* Data Quality Assessment sub-system analyses the monitoring histograms in order to spot any possible malfunction of the detector or the TDAQ system itself

Figure 11.12 shows how these sub-systems will interact with one another in order to provide
the required functionality.
<figure><img src="image_203.png" /><figcaption>Figure 11.12: *Physics Monitoring sub-systems*</figcaption></figure>

**Event Sampling sub-system** The Event Sampling sub-system provides a way of subscribing to a specific type of raw physics event while it passes through the Dataflow sub-system. Figure 11.13 shows how the Event Sampling sub-system interacts with the other components to accomplish its task. When the Event Sampling system receives a subscribe request it stores selection criteria which are passed as parameters of the request. It will set up a hook function to an appropriate place in the Dataflow system. The Dataflow sub-system calls this function each time a new event passes through so the Event Sampling sub-system can check if the given event matches the provided criteria. If it does match a copy of the event is sent to the subscriber while in the opposite case the event is ignored by the Sampling sub-system. Ideally Event Sampling should also take into account the processing speed of the subscriber in order to send only as many events as the subscriber can afford. If the subscriber does not want to receive any more events it can send an unsubscribe request to cancel the subscription.

**Information Sharing and Gathering sub-system** All histograms produced online by an Event Analysis application or by an EF Processing Unit are registered to the Information Sharing (IS) sub-system, which will implement a subscribe/callback pattern to share them with other interesting applications. When another application or sub-system, e.g. Information Gathering (IG), wants to get some histograms it sends a subscribe request to the Information Sharing system, providing it with a list of interesting histograms. From that moment IS takes responsibility for sending all requested histograms to the reader whenever they are updated by the providers. These interactions are shown in Figure 11.14.

The Data Quality Assessment (DQA) subsystem will also subscribe to merged histograms, which provide complete statistics for a given system component, and will receive them
<figure><img src="image_204.png" /><figcaption>Figure 11.13: Event Sampling sub-system interactions</figcaption></figure>

<figure><img src="image_205.png" /><figcaption>Figure 11.14: Information Sharing and Gathering sub-systems interactions</figcaption></figure>

from the IS system whenever they are updated by the IG sub-system. DQA will apply some
pre-defined automatic analysis algorithms to those histograms in order to spot anomalies
which may point to problems with the current data taking session. If such an anomaly is
detected DQA will raise an alarm to notify the shift crew about the issue, which will allow
potential problems to be fixed early on.
## 11.7.5 System Administration

As with any large computing installation, system administration is key to successful operation of the ATLAS TDAQ infrastructure. System administration is an area that evolved substantially in the past few years, with the advent of large commercial data centres, virtualisation, and cloud computing. While the modernisation of individual host management and monitoring tools are areas of continuous development, the Phase-II upgrade could be an opportunity to revise the boundaries between system administration, DAQ, and DCS.

The borders between DAQ, system administration and DCS that used to be well defined have been eroded by recent developments such as: technology evolution that has made networking pervasive in all aspects of computing; the introduction of embedded computers with fully fledged operating systems in parts of the experiment (which traditionally used lower level electronics devices) and the invention of containers to constrain and define the scope of applications running on servers. Such developments provide an opportunity to rethink the interplay and integration between these three domains.

Although containerisation brings many performance benefits, it also poses potential security risks which would need to be addressed. Because the operating system kernel is shared among containers on the same host, any vulnerabilities in the host would also be present within the containers. To address this, some level of vulnerability scanning and quarantine enforcement will be necessary to ensure the continued security of the container infrastructure [11.37].

Traditionally, system development and operations were two distinct fields. However, recent trends mean that what is known as 'DevOps' is now the paradigm of choice for the integration of complex systems. This requires the involvement of operations and development engineers working together during the entire system lifecycle; from the design, through the development and deployment process, to subsequent production support, and finally to deprecation. With these trends in mind, the approach taken during the last major shutdown (LS1) was to establish a more flexible infrastructure and replace tools which had become outdated and cumbersome over time. The development and continuous improvement of the current toolset is ongoing and will continue to be maintained during future upgrades.

As mentioned also in Section 11.7.2, in modern software architectures cluster orchestration services are used to manage large clusters of computing nodes. In this respect, container orchestration exists across the boundary between the application layer (Trigger/DAQ software) and the system/OS layer. Going forward, the usage of containerised systems may require a review of currently available data storage topologies in order to provide a distributed storage implementation, and improve the scalability of the current storage infrastructure. It would also be necessary to integrate the container infrastructure with the configuration management system that will be in use at the time of deployment. By leveraging some of these more recent technologies, it will be possible to improve computational efficiency
as well as the maintainability of the computing infrastructure, thereby offering a number of
significant advantages over the current implementation.

11.8 Standalone DAQ Mode & Commissioning

While the integration of the full DAQ system for ATLAS data taking is the ultimate goal
of all development activity, it is important to maintain support for use cases outside of
primary data taking. Critical among these are the use of the DAQ architecture for subdetec-
tor development and commissioning (away from Point 1) as well as testing and calibration
once the final system is in service.

To facilitate such operations to date, the system has been designed in a modular way, with
flexibility in terms of hardware platforms. Such a design makes it possible to set up small
scale data taking slices featuring subdetector components under testing alongside primary
DAQ system components, meaning the environment at Point 1 can be accurately emulated.
For subdetectors at Point 1 it is also possible to take control of specific slices of the ATLAS
detector, alongside related DAQ system components, in order to run testing and calibration
to prepare and maintain the system ahead of continued data taking.

In order to maintain support for such activities in the future, any upgrade of the DAQ sys-
tem and its components should ensure that it is possible to run logical slices of the system
both at Point 1 and at smaller scales elsewhere. Such a design will also make it easier to test
and commission DAQ system components ahead of their final integration into the ATLAS
system for Run 4. The DAQ component commissioning process itself will then take place at
multiple sites, each with slices of the system according to their development requirements.
The modularity of the system components should then make it possible to test combina-
tions of components (e.g. FELIX and Data Handler, or Data Handler and Event Builder) in
isolation before integrating into larger slices. A fully integrated 'whole system' slice will
also be created at CERN as part of the final step before installation in Point 1.

References

[11.1] *Versatile Link*, (online). https://espace.cern.ch/project-versatile-link/public/default.aspx.

[11.2] *Versatile Link PLUS*, (online). https://espace.cern.ch/project-Versatile-Link-Plus/SitePages/Home.aspx.

[11.3] *Specifications for the FELIX FULL mode link*, (online).
https://atlas-project-felix.web.cern.ch/atlas-project-felix/user/docs/FullMode.pdf.
[11.4] P. Moreira, R. Ballabriga, S. Baron, S. Bonacini, O. Cobanoglu, F. Faccio, T. Fedorov, R. Francisco, P. Gui, P. Hartin, K. Kloukinas, X. Llopart, A. Marchioro, C. Paillard, N. Pinilla, K. Wyllie, and B. Yu, *The GBT Project*, https://cds.cern.ch/record/1235836.

[11.5] P. Farthouat et al., *Report of the preliminary design review of the FELIX for Phase 1*, Tech. Rep. ATU-TC-ER-0048 v.1, EDMS Id 1518157 v.1, CERN, Geneva, Aug, 2015. https://edms.cern.ch/document/1518157.

[11.6] *FELIX Firmware Status Report for Phase-I Design Review*, (online). https://indico.cern.ch/event/676493/contributions/2786317/attachments/1557222/2449510/PR_13-11-2017_firmware.pdf.

[11.7] ATLAS FELIX Group Collaboration, *FELIX Phase-I Technical Specification and Implementation*, Nov, 2017. https://svnweb.cern.ch/cern/wsvn/FELIX/trunk/Documents/Phase1_FDR/FelixPhase1FDR.pdf.

[11.8] *CEPH Filesystem*, (online). http://docs.ceph.com/docs/master/cephfs/.

[11.9] *GlusterFS Documentation*, (online). http://docs.gluster.org/en/latest/.

[11.10] E. Kakoulli and H. Herodotou, *OctopusFS: A Distributed File System with Tiered Storage Management*, in *Proceedings of the 2017 ACM International Conference on Management of Data*, SIGMOD '17, pp. 65–78. ACM, New York, NY, USA, 2017. http://doi.acm.org/10.1145/3035918.3064023.

[11.11] *Software Defined Block Storage – ScaleIO*, (online). https://www.dellemc.com/en-us/storage/scaleio/index.htm.

[11.12] *Apache Hadoop*, (online). http://hadoop.apache.org/.

[11.13] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, *The Hadoop Distributed File System*, in *2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST)*, pp. 1–10. May, 2010.

[11.14] F. Schmuck and R. Haskin, *GPFS: A Shared-disk File System for Large Computing Clusters*, in *Proceedings of the 1st USENIX Conference on File and Storage Technologies*, FAST'02, pp. 16–16. USENIX Association, Berkeley, CA, USA, 2002. http://dl.acm.org/citation.cfm?id=1973333.1973349.

[11.15] ATLAS Collaboration, *Technical Design Report for the Phase-I Upgrade of the ATLAS TDAQ System*, Tech. Rep. CERN-LHCC-2013-018. ATLAS-TDR-023, CERN, Geneva, Sep, 2013. https://cds.cern.ch/record/1602235.

[11.16] *The 2016 Ethernet Roadmap*, (online). http://ethernetalliance.org/roadmap/.

[11.17] *Infiniband Roadmap*, (online). http://www.infinibandta.org/content/pages.php?pg=technology_overview.
[11.18] PCI Express Specifications and ECRs for Review, (online).
https://pcisig.com/specifications/review-zone.

[11.19] Brocade Data Center Fabric Architectures, http://www.brocade.com/en兑 backend-content/pdf-page.html?/content/dam/common/documents/content-types/solution-design-guide/brocade-dc-fabric-architectures-sdg.pdf.

[11.20] Active Optical Cables VS Optical Transceivers, (online).
http://www.mellanox.com/blog/2016/12/aocs-active-optical-cables-why-are-2-transceivers-and-fibers\-priced-less-than-one-connectorized-transceiver/.

[11.21] CMake Project Page, (online). https://cmake.org.

[11.22] The Athena Framework, (online).
https://atlassoftwaredocs.web.cern.ch/athena/athena-intro/.

[11.23] Kubernetes: Production-Grade Container Orchestration, (online).
https://kubernetes.io/.

[11.24] An update on container support on Google Cloud Platform, (online).
https://cloudplatform.googleblog.com/2014/06/an-update-on-container-support-on-google-cloud-platform.html.

[11.25] The Kubernetes Open Source Project on Open Hub, (online).
https://www.openhub.net/p/kubernetes.

[11.26] Docker: the world's leading software container platform, (online).
https://www.docker.com/.

[11.27] CERN Cloud Infrastructure, (online). https://openstack.cern.ch/.

[11.28] SLC6: Scientific Linux CERN 6, (online).
http://linux.web.cern.ch/linux/scientific6/.

[11.29] CernVM File System, (online).
https://cernvm.cern.ch/portal/filesystem.

[11.30] Scalability updates in Kubernetes 1.6: 5000 node and 150000 pod clusters, (online).
http://blog.kubernetes.io/2017/03/scalability-updates-in-kubernetes-1.6.html.

[11.31] Google Cloud Platform: Machine Types, (online).
https://cloud.google.com/compute/docs/machine-types.

[11.32] Icinga 2, (online). https://www.icinga.com/products/icinga-2/.

[11.33] Ganglia Monitoring System, (online). http://ganglia.sourceforge.net/.
[11.34] IPMI: Intelligent Platform Management Interface Specification, Second Generation V2.0, Revision 1.1, (online). https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ipmi-second-gen-interface-spec-v2-rev1-1.pdf.

[11.35] *A Simple Network Management Protocol (SNMP): RFC 1098*, (online).
https://tools.ietf.org/html/rfc1098.

[11.36] *Deploying Prometheus at DigitalOcean*, (online). https://promcon.io/2016-berlin/talks/deploying-prometheus-at-digitalocean/.

[11.37] N. Bila, P. Dettori, A. Kanso, Y. Watanabe, and A. Youssef, *Leveraging the Serverless Architecture for Securing Linux Containers*, in 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW), pp. 401–404. June, 2017.
<figure><img src="image_206.png" /></figure>
# 12 Event Filter

The Event Filter (EF) is responsible for executing sophisticated event selection algorithms at high rate, in a massively parallelised form, within the hardware and software infrastructure discussed in Chapter 11. The selections taking place in the EF reflect the physics goals of the collaboration, and are the means by which it is possible to deliver a rich experimental dataset within the agreed rate and bandwidth budget. Much of the rejection power of the EF will be based on hardware-based tracking, discussed in more detail in Chapter 13. As such, this chapter will focus on the software component of the EF, as well as presenting how all information sources are combined to reach the final EF decision. Section 11.7.2 describes the farm management aspects of the EF.

The challenge for the EF in Phase-II is to support the increased event input rate, while mitigating the rise in algorithm execution times resulting from the increased level of pile-up at higher luminosity. The physics motivation and expected physics performance of the EF trigger algorithms are discussed in Chapters 2 and 6, respectively. In particular, Table 6.4 is used as the basis for the estimation of the required CPU compute power.

Upgrades are required both to the farm hardware, to provide the needed compute power, and to the selection software to provide the required selection power and algorithm speed. In the present (Run 1-Run 3) trigger architecture, the track-based selections that provide much of the rejection power can only be performed in the EF. For Run 3 we expect a fully commissioned Fast TrakKer (FTK) system, which will augment the current software tracking with hardware-based full-event tracking. This will allow the EF-based event selection to move even closer to the offline one. In Phase-II, hardware-based track selections will also be possible inside RoIs. In order to maintain rejection, suppress pileup and keep efficiency high, the EF will need to make use of more CPU-intensive offline-type selection techniques, including a greater use of global (full-event) tracking.

The most affordable computing platform (commodity server PCs) is developing towards systems hosting multi-core CPUs with an increasing core-count and heterogeneous hardware architectures. The exact architecture is however difficult to predict. As an example of the quickly evolving market, in November 2017 Intel made the following two announcements: the 8th generation Intel Core processors will come with a “custom-to-Intel third-party discrete graphics chip from AMD’s Radeon Technologies Group” [12.1] all in the same processor package; and the next generation Intel Xeon Phi is cancelled in favour of a “new platform” [12.2] but without giving more details on this new platform. Considering other recent events, it is likely that we will see CPUs with GPGPUs and/or FPGAs all integrated
into the same package allowing high-speed memory transfers between them. Therefore, the upgraded EF software should allow for both parallel algorithm execution as well as exploitation of internal parallelism of those which are the most costly in terms of CPU. The optimal degree of parallelism (processing multiple events at the same time, parallel regional reconstruction within the same event, or parallelism within individual algorithms) will be found by balancing the potential benefits in throughput and the effort needed to modify and maintain the code.

The EF in Phase-II will be required to interface to the following subsystems (see Fig. 12.1):

* The hardware track trigger system, to request tracks from regions via rHTT (tracks with $p_T$ > 2 GeV) or the full ITk via gHTT ($p_T$ > 1 GeV) both covering the region $|η| < 4$.

* The Dataflow system, which will distribute event processing tasks, provide input data to and collect the output from the EF for each event.

* Online software will provide monitoring services and control of EF applications.

<figure><img src="image_207.png" /><figcaption>Figure 12.1: Diagram showing the interaction between the EF processing unit (EFPU), Dataflow system and the hardware tracking (HTT). After receiving the event data from the Dataflow system, the EFPU decides, based on the Level-0 decision, if regional hardware tracking is needed for this event. If yes, requests are sent to the relevant rHTT units along with the necessary ITk data. After receiving the HTT result the EFPU decides if the event should be rejected or continues with event processing including a possible second request for full event hardware tracking via the gHTT based on the Level-0 decision or additional EF-based selections.</figcaption></figure>
The EF processing is expected to occur promptly, i.e. with a minimum delay after the L0 accept decision. The large buffer in the Storage Handler will allow for processing timeout settings which are less strict than those used in the Run 2 and Run 3 systems. It is also conceivable that processing could continue beyond the end of a data-taking run. However, delayed processing that makes full use of the LHC inter-fill periods is currently not seen as desirable from an operational point of view. The option of an EF calibration loop (similar to the prompt calibration loop deployed at Tier-0) that would derive certain calibrations before the EF processing starts has been studied. While there is no clear use-case for calorimeter or muons, a full ITk alignment could be derived within one hour of data-taking. However, assuming the new ITk detector is at least as stable as our current detector it is not clear that this would result in significant improvements for the EF selection. In any case, should a use-case emerge, the Storage Handler could serve as a buffer of the L0 accepted data until calibrations have been derived.

## 12.1 Event Filter Hardware

Current estimates of the evolution of event processing times indicate the need for $4.5_{-0.7}^{+2.7}$ million HEP-SPEC06¹ (MHS06) to handle a Level-0 rate of 1 MHz with an initial reduction to 400 kHz achieved entirely by using information from Level-0 and rHTT as shown in Table 6.4. The CPU estimate is based on an extrapolation of current (Run 2) CPU usage, taking into account scaling with pile-up, CPU time reduction due to the use of hardware tracking, and other software improvements. More details on the estimation, the uncertainties and the pile-up dependence are given in Section 12.4.

All the commodity computing power must be accommodated within a fixed rack-space located in the surface computing infrastructure. This rack-space is shared with other components of the data-acquisition system, such as storage and networking. An extrapolation based on the evolution of compute-power in the ATLAS TDAQ farm over the past ten years (see Fig. 12.2a) results in an estimated computing capacity of 1.5 kHS06 per dual-socket server or approximately 3000 motherboards on the time-scale of Phase-II. Assuming the current format of four motherboards in a 2U server chassis, these can be accommodated in 38 racks of the type that will be available in level one of the SDX building. The scale of the system is summarised in Table 12.1.

<table><caption>Table 12.1: Summary of EF farm size estimates, based on projections of compute capacity requirements and compute power of servers from current data, as described in the text.</caption><tr><td>Compute capacity required</td><td>4.5 MHS06</td></tr><tr><td>Equivalent dual-socket servers</td><td>3000</td></tr><tr><td>Racks</td><td>38</td></tr></table>

<footnote n="1">¹ http://w3.hepix.org/benchmarking.html</footnote>
<figure><img src="image_208.png" /><figcaption>Figure 12.2: (a) Evolution of the compute power per dual socket motherboard installed in the ATLAS TDAQ farm over the last ten years. (b) Observed and expected price/performance evolution provided by CERN IT with the reasonable (1.4 CHF/HS06) and pessimistic (3.3 CHF/HS06) scenarios provided by CERN IT (see Fig. 12.2b). Based on this, we expect a cost of 3000 CHF per motherboard.</figcaption></figure>

Predicting how the cost of commodity computing technology will evolve in the next decade is very difficult. The cost on the other side is expected to be driven mainly by financial and market developments rather than technology. Furthermore there is a possibility that a disruptive technology has a large unforeseen impact. We assume a cost of 2 CHF/HS06 by 2026, consistent with the *reasonable* (1.4 CHF/HS06) and *pessimistic* (3.3 CHF/HS06) scenarios provided by CERN IT (see Fig. 12.2b). Based on this, we expect a cost of 3000 CHF per motherboard.

It is envisaged that the EF farm could be a rather heterogeneous system, possibly containing different classes of hardware such as GPGPUs, FPGAs and commodity servers. Advances in these areas will be closely monitored (see Section 12.3), but the current baseline assumption is that CPUs will provide the required compute density on the time-scale of Phase-II. However, since the technology decisions shaping the EF infrastructure will be taken as late as possible, it is important to establish interfaces that will allow for operation of a heterogeneous system.

## 12.2 Selection software

The selection software upgrades fall into two parts: further evolution of the framework and development of selection algorithms to meet the Phase-II requirements.

The present software framework is undergoing a major upgrade to provide the new functionality needed for the start of Run 3. The new framework, AthenaMT [12.4][12.5], is being implemented as a common trigger and offline computing framework. Key features are
built-in: multi-threading support, provision for seamless integration of offline algorithms and infrastructure to support external accelerators. The AthenaMT framework will be able to run existing EF and offline algorithms with limited changes, but additional changes will be needed to fully exploit the potential of the new framework. Most of the changes in the selection software are expected to be done in time for Run 3. Further changes for Run 4 will be needed to adapt the system to the evolved DAQ architecture, i.e. with regards the Storage Handler and hardware tracking interfaces. In addition, new trigger hardware (e.g. the Global Trigger) will have to be included into the trigger configuration database. In case GP-GPU or FPGA accelerators come to be used, the necessary framework services to offload the computing load to these devices will have to be developed unless the manufacturers provide compilers that automatically make use of these accelerators.

Significant effort will be required to upgrade the selection software to provide the required rejection in the EF within CPU resource constraints. Experience from Run 1 and Run 2 has shown that the rates of certain triggers, especially multi-jet triggers and triggers based on $E_T^{\text{miss}}$, rise rapidly with increasing pile-up. It is expected that an initial reduction from the Level-0 rate of 1 MHz to 400 kHz can be achieved by the use of hardware-reconstructed tracks and other information from the Global Trigger. After this initial 'fast' rejection, the remaining required EF rejection power will be achieved by importing techniques currently used offline to provide selections that are robust against the effects of pile-up. The latter often rely on tracking information to correct the energy of a calorimeter cluster. More details on the tracking use cases can be found in Section 6.12.

The tracking trigger is subdivided into *fast tracking* and *precision tracking* stages. The fast tracking consists of trigger-specific pattern recognition, whereas the precision stage relies heavily on offline tracking algorithms. The tracking algorithms are typically configured to run within an RoI identified by L0. To reduce CPU usage even further, the offline track-finding is seeded by tracks and space-points identified by the fast tracking stage. The biggest change is expected to the inner detector tracking software, as it will have to be adapted to the new ITk detector and the hardware-reconstructed tracks provided by rHTT and gHTT. The former has already been done for the offline tracking software and is being used routinely for upgrade studies of the ITk detector. Since the trigger tracking software is largely based on the offline tracking software, any additional trigger-specific changes with regards to ITk are expected to be of limited scope. A lot of experience in the use of hardware-reconstructed tracks is expected to be gained once the FTK system is fully operational by the end of Run 2.

ATLAS will rely on simulation of the upgraded trigger for development of EF software and performance studies prior to the upgrade and processing of fully-simulated data for analysis of high-luminosity data. The same EF selection software can be run on simulated data in the ATLAS production system, as is the case already with the current HLT selection software. This is straightforward thanks to the common framework and large overlap of services, algorithms and event data classes with reconstruction software. Simulation of the
detectors and hardware triggers will provide the input data to the EF. The ATLAS simulation software already includes hardware-based tracking for FTK. It will be updated to include full simulation of the HTT system, which is currently developed standalone. A fast simulation of the HTT will also be included, which will reduce processing time significantly.

The current muon trigger reconstruction [12.6] is composed of a fast muon and inner detector track reconstruction, followed by a combination step and final precision reconstruction in both the inner detector and muon systems. Both the trigger-specific and offline reconstruction software will have to be adapted to include and leverage the information provided by the new MDT Level-0 trigger as well as hardware-reconstructed tracks, both of which should result in a significant CPU usage reduction. It is expected that at least a factor two improvement in the muon reconstruction time is necessary on the timescale of Run 4.

The first stage in the calorimeter reconstruction involves unpacking the data from the calorimeters. The unpacking can be done in two different ways: either by unpacking only the data from within the RoIs identified at L0 or by unpacking the data from the full calorimeter. The RoI-based approach is used for well-separated objects (e.g. electron, photon, muon, tau), whereas the full calorimeter reconstruction is used for jets and global event quantities (e.g. $E_T^{miss}$). In both cases the raw unpacked data is then converted into a collection of cells. Two different clustering algorithms are used to reconstruct the clusters of energy deposited in the calorimeter, the sliding-window and the topo-clustering algorithms [12.7]. While the latter is significantly slower it provides performance closer to the offline reconstruction. The calorimeter software will remain largely unchanged, but with the aim to further harmonise the trigger and offline reconstruction. The focus will have to remain on improving the performance of the software, i.e. since some of the fast rejection algorithms will move to the calorimeter hardware trigger. Possible new developments include the use of hardware-based topo-clusters from the Global Trigger.

## 12.3 Study of GPGPU usage in the EF

A demonstrator has been developed to evaluate the potential benefit of offloading specific trigger reconstruction tasks to GPGPUs. Selected trigger reconstruction algorithms from the current (C++) software framework (Athena) have been re-written as Compute Unified Device Architecture (CUDA) modules to be executed on a NVIDIA GPGPU. Within the CPU code, the selected algorithms are replaced by code that converts the required input data to a format suitable for the GPGPU and makes a request, including the data payload, to a CPU-based server application, known as the Accelerator Process Extension (APE) server. The server forwards the work to a GPGPU. On completion, the server returns the output to the CPU client that then performs a conversion of the output data back to the Athena data-format. The APE server handles requests from multiple clients and can be configured to
send work to one or more **GPGPU** (or other resource). The increase in throughput (events processed per second) that can be achieved when **GPGPUs** are added to a **CPU**-only system depends on the fraction (by execution time) of the **CPU** workload that is exported to the **GPGPU**. For example, throughput could be doubled if it were possible to export 50% of the **CPU** work-load to **GPGPU**. The cost-effectiveness of this solution depends on the relative costs of the **CPU** and **GPGPU** hardware and the relative algorithm execution times on the two platforms, which determines the relative number of **CPU** to **GPGPU** needed to perform the task.

<figure><img src="image_209.png" /><figcaption>Figure 12.3: (a) Throughput increase as a function of the number of Athena worker threads when executing only ID reconstruction algorithms. (b) Event processing rate for GPGPU-based pattern matching as a function of the number of pile-up events.</figcaption></figure>

Processing modules for GPGPU have been implemented for CPU-intensive parts of the inner detector tracking (data preparation and track-seeding) [12.8], calorimeter topological clustering [12.9], cluster-splitting, jet reconstruction and a Hough transform based algorithm for muon tracking [12.10]. Throughput measurements have been made for a system comprising two 14-core Intel Xeon E5-2695 CPU (2.3 GHz clock-speed) and a system with the same CPU plus a GPGPU serving as accelerator. For the GPGPU two scenarios were tested:

i) up to four Nvidia GK210GL Kepler architecture GPGPU (in two K80 PCI cards) on the same PCIe back-plane as the CPU,

ii) a GTX1080 PCIe card with a (newer) Pascal architecture GPGPU (GP104) in a separate unit with a local CPU acting as a server and connected to the client node via a 10 GbE network.

For the inner detector track-seeding algorithm a speed-up factor of 28 (5.8) is obtained on the Pascal (Kepler) GPGPU relative to execution on CPU. Overheads for data conversion