<figure><img src="image_210.png" /><figcaption>Figure 12.4: Breakdown of the time per event (measured on the Kepler system) for (a) inner detector track-seeding and (b) calorimeter clustering offloaded to a GPGPU for the kernels running on the GPGPU and the overhead associated with offloading the work (other). The overhead comprises the time to convert data-structures between CPU and GPGPU data-formats, the data transfer time between CPU and GPGPU and the Interprocess communication (IPC) time that accounts for the transfer of data between the Athena processes and the process handling communication with the GPGPU.</figcaption></figure>

and inter-process communication reduce this to an effective speed-up of 15 (5) for the Pascal (Kepler) GPGPU cards respectively. Fig. 12.3a shows the throughput increase factor (defined as the ratio of the event rate with GPGPU to the CPU-only event rate) as a function of the number of Athena clients for full-event ID track reconstruction accelerated by a Pascal-architecture and a Kepler-architecture GPGPU. The input was a simulated Phase-I ttf dataset converted to raw detector output format (bytestream). An average of 46 minimum bias events per simulated collision were superimposed. In the CPU-only case, the track-seeding algorithm takes 28% of the total event processing time, which limits the maximum possible throughput increase to a factor of about 1.4. The Kepler GPGPU shows evidence of saturation for 20 or more clients, while no saturation is seen for the Pascal GPGPU with 60 clients due to the shorter algorithm execution time on this GPGPU. The speed-up factor obtained for the calorimeter algorithm is 3.6 on the Kepler GPGPU, but in this case the shorter algorithm execution time on the GPGPU is completely offset by the data-conversion and inter-process communication overheads, meaning no throughput increase was obtained. The latter can be observed in Figs. 12.4a and 12.4b showing the breakdown of the time per event for inner detector track-seeding and calorimeter clustering, respectively. This illustrates the importance of implementing a suitable event data format in the offline and trigger code to avoid expensive data-format conversions. More details can be found in [12.11].

Based on the measurements made with the demonstrator, it is estimated that using the tested hardware it would cost approximately the same to increase the farm throughput by adding GPGPU or CPU. For example, it is estimated that two hypothetical racks, one with 120 CPU and another one with 80 CPU plus 40 GPGPU, would have approximately the same cost, throughput and cooling requirements (assuming that one GPGPU serves two CPU and gives a 50% increase of the CPU throughput and that a unit with 4 GPGPU costs about half of a unit with 8 CPU). However, the cost-effectiveness of adding GPGPU
to the EF depends on the evolution of CPU and GPGPU in terms of price, performance and packaging. Implementing trigger algorithms to run on GPGPU requires redesigning and parallelising the code to efficiently use the GPGPU hardware. For the demonstrator, this task took approximately 0.5 staff years per algorithm and relied on expert knowledge and experience to implement performant code.

The Inner Detector (ID) algorithm discussed above employs a combinatorial approach to inner detector pattern recognition. A second GPGPU demonstrator has been developed to assess an alternative pattern-matching approach similar to the method employed by rHTT and gHTT. A high-end gaming card (Nvidia Titan X) with a GP102 Pascal GPGPU chip has been used for these studies. The pattern data were stored in the GPGPU registers and event data were read from the GPGPU device memory for each new event. For this setup, the number of patterns per GPGPU is limited by the GPGPU register size to about 0.57 million. The patterns are processed by 56 blocks of threads (the GP102 has 28 multiprocessors that can each run two blocks simultaneously); each block has 1024 threads and each thread handles 10 patterns. Measurements have been made with L1Track patterns with eight layers, including one Pixel layer, using a simulated Phase-II dataset consisting of single muon events with the addition of a number of minimum bias events following a Poisson distribution about a mean that was varied for successive measurements. Fig. 12.3b shows the event processing rate as a function of the mean number of superimposed minimum bias events. Using GPGPU-based pattern matching, an event processing rate of about 140 kHz can be achieved for a pile-up of 200 minimum bias events. Based on a total bank size of 6 billion patterns (as for gHTT), a total of 10 thousand GPGPUs would be required for the full system.

In summary, GPGPUs have been studied as a potential commodity hardware accelerator to which suitable compute-intensive processing tasks can be offloaded from the main CPU. A study for the Phase-I upgrade based on current technology has concluded that GPGPUs and CPUs could be used to increase throughput of the EF at roughly the same cost and with similar power and space requirements. The relative cost-effectiveness of these technologies for the Phase-II EF depends on the relative evolution of CPU and GPGPU in terms of price, performance and packaging. The decision on the commodity compute hardware for the EF farm will be taken nearer the time of purchase following an evaluation of hardware accelerators. The above work should be taken as an indication that the software can be successfully adapted to other architectures in order to perform a full cost/benefit evaluation.

## 12.4 Model for CPU estimation

To estimate the EF CPU requirements for Phase-II for a given pile-up value $\mu$ the following model is used:
1. The HLT CPU usage of a data-taking run at $\mu = 43$ is measured and broken down into its individual components: ID Data-preparation (11%), ID Tracking (41%), Calorimeter (11%), Muon (26%) reconstruction and object reconstruction for electron, muon, jet, $E_T^{miss}$, etc. (11%).

2. The Inner Detector (ID) tracking is further split into the trigger-specific *fast tracking*, *precision tracking* and TRT tracking. The latter is subtracted for all further projections as this detector no longer exists in Phase-II.

3. For each object type (electron, muon, jet, etc.) a rate scale factor is derived based on the expected trigger rates after the regional tracking cuts shown in Table 6.4.

4. The CPU times for the individual components are scaled with the relevant rate and pile-up (see below) scale factors and the total CPU requirement in units of HS06 is calculated.

The following additional assumptions are made for the CPU projections:

(A) The CPU time for software tracking for ITk scales with $0.5 \times f_\mu$ compared to today's inner detector tracking where $f_\mu$ is the pile-up ratio (e.g. ITk tracking at $\mu = 200$ is 5 times slower than today's tracking at $\mu = 20$). This advantageous scaling results from the improved layout and design of the ITk detector for high-pileup conditions [12.12].

(B) The inner detector data preparation and muon spectrometer tracking scales with $f_\mu$, i.e. linear with pile-up. This assumption is based on past experience taking into account code improvements as the pile-up increases.

(C) The calorimeter, jet and $E_T^{miss}$ reconstruction times are constant with respect to pile-up when the appropriate pile-up noise corrections are applied. This has been verified during the current data-taking run.

(D) The CPU time required by today's *fast tracking* will be reduced by 80% thanks to the use of tracks from HTT. The remaining time is required for re-fitting the tracks obtained from HTT. This is based on measurements with the Run 2 FTK system.

(E) Tracking for hadronic triggers will be provided by gHTT. Consequently the CPU times for vertexing, beamspot measurement and other tracking-based pre-selections (e.g. the first stage of the two-stage tau tracking [12.6]) are reduced to zero.

(F) The unpacking and conversion time for hardware-reconstructed tracks received from HTT is 0.05 ms per track, which is extrapolated from today's 0.1 ms per track in the FTK system. We assume 2000 tracks at 100 kHz from gHTT and 50 tracks at 1 MHz from rHTT.

(G) The precision muon reconstruction is a known bottle-neck in today's HLT reconstruction software. We assume that a factor two speedup can be achieved on the timescale of Phase-II as described in Section 12.2.

(H) The jet and $E_T^{miss}$ reconstruction times are increased by 50% to take into account the additional use of (hardware) tracks in their calculation.

(I) Topo-clusters for the full event (in addition to the RoI-based topo-clusters used in the electron and tau reconstruction) are created at a rate of 100 kHz based on the rate of hadronic triggers.
Given the above model, the CPU estimates have some rather large uncertainties assigned to them. An attempt to quantify these has been made and is summarised in Table 12.2. Since the correlations of these uncertainties are unknown, the total uncertainty is given for the cases of fully (un)correlated uncertainties. For simplicity we quote in the following the average between these two extreme values as the uncertainty on the CPU estimate.

<table><caption>Table 12.2: Uncertainties on the CPU estimates in percent. The total uncertainty is given for the cases of fully (un)correlated uncertainties. Since the correlations are unknown we choose the average between these two extreme values as our model uncertainty.</caption><tr><td>Trigger and object rates after rHTT rejection are larger than expected</td><td>+20%</td></tr><tr><td>The CPU time for software tracking for ITk (A) scales with 1.0 × f<sub>μ</sub></td><td>+22%</td></tr><tr><td>The speedup in the fast tracking (D) due to the use of hardware tracks is only 50% instead of 80%</td><td>+11%</td></tr><tr><td>Hardware-reconstructed tracks are not re-used in the software tracking for electron and muon signatures (apart from the initial rejection)</td><td>+8%</td></tr><tr><td>±50% uncertainty on the unpacking times for HTT tracks (F)</td><td>±3%</td></tr><tr><td>±50% uncertainty on the speedup of the precision muon reconstruction (G)</td><td>±12%</td></tr><tr><td>Topo-clusters are created for 300 kHz of events</td><td>+9%</td></tr><tr><td>Hardware topo-clusters from the Global Trigger can be re-used in software</td><td>-4%</td></tr><tr><td>Total (correlated)</td><td>+83% -20%</td></tr><tr><td>Total (uncorrelated)</td><td>+36% -13%</td></tr><tr><td>Total ((corr+uncorr)/2)</td><td>+59% -17%</td></tr></table>

Table 12.3 summarises the CPU requirements in million HEP-SPEC06 (MHS06) 200 pile-up interactions, split into the different reconstruction domains.

<table><caption>Table 12.3: EF CPU requirements in million HEP-SPEC06 (MHS06) for 200 pile-up interactions for the different reconstructions domains.</caption><thead><tr><th>CPU [MHS06]</th><th>μ = 200</th></tr></thead><tbody><tr><td>HTT Unpacking</td><td>0.30</td></tr><tr><td>ID Dataprep.</td><td>0.91</td></tr><tr><td>ID Tracking</td><td>0.99</td></tr><tr><td>Muon tracking</td><td>1.37</td></tr><tr><td>Calo</td><td>0.32</td></tr><tr><td>Egamma/Tau</td><td>0.08</td></tr><tr><td>Jet/MET</td><td>0.50</td></tr><tr><td>Total</td><td>4.47<sup>+2.7</sup><sub>-0.7</sub></td></tr></tbody></table>
<figure><img src="image_211.png" /><figcaption>Figure 12.5: EF CPU extrapolation vs pile-up in million HS06 for the different components of the EF reconstruction software. Only the effects of pileup are considered but trigger rates remain the same.</figcaption></figure>

Based on the above model an attempt can be made to estimate the CPU requirements without rHTT. In case no rHTT is available, the input rate to the EF increases from 400 kHz to 1 MHz and all regional tracking needs to be done in software. In order to estimate the CPU required in this scenario the same model as explained above is used with the following changes. For each object type (electron, muon, tau, etc.) the L0 rate shown in Table 6.4 is used to determine the rate of regional tracking (e.g. this results in a factor five increase for single electrons). In addition, assumption (D) above no longer holds and the full time for software tracking needs to be accounted for (resulting in a ≈ 70% increase of the ID data preparation and tracking CPU requirements shown in Table 12.3). On the other side, the time for converting rHTT tracks (F) can be neglected. Putting all this together would result in an additional CPU requirement of 13.4 MHS06 for the EF farm.

Without gHTT all full-event tracking will have to be done in software. From offline tracking studies with the ITk layout it is estimated that 270 HS06 per event are required to perform full-event tracking at $\mu = 200$ and with a $p_T$-cut of 900 MeV [12.12]. For 100 kHz of events this would result in an additional CPU requirement of 27 MHS06.

For both cases, the calculation is based on the currently available ITk tracking software used in the preparation of its TDR [12.12]. Further improvements in the tracking software are to be expected over the next decade, which could reduce the cost of software tracking. On the other side, the large uncertainties in the market evolution for commodity computing make it difficult to do a direct cost comparison between a software-based and a custom hardware tracking solution with the latter being less susceptible to market driven cost changes.
References

[12.1] New Intel Core Processor Combines High-Performance CPU with Custom Discrete Graphics from AMD to Enable Sleeker, Thinner Devices, (online).
https://newsroom.intel.com/editorials/new-intel-core-processor-combine-high-performance-cpu-discrete.

[12.2] Unleashing High-Performance Computing Today and Tomorrow, (online).
https://itpeernetwork.intel.com/unleashing-high-performance-computing.

[12.3] H. Meinhard and B. Panzer-Steindl, Technology Evolution, WLCG Workshop 2016, https://indico.cern.ch/event/555063/contributions/2285842/.

[12.4] J. Baines, T. Bold, P. Calafiura, S. Kama, C. Leggett, D. Malon, G. Stewart, and B. Wynne, ATLAS Future Framework Requirements Group Report, Tech. Rep. ATL-COM-SOFT-2014-048, CERN, Geneva, Dec, 2014.
https://cds.cern.ch/record/1974156.

[12.5] J. Baines, T. Bold, D. Oliveira Damazio, S. Martin-Haugh, J. Masik, T. Martin, F. Pastore, H. J. Stelzer, W. Panduro Vazquez, R. White, W. Wiedenmann, F. Winklmeier, B. Wynne, C. Bernius, M. Hodgkinson, C. Leggett, E. Moyse, I. Shapoval, S. Snyder, and G. Stewart, Design of the High Level Trigger in AthenaMT, Tech. Rep. ATL-COM-DAQ-2017-047, CERN, Geneva, Jun, 2017.
https://cds.cern.ch/record/2268736.

[12.6] ATLAS Collaboration, Performance of the ATLAS Trigger System in 2015, Eur. Phys. J. C 77 (2017) 317, arXiv:1611.09661 [hep-ex].

[12.7] ATLAS Collaboration, Topological cell clustering in the ATLAS calorimeters and its performance in LHC Run 1, arXiv:1603.02934 [hep-ex].

[12.8] D. Emeliyanov and J. Howard, GPU-Based Tracking Algorithms for the ATLAS High-Level Trigger, Journal of Physics: Conference Series 396 (2012) no. 1, 012018.
http://stacks.iop.org/1742-6596/396/i=1/a=012018.

[12.9] ATLAS Collaboration, P. Loch, S. Menke, P.-A. Delsart, C. J. Young, Z. Marshall, and B. Axen, Topological cell clustering in the ATLAS calorimeters and its performance in LHC Run I, https://cds.cern.ch/record/1967028.

[12.10] R. Duda and P. Hart, Use of the Hough transformation to detect lines and curves in pictures, Communications of the ACM (1972) no. Volume 15 Issue 1, 11–15.

[12.11] ATLAS Collaboration, P. Conde Muiño, MultiThreaded Algorithms for GPGPUs in the ATLAS High Level Trigger, Tech. Rep. ATL-DAQ-PROC-2016-045, CERN, Geneva, 2016. https://cds.cern.ch/record/2239823.
[12.12] ATLAS Collaboration, *Technical Design Report for the ATLAS Inner Tracker Pixel Detector*, Tech. Rep. CERN-LHCC-2017-021, CERN, Geneva, (in preparation).
# 13 Hardware-based Tracking for the Trigger
(HTT)

Charged particle reconstruction in the high-pileup conditions expected at the HL-LHC presents a special challenge for the trigger. Use of information from the upgraded tracking detector ITk as early as possible in the trigger selection is a key ingredient in the ATLAS Trigger strategy for the Phase-II TDAQ upgrade. This chapter describes the ATLAS baseline design of a Hardware-based Tracking system for the Trigger (HTT).

Different technologies may be considered for a tracking system for the EF: 1) a hardware-based system (HTT) based on custom-designed Associative Memory (AM) ASICs for pattern recognition and FPGAs for track reconstruction and fitting, and 2) commodity CPU-based servers with or without accelerators (e.g., GPGPUs). The baseline option, in which the HTT is used as a tracking co-processor in the EF, meets the high trigger rate and throughput requirements at the HL-LHC. Several reasons motivate this decision, which include considerable experience in the AM technology, the potential for short latency, a lower power budget and less demanding space requirements compared to other technologies, its cost effectiveness and the independence of its cost from the commodity computing market, availability of in-house expertise, and the capability to evolve the HTT system for use in the hardware-based Level-1 trigger, should ATLAS need to change to a dual L0/L1 trigger system, as described in Chapter 14. Studies of GPGPU usage in the EF, are detailed in Section 12.3, and a discussion of the estimated EF CPU resources that would be required without the HTT is presented in Section 12.4.

As illustrated in Fig. 12.1, depending on the trigger signature, two types of requests from the EF will be transmitted to the HTT: finding tracks in regions of interest identified by the previous stage of selection (regional tracking, rHTT), and reconstructing tracks in the entire ITk coverage (global tracking, gHTT). Both regional and full-scan track reconstruction over the full ITk acceptance ($|η| < 4$) are provided by the same hardware system.

The rHTT searches for all tracks with $p_T > 2$ GeV in limited regions around Level-0 trigger objects that can profit from track information (for example, single high-$p_T$ leptons and four-jet triggers, as summarised in Table 6.5). The rHTT will operate at the full L0A rate of 1 MHz and will process on average 10% of the ITk detector data in these events (see Section 6.12 for a detailed discussion of this design parameter). For these tracks, single-stage reconstruction is performed using eight ITk detector layers¹; this choice is the result of optimising the overall rHTT performance. As demonstrated by the studies presented in
Chapter 6, regional tracking will accomplish a significant reduction in the Level-0 output
rate from 1 MHz to 400 kHz; the remaining events will be further processed by the EF.

The gHTT searches for all tracks with $p_T$ > 1 GeV at a nominal rate of 100 kHz (see Section 6.12 for a detailed bottom-up estimate of this rate). For these tracks, in addition to a first stage of track reconstruction (in common with the rHTT tracks), a second stage of processing is performed over all ITk detector layers to provide high-quality tracks with better purity and track parameter resolutions than those found by regional tracking. These tracks enable EF algorithms such as lepton isolation, primary vertex reconstruction, b-tagging, and missing transverse energy reconstruction as described in Chapter 6 and summarised in Table 6.6.

The implementation of the HTT design is an evolution of the FTK [13.1] system being commissioned in Run 2, which in turn builds on a similar system developed for the Silicon Vertex Tracker [13.2][13.3] at the Collider Detector Facility (CDF) at the Fermilab Tevatron [13.4]. For the Phase-II upgrade an ATCA-based hardware implementation is foreseen, where tracking processor carrier boards each house two mezzanine cards. A high-level overview of the HTT system is given in Fig. 13.1; the number of boards in the system is allocated to have a six-to-one ratio of first-stage to second-stage track reconstruction. AM ASICs are used for pattern recognition, while FPGAs are used for hit clustering, track fitting, track extrapolation, and duplicate removal. A comparison between the HTT and the FTK is presented in Section 13.3.

Track reconstruction in the HTT proceeds as follows. The inputs to the HTT are ITk hits, which are transmitted via commodity network along with the tracking request(s) from the EFPUs. There are two stages of processing. The first stage of processing is the same whether the request is for regional or global tracking. The hits from the eight ITk layers are clustered into consecutive ITk strip or pixel channels (so-called “superstrips”). Next, these superstrips are compared to a large bank of pre-computed template patterns (derived from simulated single-muon tracks); this comparison is performed in the AM ASICs. For each hit pattern that matches a template, the track parameters and quality are computed from the corresponding full-resolution hits in a FPGA. Tracks which share more than a given number of ITk hits (duplicate tracks) are removed. rHTT tracks are complete after this stage and are transferred back to the EFPU. For global HTT processing, each eight-layer track candidate goes through a second stage of processing in which its track fit is extrapolated to the remaining ITk layers and associated to any matching hits. A full track fit is then performed to achieve the best possible track parameter resolution. Duplicate track removal is also performed on gHTT tracks after the second-stage track fit. gHTT tracks are then transferred back to the EFPU for use in the final EF decision.

<footnote n="1">¹ The eight ITk layers used in the first-stage processing are chosen amongst the strip and pixel detector layers as follows: up to seven strip layers, if available, complemented by the appropriate number of pixel layers (the one(s) furthest away from the luminous region) to reach a total of eight. Hence, the choice of layers and the mixture of strip vs. pixel layers varies as a function of η.</footnote>
An overview of the HTT hardware implementation is presented in Section 13.1, followed by a description of the interface to the EF in Section 13.2 and a comparison to FTK in Section 13.3. The functional description of the system is given in Section 13.4. Performance studies are shown in Section 13.5 and, finally, Section 13.6 describes each individual hardware component.

## 13.1 Overview of the HTT Architecture

The HTT is organised as an array of independent tracking units called HTT units as shown in Figure 13.1. The main hardware building blocks of the HTT are ATCA boards called Tracking Processors (TPs). There are two types of TP boards: Associative Memory Tracking Processor (AMTPs) and Second-Stage Tracking Processor (SSTPs). Each TPs connects to the EF processor farm through a HTT InterFace (HTTIF).

<figure><img src="image_212.png" /><figcaption>Figure 13.1: Overview diagram of the HTT system showing interconnections within HTT units and with the HTTIF.</figcaption></figure>

Each HTT unit comprises a set of six AMTP boards and one SSTP board and will perform both regional and global tracking in a specific $\eta - \phi$ region of the track parameter phase space. Each TP holds two mezzanine cards. The AMTP boards each hold two Pattern Re-
cognition Mezzanines (PRMs) and the SSTP boards each hold two Track-Fitting Mezzanines (TFMs), as shown in Figure 13.2.

<figure><img src="image_213.png" /><figcaption>Figure 13.2: Diagram of a HTT unit showing interconnections within HTT modules and with the HTTIF. Each HTT unit comprises six AMTPs and one SSTP. Each AMTP and SSTP house two mezzanine cards called PRMs and TFMs, respectively.</figcaption></figure>

The track-finding algorithm is divided into two stages; the first stage is used for both regional and global tracking, while the second is only used for global tracking.

Clustering is the first processing step of the HTT. Both the AMTPs and SSTPs cards share ITk clusters; all clusters for a given event are aggregated and then sent for processing to mezzanine cards. The heart of the first-stage processing is AM ASICs pattern matching. The AM pattern matching occurs in the PRM and incorporates 8 layers of the ITk. Candidate tracks from AM pattern matching are used to retrieve full resolution clusters and perform first-stage track fitting in the FPGA.

The second step is done only for global tracking where the tracks found in the AM step are extrapolated to the remaining ITk layers to find all hits belonging to them, and a full track fit is performed to achieve the best possible track parameter resolution, as required for the use cases of global tracking. This operation is called second-stage processing and is performed in the TFMs) that each house two FPGAs; two TFMs are mounted on each SSTP.
Several important considerations were taken into account in the system design. First, the partitioning of the **HTT** system is optimised by extrapolating the **FTK FPGA** resource usage to the **gHTT** using a ratio of 6 **AMTPs** to 1 **SSTP**. This results in a 50% logic utilisation in the **TFM FPGAs**, which satisfies the target requirement. Second, the **HTT** system is designed for regional tracking in 10% of the detector at 1 MHz using the outer eight **ITk** layers, which is sufficient to reduce the Level-0 output rate to 400 kHz. The use of inner layers also for regional tracking at constant system cost has been evaluated. For example, a reduction of regional processing from an average of 10% of the detector to an average of 7% of the detector (with a corresponding reduction in the number **AMTP** cards to be replaced with additional **SSTP** cards and **HTTIF** servers) would allow second-stage processing for regional tracking for an average of 7% of the detector data per event. Finally, the distribution of data from **HTTIF** to the seven **TP** cards is balanced to send a similar amount of **ITk** data per **TP** card. This is done to balance the amount of cluster-finding processing on each **TP** board.

The concept of a **HTT** $\eta - \phi$ region is distinct from the concept of a **RoI**. A **HTT** $\eta - \phi$ region comprises all the **ITk** detector elements intercepted by tracks that have track parameters $\eta$ and $\phi$ within the specified $\eta - \phi$ range, $p_T$ above a certain threshold (2 GeV for rHTT and 1 GeV for gHTT), $|z_0| < 15$ cm and $|d_0| < 2$ mm. Therefore, two **HTT** $\eta - \phi$ regions adjacent in $\phi$ share a number of **ITk** detector elements due to the curvature of tracks, while two **HTT** $\eta - \phi$ regions adjacent in $\eta$ will also share a number of **ITk** detector elements due to the range of track $z_0$ that they cover. This overlap is discussed further in subsequent sections and is taken into account fully in the dataflow estimates.

## 13.2 Interface with Event Filter

Tracking requests with the corresponding **ITk** data will be sent from **EFPUs** to the **HTT** over the network. The size, shape and number of **ITk** regions to be processed for regional tracking can be freely chosen within the **HTT** processing limits (up to 10% of **ITk** data at 1 MHz). A static mapping between **ITk** modules and the **HTT** units will be used. An **ITk** module will be mapped by the **EF** to one or more **HTT** units to allow for overlaps. Each **HTT** unit will be assigned to find tracks in a given **HTT** $\eta - \phi$ region that corresponds to all tracks with the $\eta$ and $\phi$ parameters within that range. In order to do this, the **ITk** module mapped to a **HTT** unit will include those needed to have full acceptance for the bending of the tracks in the magnetic field and for the size of the luminous region (beam spot).

The steps that will return **HTT** tracks are as follows:

1. The **EFPU** algorithm decides whether the request is for regional or global tracking. For global tracking, data from all **ITk** modules mapped to a given **HTT** unit will be sent to the unit for tracking. In the case of regional tracking, the **EFPU** will determine which **ITk** modules correspond to the **RoI** where tracks are requested. All information available at this stage of **EFPU** processing can be used to select the **ITk** modules. The tracking requests will be sent to all relevant **HTT** units.
2. The EFPU will build a message for each HTTIF that is involved in the processing of the given event. The message will contain data for the corresponding modules and for each module the indication of the HTT destination(s) (HTT card or input link). Multiple destinations are indicated to duplicate data to different HTT units connected to the same HTTIF within the HTTIF itself. The single message is important for keeping together all data to be processed for one request by the 4 HTT units connected to the same HTTIF. The message will also contain the indication of regional or global tracking to determine the $p_T$ threshold and whether to perform second-stage processing.

3. The HTTIFs will process messages, distributing module data to optical links feeding the TP cards

4. The HTT units will process incoming data providing first-stage or second-stage tracks back to the HTTIF.

5. Through the data processing, an identifier is kept that will allow returning the tracks found to the requesting EFPU.

Once an EFPU receives the tracks found by all relevant HTT units, a check for duplicate track removal will be performed amongst tracks found in AMTPs covering adjacent $\eta$ or $\phi$ regions. The chances of duplicate tracks appearing in different units is small, given that each AMTP has a unique set of patterns stored in its AM ASICs. For both regional and global tracking the duplicate removal is done at the level of the AMTP as described. For global tracking a second pass of duplicate removal is performed in the SSTP after second-stage track finding.

Figure 12.1 shows that for some events the EFPU will send, in a first step, an HTT request to the regional tracking and in a second step to the global tracking.

## 13.3 Comparison to FTK

The HTT design builds on the FTK experience. The basic firmware algorithms from FTK are also present in the HTT system as described in this chapter. From the hardware point of view, the HTT improves in modularity, reducing the type of cards to a single main board and two mezzanine cards. This reduces the number of interfaces between systems and with a corresponding reduction in the amount of firmware needed. The HTT will allow expedited commissioning for three main reasons. The first is intrinsically related to the higher modularity that will allow a minimal processing slice to be built with just two main cards. The second is that the HTT will be organised as a co-processor farm, as indicated in Fig. 13.1, in which each unit is independent of the others. An HTT unit is built of just 7 modules. For comparison the FTK, which works as a single system, combines 322 modules working together. The third is that the HTT will receive data from the network rather than directly from the detector. This will allow the commissioning to be initiated before the start of Run 4 data-taking. Most of the commissioning will be performed with simulated data
only using pre-production hardware as described in 16.4.4. The proposed system design is more integrated and structurally simpler than the current FTK, which will make the synchronisation of the components easier. Furthermore, the general network access for HTT would in principle allow its usage also as an offline co-processor for Monte Carlo events.

# 13.4 Functional description of HTT

## 13.4.1 Data preparation

The data input to HTT are hits from the ITk. The pixel data is made into clusters using clustering algorithms. The clusters are converted to so-called superstrips, which are groups of consecutive silicon strip or pixel channels (see Fig. 13.4). Each superstrip is given a unique SuperStrip Identifiers (SSIDs). Pixel detector elements have coordinates in two dimensions and are formed into superstrips by dividing by both a superstrip width and a superstrip length. The superstrip width can be any integer value. The choice of superstrip dimensions is closely linked to the choice of the number of layers to use at the AM step and the choice of the actual layers, for a given fixed size of the pattern banks, and the choices presented in this Chapter are the outcome of optimisation studies. The list of strip and pixel superstrip widths used in the performance studies is shown in Table 13.1. These widths were found to yield a relatively low number of pattern matches from combinatorial background, while offering high efficiency for single muons and a total number of patterns within the expected hardware capacity. The choice of superstrip width is important as it has a strong effect on the number of patterns required and the number of false matches; hence, different superstrip widths are used in different parts of the detector.

## 13.4.2 Pattern matching

The HTT pattern recognition method is based on the FTK method currently used for the ATLAS Inner Detector, applied to the ITk layout and the HL-LHC levels of pile-up. In this method, a large number of single-muon tracks in simulated training events is used to form template patterns with coarse-granularity SSIDs from a number of layers of the ITk. A collection of these template patterns is called a pattern bank.

In the hardware implementation (described in detail in Section 13.6), these banks are stored in AM ASICs in the PRM. A pattern describes a sequence of eight SSIDs in different layers of the detector.
<figure><img src="image_214.png" /><figcaption>Figure 13.3: Layers used for 1st and 2nd stage fitting in four η-regions. The hits from the eight layers used in the first-stage processing are shown in blue, while hits in layers used in the second-stage processing are shown in green.</figcaption></figure>

**Layers** Choosing which of the available layers to use affects the number of patterns required, the number of false matches and the resolution of the fitted track. The two sides of the double-sided strip staves (petals in the end-caps) are treated as two separate layers while one physical pixel layer is one layer in the pattern bank. Which layers are used depends on the pseudorapidity of the track. For the barrel region, a fixed set of layers is used. For the transition and end-cap regions, multiple sets of layers are defined depending on which layers are hit in the training events. The first set of layers at low η are all barrel layers and progressively going to higher η more end-cap layers are substituted in each subsequent set. At high η, tracks do not leave hits in the strip end-cap layers, hence multiple pixel end-cap hits must be used. Figure 13.3 shows layers used for first- and second-stage fitting in four η-regions.

**Creating the pattern bank** The pattern banks are generated from the full simulation using training muons. It is planned to speed up the pattern generation in the future using samples with multiple training muons. In each of the $\phi \times \eta = 0.2 \times 0.2$ regions up to 100M muon events are processed. Clusters from ITk layers hit by the training muon are recorded. For each event, only clusters with a bar-code indicating that they come from the primary interaction are considered. Each cluster is converted to a superstrip with a width chosen for its layer (for pixel clusters this is done for both coordinates). The superstrip is then stored in the appropriate logical layer in a pattern. If the pattern already has a hit stored for the logical layer, the new hit is ignored, i.e. patterns are not made with all the possible combinations. If there are enough layers hit in the training event, the pattern is inserted into a
set. If the insertion fails because there is already a pattern with this sequence of superstrips, the usage count of the existing pattern in the set is incremented to indicate that it has been found again. At the same time, patterns with superstrips of twice and four times the width are also made and inserted into pattern sets. The relationship between the stored patterns for different widths of superstrips is stored for later creation of pattern banks with variable width superstrips. The usage count of a pattern is later used to reduce the size of the pattern bank to fit the available space in hardware. A pattern with low usage count is a rare pattern typically created by a scattered track. Removing patterns with low usage count has a very small effect on signal efficiency but a large effect on space required to store patterns.

**Variable-width superstrips and “Don’t Care” bits** Once all the patterns have been made, similar patterns (tracks A and B in Fig. 13.4) can be combined by setting some of the least significant bits of the superstrip number to be ignored in some patterns. Such bits are called “Don’t Care” (DC) bits. The use of DC bits offers better efficiency and lower pattern matching rate for a fixed number of patterns. The DC bits are assigned with the following procedure. Using the relationship between the finest granularity pattern bank and a corresponding bank with double the superstrip size, a list is built of all the “child” patterns shared by each “parent”. These lists are iterated over, comparing the patterns to see if they can be combined into a single pattern by ignoring the least significant bit of the superstrip number. This process can set at most 1 ignore bit per layer, since the parent list has double the superstrip width. To add more DC bits on a single layer, the process is repeated with a parent created after doubling again the superstrip width etc. The number of DC bits that may be set can be specified on a per layer basis.

Finally, the remaining patterns are sorted in decreasing “popularity” and written to the stored pattern bank. The “popularity” of a pattern is the number of muons that would create this same pattern.

<figure><img src="image_215.png" /><figcaption>Figure 13.4: Illustration of tracks traversing layers divided into superstrips in a tracker.</figcaption></figure>
**Wild cards** When a pattern has fewer than eight layers hit, the missing layers are marked as wild cards in the pattern. In the pattern matching stage, wild cards are always considered to match. The missing hits in the training muon tracks is usually due to the particle passing through an inactive part of the detector. Figure 13.5 shows where there are missing hits in the silicon strip sensors on both sides of a stave as a function of η and z₀. In the left hand plot, the gaps between the silicon strip sensors can be clearly seen, since the gaps are aligned for tracks travelling almost vertical away from the beamline in this η range. The pattern bank for this range has around 150k out of a million patterns with two wildcards. In the right-hand plot, no correlation can be seen since tracks at the larger η that go through a gap on one side of the stave hit the edge of the module on the other side. The pattern bank for this range has only 17k out of a million patterns with two wildcards.

<figure><img src="image_216.png" /><figcaption>Figure 13.5: Correlation between missing hits on opposite sides of the same stave in the ITk strips in the η range 0.1-0.3 on the left and 0.7-0.9 on the right.</figcaption></figure>

Since missing hits due to non-geometrical inefficiencies are rare, patterns with accidental DC bits would have a low “popularity” and would be unlikely to make the cut on the number of patterns described below.

Wild cards are only used to account for inefficiencies due to detector geometry, not to reduce the size of the pattern bank required. For the latter, variable width superstrips are used as described above.

### 13.4.3 Track fitting

**First-stage track fitting** Track fitting is implemented in firmware and performed in an FPGA in the PRM. The firmware implementation will be discussed 13.6. It takes the full-resolution hits from the roads passed by the pattern matching and calculates the track parameters and χ² of the fit. The track parameters pᵢ are calculated using a linear interpolation:

$$p_i = \sum_{j=1}^{N} C_{ij}x_j + q_i, \qquad (13.1)$$
where $x_j$ are the full-resolution local cluster coordinates and $(C_{ij}, q_j)$ are constants that are unique for each sector, where a sector is defined as a combination of one module from each of the 8 layers used in the first-stage processing. The constants are determined from a large sample of simulated muon tracks with the same parameter ranges and distributions as those used in generating the patterns (see section 13.4.2). The quality of the fit is evaluated using a linearised $\chi^2$ method which is fast to compute with FPGAs:

$$ \chi^2 = \sum_{i=1}^{N-5} \left( \sum_{j=1}^{N} A_{ij}x_j + k_i \right)^2, \qquad (13.2) $$

where $A_{ij}$ and $k_i$ are additional constants needed per sector. Several thousand sectors, each with its one sets of constants, are required for fitting a $\eta \times \phi = 0.2 \times 0.2$ region. Each sector consists of multiple sets of constants to cover possible wildcard configurations in the pattern banks. Without any compression or optimisation, a $0.2 \times 0.2$ region requires about 40 million coefficients, which need to be stored either in external memories on the PRM or in the internal FPGA memory. Preliminary studies to reduce this number indicate that significant savings are possible.

**Second-stage track fitting** The second-stage fitting is performed in a FPGA on the TFM. For each track the TFM calculates the 5 helix parameters and the $\chi^2$ of the fit. The TFM receives through the SSTP the 8-layer tracks from 6 PRM cards and all the hits from the detector layers not used by the PRM. There are two main functions carried out on the TFM: the Extrapolator and the Track Fitter. The Extrapolator finds the hits on the additional silicon layers that are close to a PRM track, starting from the clusters associated to the first-stage track. The Track Fitter fits the hits on the PRM track with each combination of hits on the other layers and applies a $\chi^2$ cut. Those track candidates passing the cut are sent to the SSTP where duplicate track removal is carried out before sending the tracks to the Event Filter.

The first step in the Extrapolator process is the Data Organiser, which is a database built on the fly that stores the hits for each detector layer so that those near a PRM track can be rapidly retrieved without having to scan the full hit list. The Extrapolator uses the hits in the PRM track, along with the detector sector the track is in, to define regions in the other layers in which to search for hits. Those hits along with the PRM track are sent to the Track Fitter. For each track candidate, the Track Fitter carries out linear calculations to determine each of the helix parameters and the $\chi^2$ of the fit (Equations 13.1 and 13.2). The inputs to the calculation are the hit coordinate in each detector layer and a set of pre-stored constants that differ for each sector of the detector.
### 13.4.4 Duplicate removal

The removal of duplicate tracks in an event is done with the HitWarrior algorithm run in the TP. The HitWarrior is a part of both track fitting stages. To identify duplicates, the HitWarrior first forms groups of tracks that share more than a given number of ITk hits, $N_{min}$. The parameter $N_{min}$ can be adjusted to tune the HitWarrior to achieve high efficiency and low duplicate rate. Tracks in a group are compared and those with fewer ITk hits are removed. Amongst tracks with the same total number of ITk hits, the one with the lowest $\chi^2$ is kept. The selection criteria could easily be replaced by a more sophisticated set in the future.

For gHTT, the fake rate after second state fitting is <0.2%, where fake tracks are defined as sharing less than 80% of its clusters with a truth track.

## 13.5 HTT Performance Studies

### 13.5.1 Associative Memory (AM) Pattern-Matching Performance

The performance studies described in this section were carried out predominantly for $p_T > 4$ GeV using full detector simulation in four representative $\eta - \phi$ HTT-regions of 0.2 x 0.2 within the acceptance of the ITk strips. In one of these regions ($0.7 < \eta < 0.9$), detailed studies all the way down to $p_T = 1$ GeV were also performed and used to extrapolate/compare to the performance in other regions and/or the entire detector. As will be shown, the performance vs. $\eta$ is rather uniform, lending confidence that the results are valid across the entire ITk coverage. In the pattern matching studies, one million patterns with the highest "popularity" were used for $p_T > 4$ GeV, two million patterns for $p_T > 2$ GeV, and four million patterns for $p_T > 1$ GeV. The four million patterns per HTT-region matches the overall proposed size of HTT.

The superstrip widths and selection of DC bits were chosen to ensure high efficiency within the one million patterns.

Adding more DC bits or increasing the superstrip widths would increase the number of fake matches due to combinatorics, indicating that the current working point is close to optimal. The track-finding efficiency was studied for single muons (and in some cases electrons and/or pions), with a flat $1/p_T$ spectrum between 4 (or 1) and 400 GeV, and the overall performance, including all dataflow estimates, was estimated from minimum bias events with $< \mu >$= 200 pile-up (referred to simply as pile-up) and from dijet events filtered so that a parton with $p_T > 30$ GeV points to one of the $\eta - \phi$ regions under study (referred to as "jets" throughout this section).

Using a configuration with 7 strip layers and a single pixel layer (the outermost barrel layer), and a "7-out-of-8" matching logic, an efficiency of ≈ 99% can be achieved, with
<table><caption>Table 13.1: Pattern-matching performance for the AM step simulated on minimum bias $<\mu>=200$ pile-up events ($\eta \times \phi = 0.2 \times 0.2$ regions, $p_T > 4$ GeV). All numbers are average per event. The superstrip dimensions and DC bits configuration (per layer) is also given (see text). The efficiency and the number of matches refers to the number of roads found during the AM processing. The found roads are input to the next processing steps.</caption><thead><tr><th>η range</th><th>muon eff.</th><th>mean matches pile-up</th><th>99% interval matches in pile-up</th><th>Superstrip width pixel</th><th>barrel</th><th>end-cap</th><th>DC bits</th></tr></thead><tbody><tr><td>0.1 &lt; η &lt; 0.3</td><td>99.1%</td><td>31</td><td>151</td><td>33/402</td><td>40</td><td></td><td>21111122</td></tr><tr><td>0.7 &lt; η &lt; 0.9</td><td>99.2%</td><td>21</td><td>93</td><td>33/402</td><td>40</td><td></td><td>21111122</td></tr><tr><td>1.2 &lt; η &lt; 1.4</td><td>98.8%</td><td>42</td><td>159</td><td>33/402</td><td>40</td><td>20</td><td>21111122</td></tr><tr><td>2.0 &lt; η &lt; 2.2</td><td>98.7%</td><td>10</td><td>56</td><td>16/200</td><td></td><td>10</td><td>21111122</td></tr></tbody></table>

a rate of matched patterns that can be handled by the fitter, as shown in Table 13.1 for $p_T > 4$ GeV. The inner and two outer layers use a maximum of two DC bits and the rest a maximum of one DC bit. The rates of matched patterns for $p_T > 2$ GeV and $p_T > 1$ GeV are shown in Tables 13.3, 13.4, 13.5, and 13.6.

**Pattern matching at high η** Beyond $|η|$ of 3.0 the tracks do not hit the ITk strip endcap disks at all; hence, only pixel information is available to make tracks. A pattern bank for the $η$ region 3.0-3.1 using only the pixel endcaps was created. Since this region is half of the size of the other regions under study, the target pattern bank size was 0.5M patterns but even without using DC bits the resulting bank only had 0.4M patterns. The performance is summarised in table 13.2.

<table><caption>Table 13.2: Pattern-matching performance in the $η \times \phi = 0.1 \times 0.2$ region, for $3.0 < η < 3.1$ and $p_T > 4$ GeV. The pattern bank uses pixel endcap layers only. All numbers are average per event.</caption><thead><tr><th>η range</th><th>muon eff.</th><th>mean matches muon</th><th>mean matches pile-up</th><th>Superstrip width pixel</th></tr></thead><tbody><tr><td>3.0 &lt; η &lt; 3.1</td><td>95.6%</td><td>25</td><td>8</td><td>16/200</td></tr></tbody></table>

**Performance with inefficient detector channels** As a crude way of simulating a detector with only $x\%$ efficient channels, the pattern matching has been run with a random $100 - x\%$ of clusters ignored. The results of this study are shown in Fig. 13.6. The simulation shows that by using 6/8 matches the efficiency will be 95% or higher (at a cost of increased pattern matches) and even with the baseline 7/8 matching logic, the pattern matching efficiency is above 95% for cluster efficiency of 97%. This is very much a worst case simulation, since whole clusters are dropped, not single hits. Moreover, large inefficiencies are not expected because there are more layers in the ITk than used for pattern matching. In the
case of failures where single detector elements or layers stop working, the patterns can be
regenerated using redundant elements or layers.

<figure><img src="image_217.png" /><figcaption>Figure 13.6: *Effect of random inefficient channels on pattern-matching efficiency.*</figcaption></figure>

**13.5.2 Track-Fitting Performance**

The simulated track-fitting performance at 99% track-finding efficiency (for single muons) in 0.7 < η < 0.9 region for muon, jets and minimum bias events at < μ > = 200 and for three p<sub>T</sub> thresholds is shown in Table 13.3. The jets used in the studies are weighted to give a p<sub>T</sub>-spectrum corresponding to Level-0. The table shows per event the average number of fits required to fit all the hit combinations found in pattern matching, the average numbers of tracks found after the quality cut, the average numbers of tracks after duplicate removal (HitWarrior) and average number of sets of first-stage fit constants<fnref n="2" /> required to fit the event. The presented numbers are used to extract the requirements for the system.

The fitting performance for first-stage track fitting in rHTT for single jets at < μ > = 200 is shown for four η-regions in Table 13.4. The results shown for p_T > 2 GeV have been extrapolated from studies done at p_T > 4 GeV using weights from the 0.7 < η < 0.9 region (extracted from Table 13.3).

The same detector configuration as in the pattern matching studies was used. The number
of fits is significantly reduced with the HitWarrior algorithm, resulting in a corresponding
reduction on the number of tracks found.

<footnote n="2">² A set of fitting constants includes all the pre-calculated values used to calculate the χ² and the track para-
meters.</footnote>
<table><caption>Table 13.3: First-stage track fitting performance for $0.7 < \eta < 0.9$ region at $< \mu >$= 200. All numbers are averages per event.</caption><thead><tr><th>particle</th><th>min p<sub>T</sub></th><th>Eff. (%)</th><th># roads</th><th># fits</th><th># tracks<br>χ<sup>2</sup> &lt; 40</th><th># tracks<br>HitWarrior</th><th># fit constants</th></tr></thead><tbody><tr><td>muon</td><td>1 GeV</td><td>99.5</td><td>144</td><td>1115</td><td>55</td><td>4.6</td><td>73</td></tr><tr><td>muon</td><td>2 GeV</td><td>99.1</td><td>79</td><td>586</td><td>23</td><td>1.9</td><td>40</td></tr><tr><td>muon</td><td>4 GeV</td><td>99.2</td><td>48</td><td>313</td><td>16</td><td>1.2</td><td>23</td></tr><tr><td>jets</td><td>1 GeV</td><td></td><td>195</td><td>1519</td><td>77</td><td>6.2</td><td>97</td></tr><tr><td>jets</td><td>2 GeV</td><td></td><td>104</td><td>804</td><td>29</td><td>2.4</td><td>52</td></tr><tr><td>jets</td><td>4 GeV</td><td></td><td>51</td><td>344</td><td>13</td><td>1.1</td><td>26</td></tr><tr><td>min-bias</td><td>1 GeV</td><td></td><td>110</td><td>842</td><td>38</td><td>3.6</td><td>58</td></tr><tr><td>min-bias</td><td>2 GeV</td><td></td><td>48</td><td>359</td><td>6</td><td>0.8</td><td>27</td></tr><tr><td>min-bias</td><td>4 GeV</td><td></td><td>21</td><td>133</td><td>1</td><td>0.2</td><td>12</td></tr></tbody></table>

<table><caption>Table 13.4: First-stage track fitting performance in jets at $< \mu >$= 200 ($p_T$ > 2 GeV studied in $0.7 < \eta < 0.9$, and extrapolated from 4 GeV to 2 GeV in other regions). All numbers are averages per event.</caption><thead><tr><th>η range</th><th># roads</th><th># fits</th><th># tracks<br>χ<sup>2</sup> &lt; 40</th><th># tracks<br>HitWarrior</th><th># fit constants</th></tr></thead><tbody><tr><td>0.1 &lt; η &lt; 0.3</td><td>170</td><td>1521</td><td>60</td><td>3.3</td><td>75</td></tr><tr><td>0.7 &lt; η &lt; 0.9</td><td>104</td><td>804</td><td>29</td><td>2.4</td><td>52</td></tr><tr><td>1.2 &lt; η &lt; 1.4</td><td>170</td><td>1402</td><td>71</td><td>4.8</td><td>90</td></tr><tr><td>2.0 &lt; η &lt; 2.2</td><td>65</td><td>240</td><td>64</td><td>3.7</td><td>20</td></tr></tbody></table>

Results for first-stage fitting in gHTT is shown for jets in Table 13.5 and for minimum bias in Table 13.6. The performance is extrapolated from 4 GeV to 1 GeV in the same way as for the first-stage track fitting in rHTT.

<table><caption>Table 13.5: First-stage track fitting performance in jets at $< \mu >$= 200 ($p_T$ > 1 GeV studied in $0.7 < \eta < 0.9$, and extrapolated from 4 GeV to 1 GeV in other regions). All numbers are averages per event.</caption><thead><tr><th>η range</th><th># roads</th><th># fits</th><th># tracks<br>χ<sup>2</sup> &lt; 40</th><th># tracks<br>HitWarrior</th><th># fit constants</th></tr></thead><tbody><tr><td>0.1 &lt; η &lt; 0.3</td><td>314</td><td>2874</td><td>159</td><td>8.5</td><td>138</td></tr><tr><td>0.7 &lt; η &lt; 0.9</td><td>195</td><td>1519</td><td>77</td><td>6.2</td><td>97</td></tr><tr><td>1.2 &lt; η &lt; 1.4</td><td>324</td><td>2649</td><td>189</td><td>12</td><td>167</td></tr><tr><td>2.0 &lt; η &lt; 2.2</td><td>125</td><td>454</td><td>171</td><td>10</td><td>37</td></tr></tbody></table>
<table><caption>Table 13.6: First-stage track fitting performance for minimum bias ($p_T$ > 1 GeV extrapolated from 4 GeV) at $<\mu> = 200$. All numbers are average per event.</caption><thead><tr><th>η range</th><th># roads</th><th># fits</th><th># tracks<br>χ<sup>2</sup> &lt; 40</th><th># tracks<br>HitWarrior</th><th># fit constants</th></tr></thead><tbody><tr><td>0.1 &lt; η &lt; 0.3</td><td>166</td><td>1481</td><td>76</td><td>7.2</td><td>82</td></tr><tr><td>0.7 &lt; η &lt; 0.9</td><td>110</td><td>842</td><td>38</td><td>3.6</td><td>58</td></tr><tr><td>1.2 &lt; η &lt; 1.4</td><td>218</td><td>1639</td><td>192</td><td>16</td><td>111</td></tr><tr><td>2.0 &lt; η &lt; 2.2</td><td>53</td><td>196</td><td>102</td><td>9.0</td><td>19</td></tr></tbody></table>

### 13.5.3 Muon and Electron Track-finding Efficiencies

The tracking efficiency for muons and electrons is defined as the fraction of events with at least a track with $\chi^2 < 40$, only for events in which an offline track is reconstructed. This is then the efficiency measured with respect to offline. The corresponding turn-on curves for first-stage track fitting, estimated on single muons and electrons not embedded in pile-up, are shown in Fig. 13.7. The efficiency for muons is quite flat along the full $p_T$ range, while electrons show a slower turn-on below 10 GeV. The reason of this lower efficiency at low momentum is motivated by the high radiation probability of these particles, which can reduce $p_T$ below the minimum 4 GeV threshold set in the rHTT system. This effect has been verified by excluding from the sample all the electrons below 10 GeV and having a recovery of the efficiency in that $p_T$ range.

<figure><img src="image_218.png" /><figcaption>Figure 13.7: First-stage muon and electron track-finding efficiencies in $\eta$ regions for muons (left) and electrons (right) for $p_T > 4$ GeV.</figcaption></figure>

### 13.5.4 Resolutions of Track Parameters

As a result of the fit procedures, the resolution on each parameter from first stage fitting is measured in single lepton sample with a spectrum flat in $1/p_T$ and with $<\mu> = 200$. The resolution is measured as the RMS of the 95% of the residual distributions (called rms$_{95\%}$),
<table><caption>Table 13.7: First-stage track fitting resolutions (rms<sub>95%</sub>) for electrons $p_T > 4$ GeV.</caption><thead><tr><th>η range</th><th>η</th><th>φ</th><th>q/P<sub>t</sub> [GeV<sup>-1</sup>]</th><th>d<sub>0</sub> [mm]</th><th>z<sub>0</sub> [mm]</th></tr></thead><tbody><tr><td>0.1 &lt; η &lt; 0.3</td><td>0.004</td><td>0.003</td><td>0.021</td><td>0.42</td><td>2.9</td></tr><tr><td>0.7 &lt; η &lt; 0.9</td><td>0.004</td><td>0.003</td><td>0.031</td><td>0.52</td><td>4.5</td></tr><tr><td>1.2 &lt; η &lt; 1.4</td><td>0.011</td><td>0.013</td><td>0.048</td><td>0.87</td><td>19.3</td></tr><tr><td>2.0 &lt; η &lt; 2.2</td><td>0.014</td><td>0.012</td><td>0.059</td><td>1.03</td><td>22.1</td></tr></tbody></table>

<table><caption>Table 13.8: First-stage track fitting resolutions (rms<sub>95%</sub>) for muons ($p_T > 4$ GeV).</caption><thead><tr><th>η range</th><th>η</th><th>φ</th><th>q/P<sub>t</sub> [GeV<sup>-1</sup>]</th><th>d<sub>0</sub> [mm]</th><th>z<sub>0</sub> [mm]</th></tr></thead><tbody><tr><td>0.1 &lt; η &lt; 0.3</td><td>0.002</td><td>0.001</td><td>0.003</td><td>0.20</td><td>0.8</td></tr><tr><td>0.7 &lt; η &lt; 0.9</td><td>0.002</td><td>0.001</td><td>0.003</td><td>0.20</td><td>0.8</td></tr><tr><td>1.2 &lt; η &lt; 1.4</td><td>0.004</td><td>0.003</td><td>0.007</td><td>0.33</td><td>3.8</td></tr><tr><td>2.0 &lt; η &lt; 2.2</td><td>0.004</td><td>0.003</td><td>0.014</td><td>0.71</td><td>7.1</td></tr></tbody></table>

in which 2.5% of each side tail is removed, and the obtained RMS is normalised to the expected gaussian width. The residuals are calculated from the difference of each parameter between the minimum $\chi^2$ candidate and the truth track parameter. In Tables 13.7 and 13.8 compare the values measured for electrons and muons respectively, for all the eta-regions considered. The resolution in $2.0 < \eta < 2.2$ region is expected to significantly improve if the barrel layer used in this study is replaced by a disk (the disk geometry was not available at the time of the study). The resolutions from these studies has been used for the trigger performance studies of muons, electrons and tau leptons presented in Chapter 6.

Figure 13.8 shows how the $z_0$ resolution for muons changes as a function of track $p_T$ for first-stage fitting.

<figure><img src="image_219.png" /><figcaption>Figure 13.8: Comparison of $z_0$ resolutions for muons as a function of track $p_T$ for first-stage fitting.</figcaption></figure>
<figure><img src="image_220.png" /><figcaption>Figure 13.9: Comparison of the $z_0$ (left) and $d_0$ (right) resolution for first- and second-stage fitting and offline.</figcaption></figure>

Figure 13.9 shows the $z_0$ and $d_0$ resolution for 10 GeV muons as a function of $\eta$ for the HTT first- and second-stage processing and offline tracking resolution. Only the central region was studied for the second-stage fitting.

### 13.5.5 Simulated HTT Data Size

The data size has been simulated to allow data bandwidth calculations in the hardware. The requirements and estimations in Section 13.6 are derived from these simulations. The fraction of data in ITk, separated by layer, for different trigger objects is shown in Table 13.9. The average RoI fraction per event to be processed by the regional HTT is 2.3% and the average data fraction per layer is ranging between 3.5% and 6.8% for the barrel strip and pixel layers used in rHTT. This data is used for optimising the partitioning of the HTT system and for dataflow and bandwidth calculations.

The number of clusters per ITk layer in a $0.2 \times 0.2 \eta - \phi$ region for $0.7 < \eta < 0.9$, for pattern banks with different $p_T$ thresholds (corresponding to gHTT, rHTT and L1Track), is shown in Fig. 13.10 (left). The number of clusters for pattern banks with $p_T > 4$ GeV in four $0.2 \times 0.2 \eta$ regions across $\eta$ is shown in Fig 13.10 (right). The layers shown are those used in the first-stage processing, where Layer 0 is the outermost ITk pixel layer and the others are strip layers. The result is shown for jets in $< \mu > = 200$ pile-up. The cluster occupancy is dominated by minimum bias; events with a jet pointing in the given $\eta - \phi$ region only have about 10% higher occupancy. This data is used for calculating FPGA resources and power.
<table><caption>Table 13.9: The fraction of data read out in a given RoI for different trigger objects.</caption><thead><tr><th rowspan="2">Trigger</th><th rowspan="2">Object<br>Multi-<br>plicity</th><th rowspan="2">RoI<br>size</th><th rowspan="2">η - φ<br>Fraction</th><th colspan="5">Pixel Layer Data Fraction</th><th colspan="4">Strip Layer Data Fraction</th></tr><tr><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>0</th><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><td>single e</td><td rowspan="3">1</td><td rowspan="3">0.2</td><td rowspan="3">0.13%</td><td rowspan="3">4.8%</td><td rowspan="3">3.8%</td><td rowspan="3">1.8%</td><td rowspan="3">1.2%</td><td rowspan="3">1.0%</td><td rowspan="3">1.3%</td><td rowspan="3">0.8%</td><td rowspan="3">0.6%</td><td rowspan="3">0.4%</td></tr><tr><td>single μ</td></tr><tr><td>single γ</td></tr><tr><td>forward e</td><td rowspan="2">1</td><td rowspan="2">0.2</td><td rowspan="2">0.13%</td><td rowspan="2">4.8%</td><td rowspan="2">3.8%</td><td rowspan="2">1.8%</td><td rowspan="2">1.2%</td><td rowspan="2">1.0%</td><td rowspan="2">1.3%</td><td rowspan="2">0.8%</td><td rowspan="2">0.6%</td><td rowspan="2">0.4%</td></tr><tr><td>di-γ</td></tr><tr><td>di-e</td><td rowspan="2">2</td><td rowspan="2">0.2</td><td rowspan="2">0.25%</td><td rowspan="2">9.6%</td><td rowspan="2">7.6%</td><td rowspan="2">3.6%</td><td rowspan="2">2.4%</td><td rowspan="2">2.0%</td><td rowspan="2">2.6%</td><td rowspan="2">1.6%</td><td rowspan="2">1.2%</td><td rowspan="2">0.8%</td></tr><tr><td>di-μ</td></tr><tr><td>e - μ</td><td rowspan="2">2</td><td rowspan="2">0.2</td><td rowspan="2">0.25%</td><td rowspan="2">9.6%</td><td rowspan="2">7.6%</td><td rowspan="2">3.6%</td><td rowspan="2">2.4%</td><td rowspan="2">2.0%</td><td rowspan="2">2.6%</td><td rowspan="2">1.6%</td><td rowspan="2">1.2%</td><td rowspan="2">0.8%</td></tr><tr><td>single τ</td></tr><tr><td>di-τ</td><td rowspan="2">2</td><td rowspan="2">0.2</td><td rowspan="2">0.25%</td><td rowspan="2">9.6%</td><td rowspan="2">7.6%</td><td rowspan="2">3.6%</td><td rowspan="2">2.4%</td><td rowspan="2">2.0%</td><td rowspan="2">2.6%</td><td rowspan="2">1.6%</td><td rowspan="2">1.2%</td><td rowspan="2">0.8%</td></tr><tr><td>single jet</td></tr><tr><td>large-R jet</td><td>5.0</td><td>0.8</td><td>10.2%</td><td>60.0%</td><td>52.0%</td><td>34.0%</td><td>27.5%</td><td>23.0%</td><td>25.5%</td><td>20.0%</td><td>17.0%</td><td>14.5%</td></tr><tr><td>H<sub>T</sub></td><td>5.0</td><td>0.8</td><td>10.2%</td><td>60.0%</td><td>52.0%</td><td>34.0%</td><td>27.5%</td><td>23.0%</td><td>25.5%</td><td>20.0%</td><td>17.0%</td><td>14.5%</td></tr><tr><td>E<sup>miss</sup><sub>T</sub></td><td>2.8</td><td>0.8</td><td>5.7%</td><td>33.6%</td><td>29.1%</td><td>19.0%</td><td>15.4%</td><td>12.9%</td><td>14.3%</td><td>11.2%</td><td>9.5%</td><td>8.1%</td></tr><tr><td>VBF</td><td rowspan="2">2</td><td rowspan="2">0.8</td><td>4.1%</td><td>24.0%</td><td>20.8%</td><td>13.6%</td><td>11.0%</td><td>9.2%</td><td>10.2%</td><td>8.0%</td><td>6.8%</td><td>5.8%</td></tr><tr><td>Support</td><td>0.2%</td><td>1.6%</td><td>1.4%</td><td>0.8%</td><td>0.6%</td><td>0.5%</td><td>0.6%</td><td>0.5%</td><td>0.4%</td><td>0.3%</td></tr></tbody><tfoot><tr><td colspan="2">Average per Event</td><td></td><td>2.3%</td><td>17.7%</td><td>15.0%</td><td>9.1%</td><td>7.1%</td><td>6.0%</td><td>6.8%</td><td>5.1%</td><td>4.3%</td><td>3.5%</td></tr></tfoot></table>

<figure><img src="image_221.png" /></figure>

<figure><img src="image_222.png" /><figcaption>Figure 13.10: The average number of clusters as a function of layer for three pattern bank $p_T$ thresholds in an $\eta - \phi$ region with $0.7 < \eta < 0.9$ (left) and the number of clusters in four $0.2 \times 0.2$ $\eta - \phi$ regions as a function of detector layer (right). The average number of clusters (1 $\sigma$) are shown in the shaded bands.</figcaption></figure>
## 13.6 Description of the HTT Hardware and Firmware

The **HTT** core components shown in Fig. 13.2 will be custom electronic boards that receive **ITk** data from the **EF** via the **HTTIF**, perform the pattern recognition and return track candidates to the **EF**, again via the **HTTIF**.

The **TP** is the main **HTT** processing unit implemented using an **ATCA** main board, where the computing power is distributed among mezzanines plugged into it. The function of the **TP** is to receive input data, perform clustering, share data with other **TPs**, then map clusters to logical layers and send them for processing to the mezzanine cards. Once tracks are received back from the mezzanines, duplicate track removal is performed on the **TP** and then tracks are returned to the **EF** via the **HTTIF**. The physical **TP** boards will be used to implement two distinct functions: the first is the **AMTP** which will perform first-stage tracking using **ITk** data from up to 8 layers, and the second function is the **SSTP** which performs the second-stage processing in the case of gHTT, using all the remaining **ITk** layers. The **AMTP** module will house two **PRM**. The **SSTP** module will house two **TFM**.

For intra-crate communication a full-mesh backplane will be used to allow point-to-point connection, while inter-crate communication and communication with the rest of **TDAQ** will be handled through optical links.

The **TP** design features two **FPGAs** that will share the data transmission and control the data traffic through the board. For the Phase-II requirements, today's **FPGAs** can provide transceivers with enough bandwidth to handle the expected data traffic. Each board in the shelf will have the ability to communicate to any other board in the same **ATCA** shelf.

The **PRM** mezzanine is expected to occupy about half of the height of the **AMTP** and a depth of about 15 cm, which holds two **PRMs** per **AMTP**. The **PRMs** will perform AM-based pattern recognition and track fitting in **FPGA**.

Distributing most of the computing in the mezzanines allows the same **TP** to be used for the **SSTP**. Second-stage processing will be implemented with the **TFM** which will be mechanically and electrically compatible with the **PRM**.

The connection between each mezzanine and the mainboard will use a high-speed connector for data and possibly a separate connector for power. The power distribution to the **AM ASICs** is an important part of the design. To simplify this the DC-DC converters attached to the core of the **AM ASICs** will be installed directly on the **PRM** and controlled by it.

This modular design provides uniformity of the main boards, flexibility for processing in the mezzanines, and efficient management of the different components.
### 13.6.1 Dataflow requirements

The dataflow through the HTT system has implications for the size of the system and the required performance of each module. The bandwidth for the input ITk data is calculated separately for 1st stage layers and 2nd stage layers. The input to the calculation is taken from performance studies. The input parameters, their values and their sources, are given in Table 13.10.

**Data duplication** The model for first-stage calculations on the HTT-unit level is shown in Fig. 13.11. The HTT system massively parallelises its task by dividing the reconstruction into multiple sub-units, each responsible for delivering tracks with ($\eta_{track}, \phi_{track}$) contained in some rectangular region of the full detector space. Each HTTIF, HTT unit, TP board, PRM and TFM are responsible for such a region. However, a single silicon detector element may be necessary to reconstruct tracks in multiple such regions. For example, it is necessary to send data from the innermost pixel layer to multiple HTT units because the extended length of the beam-spot allows tracks of a given ($\eta_{track}, \phi_{track}$) to impinge upon pixel modules with a large spread in z. Similarly, bending of low-$p_T$ tracks in the magnetic field causes significant data duplication at large radii. The 'duplication factor' of a module is defined as the number of hardware units requiring data from that module.

Simulated samples are used to calculate the number of hardware units (i.e., TP boards, TFMs, etc.) to which a given silicon module must be sent so that the system is able to reconstruct all tracks in the full detector volume. This number is hence referred to as the 'duplication factor' of the module, defined for a specific set of hardware units. For each hardware unit in the system, single muons are used to establish a list of modules that are penetrated by the tracks which the unit is tasked to reconstruct. Next, minimum bias events are processed, with the number of hits per module recorded and multiplied by word size to find the total average event size per module. Average duplication factors are then computed by taking the weighted average of duplication factors for each module, with weights given by the per-module event sizes,

$$ \text{average duplication} = \frac{\sum_{\text{modules}} (\text{duplication factor}) \cdot (\text{module hits}) \cdot (\text{word size})}{\sum_{\text{modules}} (\text{module hits}) \cdot (\text{word size})}. $$

In this manner, duplication factors are computed at the level of the HTTIF, HTT, TP boards, PRMs, and TFMs using only the modules necessary for either the 1st or 2nd stage track-finding. Different factors are computed for the rHTT (2 GeV tracking) and gHTT (1 GeV tracking) systems, as the gHTT requires increased duplication in the outer layers to account for tracks with larger curvature. The duplication is more extreme for hardware units corresponding to smaller regions of $\eta - \phi$ space. The duplication factors are listed in Table 13.11.
<table><caption>Table 13.10: HTT parameters for rate, bandwidth, and internal dataflow calculations.</caption><thead><tr><th>Parameter</th><th>Value</th><th>Comment</th></tr></thead><tbody><tr><td>Regional detector readout rate</td><td>100 kHz equiv.</td><td>10% × 1 MHz: TDAQ TDR, Table 13.9</td></tr><tr><td>Global detector readout rate</td><td>100 kHz equiv.<sup>a</sup></td><td>TDAQ TDR, Table 13.9.</td></tr><tr><td>ITk strip data volume</td><td>0.5MB</td><td>ITk strip TDR, Table 16.2</td></tr><tr><td>ITk pixel data volume</td><td>2.4MB</td><td>ITk pixel TDR<sup>b</sup></td></tr><tr><td>HTT-region η × φ</td><td>0.2 × 0.2</td><td>region size used for performance studies</td></tr><tr><td>HTTIF coverage η × φ</td><td>2.66 × π/4</td><td></td></tr><tr><td>HTT unit coverage η × φ</td><td>2.66 × π/16</td><td></td></tr><tr><td>SSTP coverage η × φ</td><td>2.66 × π/16</td><td></td></tr><tr><td>TFM coverage η × φ</td><td>1.33 × π/16</td><td>~6.6 × (η × φ) HTT-region</td></tr><tr><td>AMTP coverage η × φ</td><td>0.44 × π/16</td><td></td></tr><tr><td>PRM coverage η × φ</td><td>0.22 × π/16</td><td>~1.1 × (η × φ) HTT-region</td></tr><tr><td>Max clusters/strip layer<sup>c</sup></td><td></td><td></td></tr><tr><td>HTT-region (Regional)</td><td>200</td><td>Fig. 13.10</td></tr><tr><td>Max clusters/strip layer<sup>c</sup></td><td></td><td></td></tr><tr><td>HTT-region (Global)</td><td>260</td><td>Fig. 13.10</td></tr><tr><td>Max clusters/pixel layer<sup>c</sup></td><td></td><td>Pixel TDR Table 2.7 and</td></tr><tr><td>HTT-region (Barrel L4)</td><td>210</td><td>Fig. 13.10</td></tr><tr><td>Max clusters/pixel layer<sup>c</sup></td><td></td><td>Pixel TDR Table 2.7 and</td></tr><tr><td>HTT-region (EC L4))</td><td>210</td><td>Fig. 13.10</td></tr><tr><td># of roads/HTT-region</td><td></td><td></td></tr><tr><td>(Regional)</td><td>170</td><td>Busiest HTT-region, Table 13.4</td></tr><tr><td># of roads/HTT-region</td><td></td><td></td></tr><tr><td>(Global)</td><td>270<sup>d</sup></td><td>Busiest HTT-region, Tables 13.5, 13.6</td></tr><tr><td># of 1<sup>st</sup> stage fits/HTT-region</td><td></td><td></td></tr><tr><td>(Regional)</td><td>1500</td><td>Busiest HTT-region, Table 13.4</td></tr><tr><td># of 1<sup>st</sup> stage fits/HTT-region</td><td></td><td></td></tr><tr><td>(Global)</td><td>2250<sup>d</sup></td><td>Busiest HTT-region, Tables 13.5, 13.6</td></tr><tr><td># of fit constants/HTT-region</td><td></td><td></td></tr><tr><td>(Regional)</td><td>90</td><td>Busiest HTT-region, Table 13.4</td></tr><tr><td># of fit constants/HTT-region</td><td></td><td></td></tr><tr><td>(Global)</td><td>140<sup>d</sup></td><td>Busiest HTT-region, Tables 13.5, 13.6</td></tr></tbody></table>

<footnote>a Combination of full-scan and large coverage tracking.</footnote>

<footnote>b Calculated from information therein.</footnote>

<footnote>c Layer defined as a silicon surface giving a cluster with full $\phi$-coverage.</footnote>

<footnote>d Average of jet and minimum bias events.</footnote>
<figure><img src="image_223.png" /><figcaption>Figure 13.11: Diagram of a HTT unit showing data duplication and resulting maximum data flow between different components in the unit. The use of a full mesh backplane for sharing clusters between the AMTPs in a unit is indicated with the sketch between the two bottom green boxes.</figcaption></figure>

<table><caption>Table 13.11: Duplication factors used in the bandwidth calculation</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>HTTIF 1st stage w/ 1 GeV tracks</td><td>1.4</td></tr><tr><td>HTTIF 1st stage w/ 2 GeV tracks</td><td>1.3</td></tr><tr><td>HTTIF 2nd stage w/ 1 GeV tracks</td><td>2.4</td></tr><tr><td>HTT Unit 1st stage w/ 1 GeV tracks</td><td>2.7</td></tr><tr><td>HTT Unit 1st stage w/ 2 GeV tracks</td><td>2.2</td></tr><tr><td>AMTP w/ 1 GeV tracks</td><td>4.4</td></tr><tr><td>AMTP w/ 2 GeV tracks</td><td>3.4</td></tr><tr><td>PRM w/ 1 GeV tracks</td><td>6.3</td></tr><tr><td>PRM w/ 2 GeV tracks</td><td>4.8</td></tr><tr><td>SSTP w/ 1 GeV tracks</td><td>4.6</td></tr><tr><td>TFM w/ 1 GeV tracks</td><td>6.4</td></tr></tbody></table>

For the 1st stage layers the data rate is expected to be 1.6 Tb/s for the 100 kHz full detector equivalent (10% of 1 MHz) of regional tracking. The same 1.6 Tb/s bandwidth will be used for the 1st stage of global tracking at 100 kHz. The 2nd stage layers used only for global tracking will use a bandwidth of 1.2 Tb/s. The bandwidth split between 1st and 2nd stage layers depends on the assignment of layers. Studies show that a baseline assignment of layers that keep data sharing under control for an HTT crate is in $\eta - \phi \ 2.66 \times \pi/16$. This
<table><caption>Table 13.12: Input data flow in HTT. The payload bandwidth (BW) per unit represents the payload data that the HTTIF receives and forwards per HTT unit.</caption><thead><tr><th></th><th>rHTT</th><th>gHTT 1<sup>st</sup> st.</th><th>gHTT 2<sup>nd</sup> st.</th></tr></thead><tbody><tr><td>event size</td><td>1.4 MB</td><td>1.4 MB</td><td>1.5 MB</td></tr><tr><td>event rate</td><td>0.1 × 1 MHz</td><td>100 kHz</td><td>100 kHz</td></tr><tr><td>raw data rate</td><td>1.1 Tb/s</td><td>1.1 Tb/s</td><td>1.2 Tb/s</td></tr><tr><td>raw data rate w/ board dup.</td><td>3.8 Tb/s</td><td>4.9 Tb/s</td><td>5.6 Tb/s</td></tr><tr><td>raw data rate w/ unit dup.</td><td>2.4 Tb/s</td><td>3.0 Tb/s</td><td>5.6 Tb/s</td></tr><tr><td>raw data rate w/ HTTIF dup.</td><td>1.4 Tb/s</td><td>1.6 Tb/s</td><td>2.9 Tb/s</td></tr><tr><td>HTTIF input BW</td><td>60 Gb/s</td><td>67 Gb/s</td><td>121 Gb/s</td></tr><tr><td>unit input BW</td><td>25 Gb/s</td><td>31 Gb/s</td><td>58 Gb/s</td></tr><tr><td>board input BW from HTTIF</td><td>4 Gb/s</td><td>5 Gb/s</td><td>58 Gb/s</td></tr><tr><td>equalised input BW from HTTIF</td><td>4 Gb/s</td><td>12 Gb/s</td><td>16 Gb/s</td></tr><tr><td>input payload BW per HTTIF</td><td></td><td>248 Gb/s</td><td></td></tr><tr><td>input payload BW per unit</td><td></td><td>112 Gb/s</td><td></td></tr></tbody></table>

assignment is used for the dataflow calculations, neglecting the $\eta$ dependence. The input dataflow from HTTIF to HTT is summarised in Table 13.12. The equalised input bandwidth from HTTIF is the per board bandwidth after equalising the bandwidth between AMTPs and SSTPs. The last two lines summarise the input bandwidth per HTTIF and per HTT unit. They are the sum of the corresponding values for rHTT, gHTT first stage, gHTT second stage. The HTT shall allow for some flexibility in the assignment of layers to first-stage and second-stage processing. In this regards, part of the bandwidth shown in Table 13.12 might be later reassigned between the first and second stages. As this section describes the HTT modules are limited by internal processing power rather than I/O bandwidth. A bandwidth of 10 Gb/s per link is assumed for all fibre links as well as for ATCA backplane links or on-board links.

**Data buffering** In order to study the buffer occupancy of the off detector HTT hardware an initial discrete event simulation has been implemented including reasonable latency estimates for the processing time of the different hardware components, data sizes and events rates.

From the initial results from this simulation, estimates of the buffer occupancy in terms of events and number of bits have been obtained.

The input rate to the simulation is 200 kHz divided equally between 10% of 1MHz from the regional tracking, and 100 kHz from the global tracking. The 100 µs potential input skew has not been included - it is assumed that events will be aligned, so any additional buffering to achieve this has been assumed.
For the input to the Clustering Finder with 75 Kb per event, approximately 40 events will need to be buffered, which includes a factor of two contingency with respect to what is required to achieve containment of 98% of events. This results in a 3 Mb buffer in total.

For the data Aggregator, directly following the Cluster Finder, 10 events will be required, to give a 750 Kb buffer. For the PRM input, 35 events will be required, again with a factor of two contingency, to yield a 2.7Mb buffer. The Duplicate Removal stage would require a 12 event buffer, for a total of 0.9 Mb. In total, the AMTP buffering would need to be approximately 8 Mb in size.

## 13.6.2 Tracking Processor (TP)

The TP is an ATCA board designed to host two FPGA Mezzanine Card (FMC) double mezzanines, which can be either PRM or TFM cards. The main functionalities implemented in the TP hardware are, broadly speaking, off-board I/O, on-board I/O, clustering of raw pixel data, time-alignment of the input data, interfacing to the ATCA crate, and monitoring of the internal firmware block dataflow. Off-board I/O will include receiving of raw detector data, the sharing of data (hits, clusters and tracks) with other TPs via the RTM or the ATCA mesh, transferring tracks to the HTTIF and/or SSTP boards. Data will then need to be time-aligned and exchanged with the on-board mezzanine cards, either to perform pattern recognition and track fitting or second-stage processing. Hits will be mapped onto logical layers in preparation for their processing by the PRM or TFM. The track fitting output will then be fed back for trigger the decision through the HTTIF.

Ancillary functionalities will include interfacing with the ATCA mesh, data switching for efficient sharing of the Pixel clustering results and monitoring of the dataflow at the I/O boundaries RTM, ATCA and FMC and between functional blocks within the TP.

The current preliminary design is based on two symmetric FPGA devices, sharing the input bandwidth and simultaneously connected to all the FMC mezzanines (Fig. 13.12). This design splits the bandwidth requirements among multiple FPGAs running concurrently, and could allow early tests to run on prototypes with a reduced number of FPGAs. Figure 13.13 breaks down these functions in logical blocks to be implemented within the electronics board.

As shown in Fig. 13.12 and 13.13, dataflow monitoring and debugging is included in the design, allowing monitoring and - possibly - injection of data in the various functional blocks to monitor and test them individually.

**Major Specifications** The functionality of the TP will be implemented in two FPGA devices. The two critical resources are the number of I/O pins required for the interface described above (on-board, off-board and monitoring) and the number of logic cells required to implement the data handling and clustering. To quantify these resources, the dataflow requirements summarised in Table 13.10 are used for calculating the dataflow in the TP. The
<figure><img src="image_224.png" /><figcaption>Figure 13.12: Main diagram of the TP, illustrating the external interfaces and the dataflow with the on-board FPGA devices. The read lines are for monitoring and debugging purposes.</figcaption></figure>
<table><caption>Table 13.13: AMTP and SSTP FPGA resources required in order to handle the dataflow described in Table 13.10, compared to the resources available in a potential choice of FPGA. It is assumed that each TP board houses two identical FPGAs, with all resources here corresponding to a single FPGA.</caption><thead><tr><th>Quantity</th><th></th><th>AMTP</th><th>SSTP</th><th>XCKU085</th><th>XCKU115</th></tr></thead><tbody><tr><td rowspan="7">Links</td><td>RTM, data input</td><td>4</td><td>3</td><td></td><td></td></tr><tr><td>RTM, cluster I/O</td><td>1</td><td>6</td><td></td><td></td></tr><tr><td>RTM, track I/O</td><td>1</td><td>6</td><td></td><td></td></tr><tr><td>Mezzanine</td><td>4</td><td>12</td><td></td><td></td></tr><tr><td>ATCA Mesh</td><td>1×13</td><td>1×13</td><td></td><td></td></tr><tr><td>FPGA-to-FPGA</td><td>4</td><td>4</td><td></td><td></td></tr><tr><td>Monitoring</td><td>2</td><td>2</td><td></td><td></td></tr><tr><td></td><td>Total Links</td><td>29</td><td>46</td><td>56</td><td>64</td></tr><tr><td rowspan="3">Logic cells (kCells)</td><td>Pixel clustering</td><td>490</td><td>490</td><td></td><td></td></tr><tr><td>Switching</td><td>70</td><td>70</td><td></td><td></td></tr><tr><td>Total</td><td>560</td><td>560</td><td>1088</td><td>1451</td></tr><tr><td rowspan="6">Memory [Mb]</td><td>HTTIF input buffer</td><td>1.6</td><td>2.5</td><td></td><td></td></tr><tr><td>Cluster input buffer</td><td>0.3</td><td>1.9</td><td></td><td></td></tr><tr><td>Cluster finder buffer</td><td>0.2</td><td>0.1</td><td></td><td></td></tr><tr><td>Aggregator buffer</td><td>0.1</td><td>0.2</td><td></td><td></td></tr><tr><td>HitWarrior buffer</td><td>0.1</td><td>0.2</td><td></td><td></td></tr><tr><td>Total</td><td>2.3</td><td>4.8</td><td>57</td><td>76</td></tr></tbody></table>

assumption is that input data is fed to the system asynchronously, but with sufficiently low time spread within an individual HTT unit consisting of six AMTPs interfaced with one SSTP through shelf-to-shelf optical links. The top block of the table describes the study's assumed input parameters and the bottom block shows the corresponding estimated requirements at the board/unit level. It is assumed that the detector data is duplicated at the input to the system so that all needed data is sent to each HTT unit. Within a HTT unit, in order to balance FPGA resource usage, pixel clustering is shared among the TP boards with the clusters sent to other TPs via the backplane.

The resulting dataflow estimates from Table 13.10 and Fig. 13.11 can be translated into resource requirements for the on-board FPGA logic. The underlying assumptions used are that I/O links will be implemented at a bandwidth of 10 Gb/s and pixel clustering requires ~ 90 FPGA kcells per Gb/s of input to be processed. Table 13.13 summarises the resulting resource requirement per AMTP and SSTP board. Please note that debugging and monitoring features are not accounted for in these estimates.

The resources listed are compatible with the implementation of the logic in two FPGAs. Each FPGA carries out all of the TP functions, but with half of the total data flow.

An early implementation of the AMTP set-up is being studied using pre-prototype hardware based on the Data Formatter board [13.5], and a FMC mezzanine card carrying 12 of
the current generation (v.6) AM ASICs (each with 128k patterns) and one Kintex Ultras-
cale 060 FPGA handling dataflow on the mezzanine, controlling the AM ASICs and doing
first-stage track fitting. The same FMC can also be interfaced to a standard PCI Xilinx FPGA
evaluation board for ATCA-independent tests and performance studies. Figure 13.14 shows
these components.

13.6.3 Pattern Recognition Mezzanine (PRM)

The PRM will contain 12 AM ASICs and a large FPGA. The FPGA will control the PRM,
prepare the input data to the AM ASICs, receive the matched roads, and perform the first-
stage fitting. The 12 AM ASICs will provide 4.6M patterns per PRM.

The PRM design will provide a direct connection between the FPGA and AM ASICs. The FPGA will be the controller and interface for the ASICs. There will be two groups of 6 AM ASICs each. Each group could be controlled independently, receiving different data and possibly working on separate events. For each road found by the AM ASICs, the road ID will be converted back to the 8 SSIDs by accessing a high bandwidth external RAM.

Figure 13.15 shows the processing functions implemented in the PRM. The first step in PRM processing will be converting ITk clusters received from the AMTP into SSIDs that will be sent to the AM ASICs for pattern matching. Next is the Data Organiser (DO) function, which is an on-the-fly database that stores clusters allowing their fast retrieval based on the roads found. The SSIDs are also sent to the DO for use as addresses in storing the clusters. This operation mode is called DO write mode. The DO uses three sets of on-chip memory for each detector layer: the Hit List Memory (HLM), the Hit List Pointer (HLP), and the Hit Count Memory (HCM). The HLM sequentially stores each hit received. The HLP stores the HLM address of the first hit stored for each SSID. The HCM stores the number of hits in each SSID. All clusters for a given SSID will have to arrive sequentially. This function is performed in parallel for the 8 logical layers.

Once all clusters for one event have been received and processed, the loading to the AM
ASICs will be complete and readout of matched roads can start. Each road IDs will be sent
to the DO and at the same time be used as an address into the external RAM to convert the
road ID into a set of 8 SSIDs and a sector number. The SSIDs are the pointers that the DO
will use to retrieve the full resolution clusters from memory. For each road found, the DO
will output in parallel, in a single clock cycle: the road ID, sector ID and one hit per layer.
Multiple hits in a layer will require additional clock cycles.<fnref n="1" /> This operation mode for the
Data Organiser is called DO read mode.

Both the DO and the AM ASICs have two-event buffers. While the first event is being
processed reading out AM roads with the DO in read mode, the second (next) event is

<footnote n="1">¹ For a road with at most one hit per layer a single clock cycle will be needed. For a road with 3 hits in the layer with the most hits 3 cycles will be needed.</footnote>
<figure><img src="image_225.png" /><figcaption>Figure 13.13: Main diagram of the TP, illustrating the major functional blocks to be implemented in the board.</figcaption></figure>
<figure><img src="image_226.png" /><figcaption>Figure 13.14: Hardware demonstration prototypes based on current-generation devices: AM mezzanine with Kintex Ultrascale FPGA (left), ATLAS Data Organiser ATCA carrier for FMC mezzanine cards based on a Virtex 7 FPGA (middle), and a Virtex 7 FPGA evaluation board (right) equipped with a mechanical adapter to host the AM mezzanine card.</figcaption></figure>

<figure><img src="image_227.png" /><figcaption>Figure 13.15: Main diagram of the PRM.</figcaption></figure>
loading SSIDs to the AM ASICs with the DO in write mode. The PRM can proceed to
process the third event once the first event has completed read mode and the second event
has completed write mode.

The road+hits packet from the DO will be sent to the Track Fitter module and buffered. For each road+hits packet, the Track Fitter will fit all combinations of hits looping over all combinations of one hit per layer. If one of the layers does not have a hit, then the combinations of one hit on each of the other 7 layers are used. For each combination, the $\chi^2$ is calculated in linear fits using the hit positions and a set of precalculated fit constants. A $\chi^2$ cut is applied to reject most fake tracks. For combinations passing the $\chi^2$ cut 5 helix parameters are calculated again in linear fits using the hit positions and additional constants. For both the $\chi^2$ and helix parameters, the fit constants are extracted from memory using the sector number as a pointer. The track fitting constants are stored in a combination of the internal memory and in an external RAM that are accessed in parallel.

Two types of fitters will be implemented. If one of the layers has no hit, then the Majority track fitter is used. One can still use the same set of constants to calculate the $\chi^2$ and helix parameters provided that there is an estimate of where the hit in the empty layer would have been. The missing hit coordinate is estimated using a linear function of the other hit coordinates. (This equation comes from minimising the $\chi^2$ over the position of the hit in the empty layer.) If there are hits in all layers, the nominal fitter is used.

The good tracks coming out of the track fitters are sent back to the AMTP. A track packet consists of the hit on each layer, the $\chi^2$, the 5 helix parameters, the track road, the sector number, and the hit map indicating which layers had real hits.

The PRM design aims at maximising the number of stored patterns and the bandwidth to the external memories. The limit is the number of I/O pins in the FPGA. This means that a large FPGA package is needed.

The two external RAMs are needed to reading SSIDs for the found roads, and constants for the fit. The baseline external RAM device is a reduced latency RAM chip (RLDRAM3) with a 1 Gbit size. The two external RAMs will use approximately 170 FPGA pins. The size to store the SSIDs is 4.6M patterns times 176 bits, which is just above 800 Mb. The size of the fit constants is assumed to be 6k bits, half of which are needed for the $\chi^2$ calculation and half for the calculation of the helix parameters. With the proposed device 160k sets of track fitter constants can be stored. The 3k bits required for the $\chi^2$ calculation need to be accessed at high rate. The maximum rate allowed is 30 MHz, which is a good match to our requirements. The internal RAM can also be used to store fit constants. This can provide additional bandwidth for the access to most frequently used constants.

The main connections between the FPGAs and each group of 6 AM ASICs are shown in Fig. 13.16. The input data to the AM and the majority of control lines will be sent from the FPGA to the first AM ASIC. Each AM ASIC will send the same data to the next AM ASIC until all 6 in a group receive the data. Within each group of 6 AM ASICs, a pair will share
a single output channel to the FPGA. The total number of pins used is 127 for each AM group, as indicated in Table 13.17. The total number of pins needed for external RAMs and AMs is 424.

<figure><img src="image_228.png" /><figcaption>Figure 13.16: Diagram with the main connections between the PRM FPGA and the 2 groups of AM ASICs.</figcaption></figure>

The PRM input bandwidth from the AMTP is chosen to be enough to feed both groups of AM ASICs at the maximum rate to cover peak AM ASIC use for the L1Track application. For each group of AM ASICs the maximum input bandwidth is 64 Gb/s, which is calculated as 8 layers times 250 MHz, which is the AM ASIC clock frequency, times 32 bits word size for one cluster.<fnref n="2" /> Given that the two groups of AM ASICs will receive shared data, we can allocate 120 Gb/s. For the usage of HTT in the baseline TDAQ 40 Gb/s between the TP and each PRM will be sufficient.

Table 13.14 reports rough estimates for the resource usage compared with resources available in a Xilinx KU085 FPGA chosen as an example FPGA for the PRM.

The main parameters for the PRM card are reported in Table 13.15. Several parameters are driven from AM ASIC performance, as discussed in Section 13.6.4. The main parameters that affect the processing speed are the cluster processing rate, road processing rate, fit constants access, and track fitting rate. These parameters are based on the following as-

<footnote n="2">² The word size for each cluster on the HTT modules is assumed to be 32 bits. These will be condensed to 16 bits for AM processing. The extra information is used for track fitting.</footnote>
<table><caption>Table 13.14: PRM FPGA resource usage compared with a Xilinx KU085.</caption><thead><tr><td>resource</td><td>required</td><td>available</td><td>usage %</td></tr></thead><tbody><tr><td>High speed links</td><td>12</td><td>48</td><td>25%</td></tr><tr><td>Logic cells</td><td>N/A</td><td>1088k</td><td>N/A</td></tr><tr><td>Block RAM</td><td>40 Mb</td><td>56 Mb</td><td>70%</td></tr><tr><td>DSP</td><td>1870</td><td>4100</td><td>50%</td></tr><tr><td>IO lines</td><td>60 + 400</td><td>104+520</td><td>80%</td></tr></tbody></table>

<table><caption>Table 13.15: Main parameters for the PRM card</caption><thead><tr><td>parameter</td><td>value</td><td>comment</td></tr></thead><tbody><tr><td>Total number of patterns</td><td>4.6M</td><td>in 2 independent gr. of 2.3M patterns</td></tr><tr><td>Peak cluster processing rate</td><td>250 MHz</td><td>in parallel for the 8 logical layers</td></tr><tr><td>Max average cluster processing rate</td><td>60 MHz</td><td>average over event and layers</td></tr><tr><td>Road readout rate</td><td>200 MHz</td><td>in parallel from each pair of AM ASICs</td></tr><tr><td>Road to 8 SSID conversion</td><td>500 MHz</td><td>from the external RAM</td></tr><tr><td>Road processing rate</td><td>200 MHz</td><td>assuming one cluster per layer.<sup>a</sup></td></tr><tr><td>Fit constants access</td><td>30 MHz</td><td>from external RAM</td></tr><tr><td>Track fitting rate</td><td>1 GFit/s</td><td>using internal DSP slices</td></tr></tbody></table>

sumptions: up to 12 coordinates are used in the fit<fnref n="3" />, integer multiplications with up to 18 bits for the local coordinate and up to 27 for the constants, and a constant size of 100-200 values of 27 bits. At most 2 missing coordinates are allowed from the sum of missing hits in the event and wildcards in the stored pattern.

Pre-prototypes of the PRM for R&D have been produced as part of the INFN RDphase2 activity. The last prototype produced was the PRM06 [13.6] that is shown in Fig. 13.17. The PRM06 is a generic R&D prototype for phase-II that uses available AM06 ASICs. Two high capacity and high bandwidth external memories (MT44K32M36) are installed.

The PRM06 is a demonstrator for the first prototype of the PRM card. The main structure could be similar, while the main change will be to upgrade to AM08 ASICs. The PRM06 has a Xilinx Kintex Ultrascale FPGA. A FPGA of the same family is a possible choice for the next prototype.

<footnote>a The number of cycles required is equal to the number of clusters found in the busiest layer.</footnote>

<footnote n="3">³ With 12 coordinates possible first-stage layer choices are: 1 pixel and 7 strips layers (using 9 coordinates), 2 pixels and 6 strip layers, or more pixel layers for up to 4 pixel and 4 strip, or 6 pixel layers that could be used in the forward region. Up to 8 pixel layers can be used discarding one coordinate in 4 layers or increasing the size of the constants.</footnote>
<figure><img src="image_229.png" /><figcaption>Figure 13.17: Pre-prototypes of the PRM card using FTK AM06 ASICs.</figcaption></figure>

### 13.6.4 Associative Memory (AM) for Phase-II

The core component of the hit pattern recognition will be the AM ASICs. This chip is a massively parallel system able to perform bit-wise comparison of incoming data with the pre-stored templates. It has been shown in reference [13.7] that the AM does better hit filtering than FPGA implementing Hough transform. The production version of the ASIC will be called AM09. As introduced in previous section, the relevant features for this device are: the number of patterns per chip, the input and output bandwidths, power consumption, and the number of layers, and bits per layer used to describe a pattern.

**Requirements** The Phase-II scenario will require more powerful AM ASICs, with respect to the current state of the art AM ASICs represented by the FTK Fast-Track Associative Memory ASIC (AM06). The current AM07 prototype [13.8] (based on 28 nm CMOS process) has been fabricated in August 2017 (Fig. 13.18) and is currently under test. The chip is fully functional and operates up to 250 MHz. LVDS drivers and receivers have been measured with good results up to 1.1 Gb/s. Power dissipation (TX+RX) is about 8 mW at 1 Gb/s. The design for AM08 ASICs is already ongoing.

With the aim to achieve the HTT requirements, key improvements in the next generation AM ASICs will be needed with respect to AM06:

* a **higher clock speed and bandwidth:** the next version of the AM ASIC will also increase the internal core-matching clock frequency, from 100 MHz to 250 MHz, and the core-readout clock frequency: from from 100 MHz to 200 MHz. With respect to FTK, the number of used layers for the pattern recognition are not expected to change hence the same number of layers will be available both in AM08 and AM09. For each
<figure><img src="image_230.png" /><figcaption>Figure 13.18: *The current prototype of the AM ASIC*</figcaption></figure>

of the 8 layers, 16 bit words can be entered at full clock speed with an input bandwidth
of 4 Gb/s per layer. We decided that one patterns is composed by 16 bits × 8 words.
The connection with the FPGA is expected to run at up to 1.6 Gb/s limited by the
Low-Voltage Differential Signaling (LVDS) protocol speed on general purpose FPGA
pins. LVDS driver and receivers working at 1.8 V will be designed for I/O ports with
a slightly higher bandwidth target of 2 Gb/s for usage with future generation FPGAs;

* **a larger density of patterns:** The cell size is expected to be about a factor 3 smaller, 675 $\text{-m}^2$ to 186 $\text{-m}^2$ per pattern, allowing to achieve the goal of storing (i.e., 384k patterns) 3 times more patterns in the same area: AM08 will contain 16k patterns and AM09s will contain 3×128k pattern. The suggested block size is 32k patterns, which requires 12 bits for a 3×128k patterns ASIC. We plan to allocate 16 bits to cover also the aggressive goal of 4×128k patterns;

* **a low energy usage per operation:** to achieve this requirement a new CAM cell has been designed: the KOXORAM [13.9]. Simulation of the new CAM cell predicts better energy efficiency: 0.30 fJ/comparison/bit compared to the 0.80 fJ/comparison/bit for the XORAM used in AM06. Other contributions to the AM ASIC power dissipation are to be accounted, resulting in an expected energy per operation for the device of 1 fJ/comparison/bit. A non-core power consumption should not exceed the 500 mW. In similar way, the idle core power consumption should not exceed the 500 mW;

* **a reduction of current peaks:** an implementation of a decoupling scheme in the Power Distribution Network (PDN) will be given to smooth the power peaks that caused minor system issues for the AM06.

* **a reduced latency:** to cope with the Level-1 needs in the evolved scenarios, and to avoid the integration of multigabit transceivers: The maximum pass-through latency (last hit in AM input, first road at AM output) must not exceed the 100 ns for AM09 and the (50 ns for AM08);

* **a more flexible layer threshold:** Since many application will use the chip, the quorum logic thresholds set will include all combination (\{0; 1; 2; 3; 4; 5; 6; 7; 8; 9\});
* a more flexible blocks enabling: the new chip will have the possibility to enable/dis-
able blocks of patterns on an event-by-event basis <fnref n="4" /> thus saving power since disabled
blocks would not make any comparison.

The HTT system also requires some specific AM ASIC global parameters that does not
change with respect to AM06:

* based on 28 nm CMOS process;

* the junction working temperatures ranges from 0 °C to 120 °C;

* AM ASIC must be mountable on both sides of the PCB.

**Work flow** The workflow toward the final ASIC is organised in three steps

* **AM08 prototype:** small area Multi Project Wafer (MPW) prototype to silicon proof all the features that will be integrated in Associative Memory 09 pre-production ASIC (AM09pre).

* **AM09pre pre-production:** the full area ASIC prototype will be fabricated with a reduced-cost production run <fnref n="5" />. The AM09pre will be developed starting from the AM08 prototype design and extending the memory area, therefore the specification of both versions must be compatible. The timing performance of the final production devices is ensured exploring the transistor performance variability with three "engineering corner runs".

* **AM09 production:** A second production run<fnref n="6" /> will be performed to include in the AM09 design refinements before going to industrial production.

**AM09 design** In this section we describe the design technique that we choose to adopted
to reach the HTT requirements:

* **Clock speed,bandwidth, and interface:** Two clocks will be used to synchronise the hit input and output road data: the CLK Hit Fast (CLKHF) running at 1 GHz, and the CLK Readout Fast (CLKRF) running at 800 MHz. The read-out latency could be further optimised exploiting a direct connection between the AM ASICs and the FPGA, performing first stage fitting within the PRM. The chip will receive detector hits through 8 input buses (HIT_IN), propagation within the chip to find correspondence with the pre-stored patterns. The results of the elaboration will be delivered through the ROAD_OUT buses. With the aim to simply the routing a cascading with a daisy chain architecture will be implemented at PRM level. For this reason, these data interfaces will be also copied in output on the HIT_OUT buses and in input on the ROAD_IN buses. Synchronicity will be obtained thanks to: CLKHF for the HIT_IN buses; CLKRF for all ROAD_IN buses; CLKHF_OUT for the HIT_OUT buses;

<footnote n="4">⁴ Implementation using a specific k-word at the beginning of the event</footnote>

<footnote n="5">⁵ Cost will be limited employing a Multi-Layer Masks (MLM) full-mask set pilot run</footnote>

<footnote n="6">⁶ still based on a MLM full-mask set pilot run</footnote>
<table><caption>Table 13.16: Frequency and serialisation capabilities for AM08 and AM09 signals</caption><thead><tr><th>SYMBOL</th><th>BITS / WORD</th><th>SERIAL. FACT.</th><th>WORK. FREQ. (MHz)</th><th>BW (Gbps)</th><th>WORDS</th><th>AURORA?</th></tr></thead><tbody><tr><td colspan="7" style="text-align: center; font-weight: bold;">ADDRESS</td></tr><tr><td>ADDR_IN</td><td>20</td><td>1:8</td><td>125</td><td>1</td><td>1</td><td>Yes</td></tr><tr><td colspan="7" style="text-align: center; font-weight: bold;">DATA</td></tr><tr><td>HIT_IN</td><td>16</td><td>1:4</td><td>250</td><td>2 (DDR)</td><td>8</td><td>Yes</td></tr><tr><td>HIT_OUT</td><td>16</td><td>4:1</td><td>250</td><td>2 (DDR)</td><td>8</td><td>Yes</td></tr><tr><td>ROAD_OUT</td><td>32</td><td>8:1</td><td>200</td><td>1.6</td><td>4</td><td>Yes</td></tr><tr><td>ROAD_IN</td><td>32</td><td>1:8</td><td>200</td><td>1.6</td><td>4</td><td>Yes</td></tr><tr><td colspan="7" style="text-align: center; font-weight: bold;">CONTROLS</td></tr><tr><td>CTRL_IN</td><td>16</td><td>1:8</td><td>125</td><td>1</td><td>1</td><td></td></tr><tr><td>CTRL_OUT</td><td>16</td><td>8:1</td><td>125</td><td>1</td><td>1</td><td></td></tr><tr><td colspan="7" style="text-align: center; font-weight: bold;">CLOCKS</td></tr><tr><td>CLKHF</td><td></td><td></td><td>1000</td><td></td><td></td><td></td></tr><tr><td>CLKHF_OUT</td><td></td><td></td><td>1000</td><td></td><td></td><td></td></tr><tr><td>CLKRF</td><td></td><td></td><td>800</td><td></td><td></td><td></td></tr><tr><td>CLKRF_OUT</td><td></td><td></td><td>800</td><td></td><td></td><td></td></tr></tbody></table>

CLKRF_OUT for all ROAD_OUT buses. The ADDR_IN buses will be used to give in input the write/read address of memory bank and/or control registers. The chip will be essentially controlled by means of different channels: 1) a fast and sync control technique; 2) an adhoc fast control mechanism; 3) a slow control commercial protocol (i.e, Serial Peripheral Interface – SPI). Tables from 13.16 to 13.17 summarise the interface specifications for the AM08 and AM09.
<table><caption>Table 13.17: Pin count for AM08 and AM09 signals and connection via FPGAs</caption><thead><tr><th></th><th>AM08</th><th>AM09</th><th colspan="2">FPGA lines</th><th></th></tr></thead><tbody><tr><td>SYMBOL</td><td>#bumps (#LVDS pairs)</td><td>#bumps (#LVDS pairs)</td><td>1-AM09</td><td>6-AM09</td><td>HP/HR</td></tr><tr><td colspan="6">ADDRESS</td></tr><tr><td>ADDR_IN</td><td>8(4)</td><td>8(4)</td><td>8</td><td>8</td><td>HP</td></tr><tr><td colspan="6">DATA</td></tr><tr><td>HIT_IN</td><td>64(32)</td><td>64(32)</td><td>64</td><td>64</td><td>HP</td></tr><tr><td>HIT_OUT</td><td>64(32)</td><td>64(32)</td><td>0</td><td>0</td><td></td></tr><tr><td>ROAD_OUT</td><td>32(16)</td><td>32(16)</td><td>8</td><td>24</td><td>HP</td></tr><tr><td>ROAD_IN</td><td>32(16)</td><td>32(16)</td><td>0</td><td>0</td><td></td></tr><tr><td colspan="6">CONTROLS</td></tr><tr><td>CTRL_IN</td><td>4(2)</td><td>4(2)</td><td>4</td><td>0</td><td></td></tr><tr><td>CTRL_OUT</td><td>4(2)</td><td>4(2)</td><td>4</td><td>0</td><td></td></tr><tr><td>SPI</td><td>4</td><td>4</td><td>4</td><td>4</td><td>HR</td></tr><tr><td>MODE</td><td>6(3)</td><td>6(3)</td><td>6</td><td>6</td><td>HP</td></tr><tr><td>HOLD_IN</td><td>4</td><td>4</td><td>1</td><td>3</td><td>HR</td></tr><tr><td>HOLD_OUT</td><td>3</td><td>3</td><td>0</td><td>0</td><td>HR</td></tr><tr><td colspan="6">CLOCKS</td></tr><tr><td>CLKHF</td><td>2(1)</td><td>2(1)</td><td>0</td><td>2</td><td>HP</td></tr><tr><td>CLKH_OUT</td><td>2(1)</td><td>2(1)</td><td>2</td><td>0</td><td></td></tr><tr><td>CLKRF</td><td>2(1)</td><td>2(1)</td><td>0</td><td>2</td><td>HP</td></tr><tr><td>CLKR_OUT</td><td>2(1)</td><td>2(1)</td><td>2</td><td>2</td><td>HP</td></tr><tr><td>TOTAL</td><td>265/268</td><td>265</td><td>103</td><td>127</td><td></td></tr></tbody></table>
* **Larger density of patterns:** The chips will be organised into cores: an independent unit able to perform parallel pattern recognition tasks. There are three different "core" concepts in the device.

- **Independent-output-cores:** the unit which contains 1/4 of total patterns. For the AM09, it contains 3 × 32k for baseline (4 × 32k for aggressive). For the AM08, it contains 4k patterns.

- **Enable-core size:** the unit of pattern-match enable/disable. It contains 32k (4k) patterns for the AM09 (AM08).

- **Design-core:** the design unit with 4k patterns.

All cores have to be compatible with the new I/O scheme. AM08 will be composed of four "design-cores": The Table 13.18 shows the order of priority in term of designing efforts for AM08:

<table><caption>Table 13.18: Core features and options</caption><thead><tr><th>Feature for each Core</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><td colspan="5"><strong>Control technology</strong></td></tr><tr><td>Variable row number (typ = 64 rows: optimisation of skews vs # row number)</td><td></td><td></td><td></td><td>x</td></tr><tr><td>SRAM-like reading</td><td></td><td></td><td>x</td><td>x</td></tr><tr><td>Quorum fullcustom</td><td></td><td>x</td><td>x</td><td>x</td></tr></tbody></table>

The core 2 is the baseline and the core 1 will be placed to reduce risks if Quorum fullcustom does not work. The core 3 and 4 will be used to test the new feature of SRAM-like reading. If this feature will not consume more than 5% w.r.t. core 1 and 2 will be integrated in AM09. The core 4 will implement the variable row number architecture. In AM07 and AM06 we used block of 64 rows. However, this is not clear that is the optimal solution. For this reason, different blocks with several rows number will be placed to find the optimal solution in terms of power, timing and area. As HTT impose strict requirements in terms of pattern density and accurate study on the area budget has been performed. The reduced-cost run have an area limit of 154 µm, an estimation of area for the AM09 has been given. Fig. 13.19 shows the pie chart. Estimation has been obtained with core 2.

An open point of AM07 design was that the Quorum circuits (previously called "majority") occupy more silicon area than memory cells because they are based on standard cells. We estimate that by developing a full custom design, a factor × 5 will be saved in area devoted to Quorum logic.

Unlike previous versions, AM08 and AM09 allow the possibility to fast read the bit content of CAM. The approach mimics the SRAM architecture and it is a nice feature to fast debug the system.

* **Low energy usage per operation:** At the same time, we see in previous requirement that a lower energy usage per operation with respect to AM06 have to be obtained. For this reason, an extrapolation study for AM09 power consumption has been given.
<figure><img src="image_231.png" /><figcaption>Figure 13.19: Area Budget for the AM09</figcaption></figure>

The AM ASICs are low power devices that use very little energy per operation. The energy per operation is in the scale of few femto-Joule. For comparing a 16 bit word with one pattern the AM09 will use 16 fj.

More AM devices can be placed with high density on a relatively small printed circuit area, thanks to their low power. The unit cost at production of an ASIC is relatively low, with respect to medium or large FPGAs. In this context, the power consumption becomes a key parameter because it determines the maximum number of AM09, and thus the maximum AM processing capability, for a given printed circuit area.

The AM08 and AM09 will operate from a single core power supply of 1.0 V. The IO voltage will be 1.8 V with all data inputs and outputs fully compatible with LVDS18.

The power distribution must keep voltage at transistors within the simulated corners (Table 13.19). Half of the allowed ripple from the simulation corners is allocated for power distribution on the PRM card, the other half for the distribution in the package and die.

<table><caption>Table 13.19: Voltage settings for AM ASIC.</caption><thead><tr><th></th><th>sim. corners</th><th>nom. V</th><th>max ripple<sup>a</sup></th></tr></thead><tbody><tr><td>Digital core power (VDD)</td><td>0.9 V – 1.1 V</td><td>1.0 V</td><td>±50 mV</td></tr><tr><td>Digital I/O power (VDDPST)</td><td>1.62 V – 1.98 V</td><td>1.8 V</td><td>±90 mV</td></tr><tr><td>Analog core power (VDDA)</td><td>0.9 V – 1.1 V</td><td>1.0 V</td><td>±50 mV</td></tr><tr><td>Analog I/O power (VDDAPST)</td><td>1.62 V – 1.98 V</td><td>1.8 V</td><td>±90 mV</td></tr></tbody></table>

The AM09 power consumption consists of a baseline component of 1 W per ASIC (the sum of I/O power and baseline internal power), plus a data-driven contribution equal to rate of bits compared times 1 fj/comparison/bit. The value of 1 fj/comparison/bit assumes an average bit flip rate of 50%. The last term is proportional to the average rate at which data words are presented at the AM09 inputs. For example comparing

<footnote>a The max ripple at balls is the required voltage stability on the PCB.</footnote>
16 bits words with an average 50% bit-flip for data sent at an average of 50 MHz on all 8 buses requires a power of 2.5 W. This means that the data-driven power consumption is 0.05 W/MHz times the average word input rate. The formula for total AM09 power is:

$$P(\text{AM09}) = 1W + \langle \text{inputrate} \rangle * 0.05W/MHz \quad (13.3)$$

Since these chips contribute to the major part of HTT power consumption, an accurate dimensioning of power system can be achieved with chip information.

To ensure the bandwidth and power consumption requirements, the chip will be packaged in a Ball Grid Array (BGA). The AM08 will be packaged in a 17×17 BGA package (289 pins), and the AM09 will be packaged in a 23×23 BGA package (529 pins).

* **a reduction of current peaks:** In order to make the power usage more uniform in time, DCO will be used to implement a digital Phase-Locked Loop (PLL) able to spread in time the commutations of internal signals through a whole clock period. In order to implement this, different block of Associative Memory logic will be driven by clocks with different phase. The change between clock domains will happen between two near by registers with a very short propagation time. The DCO is a fully-digital oscillator module capable of producing clock signals with frequencies ranging from 0.8 GHz to 4 GHz in most typical cases. To be functional, DCO requires a current reference in order to obtain an output current with a low sensitivity against Power Supply and Temperature. The nominal output current is 1 µA with a nominal variation close to ±50 nA in a temperature range from 0°C to 120°C.

* **A controlled junction temperature:** A temperature sensor will be designed. It will be based on the measurements of the base-emitter voltage in the diode-connected bipolar from the current reference in a range from 0°C to 130 °C with a resolution of 2 °C. It will require a calibration procedure along the process variation. The analog to digital conversion is based on the time-to-digital technique. It is based on a current reference respectively based on a current mode architecture, in order to obtain an output current with a low sensitivity against power supply and temperature. The nominal output current is 1 µA with a nominal variation close to ±50 nA in a temperature range from 0°C to 120°C.

### 13.6.5 Track Fitter Mezzanine (TFM)

The Track Fitter Mezzanine, which sits on the SSTP main board, finds tracks on all 13 ITk logical layers and calculates the 5 track helix parameters and the $\chi^2$ of the fit. The functional diagram of the card is shown in Fig. 13.20

There will be 192 TFMs in the system, each containing two mid-range FPGAs. The inputs to the board, 11 serial links running at 10 Gb/s, come through the TFM-SSTP mezzanine connector. One link carries the 8-layer tracks found by the 6 PRMs that feed one TFM.
<figure><img src="image_232.png" /><figcaption>Figure 13.20: *The functional diagram of the TFM.*</figcaption></figure>

The track data consist of the clusters on the track, the pattern recognition road number, the sector number used to look up fitting constants, and the cluster map which indicates which of the 8 layers had a cluster. There are 10 serial links, 2 each carrying the clusters from one of the detector layers not used in the PRM.

The first step in TFM processing is the Extrapolator, which finds the clusters on the 5 other detector layers that are close to a PRM 8-layer track. The incoming clusters enter the DO. The 5 layers are processed in parallel, with the clusters on each sorted so that those in the same SSID are forwarded sequentially. The DO uses three sets of on-chip memory for each detector layer: the HLM, the HLP, and the HCM. The HLM sequentially stores each cluster received. The HLP stores the HLM address of the first cluster stored for each SSID. The HCM stores the number of clusters in each SSID.

For each 8-layer track, the Extrapolator calculates the most likely cluster position in each of the other 5 layers. A matrix calculation uses the 9 track cluster coordinates and a set of 10 constants per layer extracted from external memory. The most likely cluster positions are converted to SSID numbers, a check is made that there are clusters in at least 4 of the 5 layers, and if so the clusters are extracted.
The track candidates from the Extrapolator are sent to the Track Fitter. A candidate consists of an 8-layer track with its clusters and all of the nearby clusters on each of the other 5 layers. All of the combinations of one cluster per layer on the 5 layers are fit. If one of the layers does not have a cluster, then the combinations of one cluster on each of the other 4 layers are used. For each combination, the $\chi^2$ and the 5 helix parameters are calculated in linear fits using the cluster positions and a set of constants extracted from external memory. A $\chi^2$ cut is applied to reject most fake tracks.

Three types of fitters will be implemented. If one of the 5 non-PRM detector layers has no cluster, then the Majority track fitter is used. The same set of constants as is used when there are no missing hits can be used here to calculate the $\chi^2$ and helix parameters provided there is an estimate of where the cluster in the empty layer would have been. The missing cluster coordinate is estimated using a linear function of the other cluster coordinates. (This equation comes from minimising the $\chi^2$ over the position of the cluster in the empty layer.) If there are clusters in all 5 layers, the Nominal fitter is used. If the $\chi^2$ for a nominal fit fails, it is possible that one of the layers contains a random cluster. The Recovery fitter sequentially drops the cluster on one of the 5 layers and refits the remaining clusters as a Majority fit. If at least one of the recovery fits passes the $\chi^2$ cut, the track with the lowest $\chi^2$ is passed on.

The good tracks coming out of the track fitters are sent back to the SSTP. A track packet consists of the cluster on each layer, the $\chi^2$, the 5 helix parameters, the track road, the sector number, and the cluster map indicating which layers had real clusters.

The tentative choice is two moderate priced Xilinx FPGAs per TFM (XCKU085) along with high speed DDR3 memory chips (Micron MT44K16M36). This is driven by the processing requirements for the board. Simulation indicates that the PRMs will send 12 8-layer tracks per event to a TFM for a minimum $p_T$ of 4 GeV. Scaling with curvature gives an expected 48 tracks per event per TFM for $p_T$ above 1 GeV, or 24 tracks per event per TFM FPGA. With the 100 kHz event rate in gHTT, that corresponds to 2.4 Mtracks/s, or 1 track per 400 ns. There will be 1 or 2 connections (sets of extrapolation constants) per track, with each set containing 100 constants. The chosen FPGA and memory allow for this rate of memory data transfer.

The expected occupancies in the 5 layers being added to the 8-layer track indicate that there will be on average 5.4 combinations of hits in those layers to be fit with the hits from a PRM track. This assumes an extrapolation window of +/- 3$\sigma$ in the 8-layer track angular resolution. The result is that a fit must be done on average in 75 ns. To obtain the $\chi^2$ and helix parameters, 18 calculations must be done, each consisting of 19 multiplies and adds. Each of those calculations would be done in a different set of DSPs and as a result they can be completed within the allowed time. One set of 342 fitting constants has to be loaded for each connection. With the number of memory chips that an FPGA can support, the constants can be loaded within the available time. An extrapolation of the logic resources used in the FTK Second-Stage Board, which carries out similar calculations, shows that the
<table><caption>Table 13.20: *TFM FPGA resource usage compared with a Xilinx KU085.*</caption><thead><tr><td></td><td>Needed per event</td><td>Capability per event</td></tr></thead><tbody><tr><td>High-speed links/FPGA</td><td>12</td><td>44</td></tr><tr><td>Logic cells/FPGA</td><td>0.4M</td><td>0.8M</td></tr><tr><td>DSPs/FPGA</td><td>2500</td><td>4100</td></tr><tr><td>I/O lines/FPGA</td><td>260</td><td>676</td></tr></tbody></table>

chosen *FPGAs* have the needed logic capacity. An example of resources needed and the capability available in the example *FPGA* is given in Table 13.20.

### 13.6.6 The HTT Interface (HTTIF) Infrastructure

The *HTT* is connected to a commodity network through a *HTT* interface (*HTTIF*). The *HTTIF* will translate the network-specific data into a lower-level protocol to distribute the hit data to individual *AMTP* cards. The system will use QSFP links giving a 10 Gb/s connection between the *HTTIF* and the *AMTP* card. The hardware used for the *HTTIF* will be based on *FELIX* with dedicated *HTTIF* firmware. A single *HTTIF* server is connected to a group of *AMTP* and *SSTP* cards. The connection to an *ATCA* card is through a rear transmission module (*RTM*). The *RTM* is passive with mostly point-to-point connections. Utilisation of *RTMs* will allow us to reduce the mechanical stress on the fibre-optical links and the number of insertions. An additional advantage is housing the fibres in the rear of the rack where room is normally available for this purpose.

The bandwidth may depend on the pseudo rapidity of the trigger tower and on the needed data duplication. Data duplication is needed because of track bending in the magnetic field and the size of the luminous region (beam spot). To reduce the data duplication in the *HTTIF* the *ATCA* backplane is used to share strip and pixel clusters between *AMTP* cards residing in the same shelf. The second-stage track processor boards *SSTP* will receive track data from *AMTPs* and hit data directly from *HTTIF*. All the reconstructed tracks are sent back from the track processor boards to the *HTTIF*.

The *HTTIF* server will process data requests received from the *EFPU* over the network. Each request will be received as a single network message in order to have all relevant *ITk* data arrive at the same time. Each *HTTIF* server is equipped with two *FELIX* I/O cards. For each tracking request, data with a granularity of one detector element will be routed to one or both *FELIX* cards and then to the *ATCA* cards. The *HTTIF* will send out data to the cards of a given *HTT* unit with a maximum skew of 100 microseconds.
<table><caption>Table 13.21: HTT 1st-stage dataflow summary. The gHTT output bandwidth and event size refer to the HTT internal data flow between AMTP cards and SSTP cards.</caption><thead><tr><th></th><th>rHTT/event</th><th>gHTT/event</th><th>HTT rate</th><th>available</th></tr></thead><tbody><tr><td># Cluster /PRM (layer average)</td><td>200</td><td>260</td><td>46 MHz</td><td>60 MHz</td></tr><tr><td># Roads/PRM</td><td>170</td><td>270</td><td>45 MHz</td><td>400 MHz</td></tr><tr><td># Constants read/PRM</td><td>90</td><td>140</td><td>23 MHz</td><td>30 MHz</td></tr><tr><td>Fits/PRM</td><td>1500</td><td>2250</td><td>400 MHz</td><td>1 GHz</td></tr><tr><td>Tracks after χ²/PRM</td><td>80</td><td>280</td><td>36 MHz</td><td></td></tr><tr><td>Tracks after HitWarrior/AMTP</td><td>10</td><td>35</td><td>4.5 MHz</td><td></td></tr><tr><td>rHTT output bandwidth /AMTP</td><td>640 Mb/s</td><td></td><td></td><td></td></tr><tr><td>&lt;Tracks after HitWarrior&gt;/AMTP</td><td>7</td><td>20</td><td>2.6 MHz</td><td></td></tr><tr><td>Total output bandwidth</td><td>250 Gb/s</td><td>750 Gb/s</td><td>1 Tb/s</td><td></td></tr><tr><td>Average event size</td><td>30kB</td><td>900kB</td><td></td><td></td></tr></tbody></table>

### 13.6.7 Dataflow summary

The internal and output dataflow for 1st-stage processing is summarised in Table 13.21. The required rates are based on studies in Section 13.5 and the HTT rate is the sum of rHTT and gHTT rates. For rHTT the values are from the jets samples with $p_T > 2$ GeV. For gHTT the values are the average of the jets and minimum bias samples with $p_T > 1$ GeV. For both cases the worst case $\eta$ region is considered and the numbers are scaled up by a factor of 1.1, because each PRM covers a size ~ 1.1 times larger than the region size used for the performance studies as indicated in Table 13.10.

The output bandwidth numbers assume 160 bits for track parameters and track identifiers, such as road ID and sector ID, plus 480 bits for clusters on tracks for a total of 640 bits/track. The lines “Tracks after $\chi^2$” and “Tracks after HitWarrior” refer to the worst case $\eta$ regions. The line “<Tracks after HitWarrior>” refers to the average number of tracks over all $\eta$ regions. The output bandwidth and event size are based on this average.

Table 13.21 reports an event size of 30 kB for rHTT and an average event size of 250 kB for gHTT. The gHTT average event size naively accounts for the fact that only 100 kHz of the 400 kHz EF pre-selected events will undergo global tracking. Table 13.22 summarizes the estimated dataflow for second stage fitting.

Figure 13.21 summarises the payload data bandwidth for 1/8 of HTT.

### 13.6.8 Size and power consumption of the HTT system

Overall, the HTT system consists of 576 AMTPs, 1152 PRMs each with 12 AM ASICs (13824 total), 96 SSTPs, 192 TFMs, and 24 HTTIFs. The AMTPs are located in 48 ATCA shelves, and the SSTPs are in 8 ATCA shelves. Each group of 6 AMTP shelves will preferentially
<table><caption>Table 13.22: HTT 2^nd^-stage dataflow summary</caption><thead><tr><th></th><th>Needed/event</th><th>Capability/event</th></tr></thead><tbody><tr><td># of clusters/TFM (max layer)</td><td>580</td><td>5000</td></tr><tr><td># of clusters/TFM (average)</td><td>380</td><td>5000</td></tr><tr><td>Cluster rate/TFM (max layer)</td><td>58 MHz</td><td>500 MHz</td></tr><tr><td>Cluster rate/TFM (average)</td><td>38 MHz</td><td>500 MHz</td></tr><tr><td># of 1<sup>st</sup>-stage tracks/TFM</td><td>60</td><td>120</td></tr><tr><td># of constant sets read/TFM</td><td></td><td></td></tr><tr><td>Extrapolator</td><td>240</td><td>500</td></tr><tr><td>Fitter</td><td>125</td><td>230</td></tr><tr><td>Fits/TFM</td><td>325</td><td>600</td></tr><tr><td>&lt;Tracks after HitWarrior&gt;/SSTP</td><td>20</td><td></td></tr><tr><td>Total output bandwidth</td><td>150 Gb/s</td><td></td></tr><tr><td>Average event size</td><td>200kB</td><td></td></tr></tbody></table>

<table><caption>Table 13.23: Size of the HTT system</caption><thead><tr><th></th><th>HTT</th></tr></thead><tbody><tr><td>Number of <a href="#">HTTIF</a></td><td>24</td></tr><tr><td>Number of <a href="#">ATCA</a> shelves for <a href="#">AMTPs</a></td><td>48</td></tr><tr><td>Number of <a href="#">AM/SSTP</a> boards per shelf</td><td>12</td></tr><tr><td>Total number of <a href="#">AMTPs</a></td><td>576</td></tr><tr><td>Number of <a href="#">PRMs</a> per <a href="#">AMTP</a></td><td>2</td></tr><tr><td>Total number of <a href="#">PRMs</a></td><td>1152</td></tr><tr><td>Number of <a href="#">AM ASICs</a> per <a href="#">PRM</a></td><td>12</td></tr><tr><td>Total number of <a href="#">AM ASICs</a></td><td>13824</td></tr><tr><td>Number of <a href="#">ATCA</a> shelves for <a href="#">SSTPs</a></td><td>8</td></tr><tr><td>Total number of <a href="#">SSTPs</a></td><td>96</td></tr><tr><td>Number of <a href="#">TFMs</a> per <a href="#">SSTP</a></td><td>2</td></tr><tr><td>Total number of <a href="#">TFMs</a></td><td>192</td></tr></tbody></table>

be located near a shelf of SSTPs and the corresponding 6 HTTIFs to ease fibre routing. The number of HTTIF is estimated assuming 115 Gb/s payload bandwidth for each HTT unit and 250 Gb/s for each HTTIF, see Table 13.12.

The system size is summarised in Table 13.23. The rHTT and gHTT functions are performed by a single overall HTT system that will respond to both type of requests in a transparent way, i.e. processing regional or global data.

The system physical size is determined by processing power density per module. The modules are expected to make a balanced use of Printed Circuit Board (PCB) area and power dissipation to optimise processing density per module. An initial power budget is shown in Table 13.24. The power allocation for FPGA, DC/DC converters and other components is estimated from similar cards. The power for the AM ASICs is directly proportional to the
<figure><img src="image_233.png" /><figcaption>Figure 13.21: A summary of data flow in 1/8 of HTT. The blue and pink boxes are ATCA crates with AMTP or SSTP boards. The numbers are for 50% link utilisation</figcaption></figure>

amount of data sent for processing, as taken from simulation, and to the average number of bit flips among consecutive data words. The power budget is calculated with a AM ASIC energy per operation of 1 fj per comparison per bit plus a baseline of 1 W per ASIC at an average cluster processing rate of 60 MHz, see Equation 13.1.
<table><caption>Table 13.24: Summary of the estimated power usage</caption><tr><td>AMTP main card (including DC/DCs)</td><td>100 W</td></tr><tr><td>PRM FPGA</td><td>30 W</td></tr><tr><td>PRM 12 AM ASIC</td><td>50 W</td></tr><tr><td>PRM others (RAMs, IO fanout, DC/DC)</td><td>25 W</td></tr><tr><td>Total/AMTP</td><td>310 W</td></tr><tr><td>SSTP main card (including DC/DCs)</td><td>100 W</td></tr><tr><td>TFM 2 x FPGA</td><td>75 W</td></tr><tr><td>TFM others (RAMs, IO fanout, DC/DC)</td><td>25 W</td></tr><tr><td>Total/SSTP</td><td>300 W</td></tr></table>

<table><caption>Table 13.25: HTT project high-level milestones. Milestones are characterised as software (SW), firmware (FW), or hardware (HW).</caption><thead><tr><td>Date</td><td>Type</td><td>Milestone</td></tr></thead><tbody><tr><td>Q2/18</td><td>FW</td><td>Update of FPGA resource usage tables from TDR for SSR</td></tr><tr><td>Q2/18</td><td>HW</td><td>System Specification Review (SSR)</td></tr><tr><td>Q4/18</td><td>HW</td><td>AM08 ASIC submission</td></tr><tr><td>Q2/19</td><td>SW</td><td>Baseline processing configuration defined (including post-TDR optimisation)</td></tr><tr><td>Q2/19</td><td>HW</td><td>Preliminary Design Review</td></tr><tr><td>Q3/20</td><td>FW</td><td>Firmware that explores one by one all critical functions of main FPGAs</td></tr><tr><td>Q4/19</td><td>HW</td><td>AM09pre ASIC submission</td></tr><tr><td>Q4/20</td><td>HW</td><td>AM09 ASIC submission</td></tr><tr><td>Q3/21</td><td>SW</td><td>Offline SW providing online SW with configuration data loaded on the slice</td></tr><tr><td>Q4/21</td><td>FW</td><td>Slice processing demonstrated on prototypes</td></tr><tr><td>Q1/22</td><td>HW</td><td>Final Design Review</td></tr><tr><td>Q4/22</td><td>SW</td><td>Large scale Monte Carlo production demonstrated</td></tr><tr><td>Q1/23</td><td>FW</td><td>HTT processing and EF-HTT integration demonstrated with 2 HTT units</td></tr><tr><td>Q2/23</td><td>HW</td><td>Production Readiness Review</td></tr><tr><td>Q2/25</td><td>HW</td><td>Hardware installation</td></tr></tbody></table>

## 13.7 Project Milestones

A detailed schedule for the HTT construction project is given in a separate Work Breakdown Structure document. A summary of the most important milestones is presented in Table 13.25.

## References

[13.1] ATLAS Collaboration, *Fast Tracer (FTK) Technical Design Report*, Tech. Rep. CERN-LHCC-2013-007. ATLAS-TDR-021, CERN, Geneva, Jun, 2013. https://cds.cern.ch/record/1552953.
[13.2] S. Belforte et al., *The CDF trigger silicon vertex tracker (SVT)*, IEEE Trans. Nucl.Sci. (1995) 860.

[13.3] J. Adelman et al., *The Silicon Vertex Trigger upgrade at CDF*, NIM (2007) 361.

[13.4] CDF Collaboration, *The CDF detector: an overview*, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment **271** (1988) no. 3, 387 – 403. http://www.sciencedirect.com/science/article/pii/0168900288902987.

[13.5] Okumura Y., Olsen J., Liu T., Yin H., *Prototype performance studies of a Full Mesh ATCA-based General Purpose Data Processing Board*, Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC), 2013 IEEE (2013).

[13.6] C. Gentsos et al., *Track finding mezzanine for Level-1 triggering in HL-LHC experiments*, in 2017 6th International Conference on Modern Circuits and Systems Technologies (MOCAST), pp. 1–4. May, 2017.

[13.7] J. Gradin, M. Martensson, and R. Brenner, *Comparison of two hardware-based hit filtering methods for trackers in high-pileup environments*, submitted to JINST. https://arxiv.org/abs/1709.01034.

[13.8] A. Annovi, *A Low-Power and High-Density Associative Memory in 28 nm CMOS Technology*, in 6th International Conference on Modern Circuits and Systems Technologies (MOCAST). 2017.

[13.9] A. Stabile, *Italian patent, 102016000077445 (UA2016A005430) del 22.07.2016 (Ns. rif.: P1637IT00),*
<figure><img src="image_234.png" /></figure>
# 14 Evolution Scenario

The single hardware-level trigger discussed throughout this document is the selected configuration of the trigger architecture design for the Phase-II TDAQ System and will be the nominal design for the start of Run 4. However, two main risks have been identified that may impact the eventual performance of the system at the ultimate HL-LHC luminosity (see Table 1.2). The first risk is the uncertainty in the projected trigger rates for hadronic objects at $\langle\mu\rangle = 200$. The second risk is the uncertainty on the occupancy in the inner pixel detector layers at the ultimate HL-LHC conditions; should this occupancy greatly exceed expectations, the increase in event size would exceed the bandwidth specification for the pixel detector readout chips. The architecture scheme described in this chapter mitigates both risks by adding a second hardware-level trigger (Level-1) which uses regional tracks built from hits in the ITk strip detector and the outer layer of the pixel detector. This additional trigger stage would provide for a rudimentary primary vertex selection for multijet triggers, allowing for additional hadronic background rejection and a lower overall readout rate for the inner detector pixel layers. Additional details regarding the motivation and criteria for such an evolution are presented in Section 14.1, followed by the requirements for the evolved system in Section 14.2 and a description of the resulting "evolved" architecture design in Section 14.3. The implications for the individual TDAQ sub-systems are described in Section 14.4, including the design considerations incorporated into the baseline system design that would allow for evolution and an estimate of the latency of the evolved system. Finally, the opportunities for a variety of physics signatures provided by the inclusion of hardware-based regional tracking in the Level-1 trigger are presented in Section 14.5. A timeline for the eventual commissioning of the evolutionary system is outside the scope of this document.

## 14.1 Criteria for Evolution

The two main criteria for the evolution to the split-level hardware trigger configuration are the hadronic trigger rates and the inner pixel detector layer occupancies. If either or both are higher than expected, the baseline TDAQ architecture would not survive the ultimate HL-LHC running conditions. In this case, the hardware of the baseline trigger system is designed so it can be used in the evolved TDAQ system with adjustments to the firmware as described later in this chapter.
<figure><img src="image_235.png" /><figcaption>Figure 14.1: Quadratic sum of simulated electronics and pileup noise per calorimeter cell for each calorimeter layer as function of pseudorapidity. Each subfigure corresponds to a different level of pileup noise; the total noise at each $\mu$ is used in the topological clustering algorithm to reconstruct hadronic events.</figcaption></figure>

### 14.1.1 Uncertainty in Hadronic Trigger Rate Estimates

In the nominal HL-LHC trigger menu for the single-level trigger architecture (see Table 6.4), the trigger rates are based on simulated data with $\mu = 200$ interactions per crossing. Lepton trigger rate projections have proven reliable. However, projections for hadronic rates are extremely sensitive to the contribution from stochastic jets (accidental overlaps of uncorrelated particles).

Hadronic activity is reconstructed using the topological clustering algorithm described in Section 6.2. The noise setting in the algorithm depends on the expected number of interactions per crossing and varies as a function of calorimeter layer and $|\eta|$, as shown in Fig. 14.1. These settings are then used in simulation to form topological clusters that are the inputs to the anti-$k_t$ jet-finding algorithm in order to determine the hadronic trigger rates as a function of $\mu$. An illustration of this is the trigger rate per unit $\mu$ for a hadronic trigger requiring four EM-scale jets above 30 GeV, as shown in Fig. 14.2. Several features can be extracted from this figure. First, adjusting the effective number of interactions per crossing by $\pm 10$ around each pileup setting results in an increasing variation in the trigger rate. Second, the central values of the rates at $\mu = 60, \mu = 140$, and $\mu = 200$ are not linearly increasing. Third, there is an overall uncertainty on these central values for $\mu = 140$ and $\mu = 200$ since we cannot yet measure the hadronic trigger rate in data at these high values of pileup. Thus, a reasonably small underestimate of the stochastic jet contribution can result in a large increase in the four-jet trigger rate. Based on the HL-LHC plan, the pileup conditions are expected to reach $\mu = 140$ at the start of Run 4. Running at the so-called nominal HL-LHC conditions will already give us important feedback as to the accuracy of the simulation in the ultimate HL-LHC conditions. If the hadronic trigger rates at $\mu = 140$ are higher than predicted by the simulation, a move to the evolved architecture would be needed to reject additional background in the Level-1 hardware trigger.
14.1.2 Uncertainty in ITk Pixel Detector Occupancies

The ITk pixel detector is made up of five layers in the barrel (layers 0 – 4, where layer 0 is the innermost pixel layer) and consists of five rings for each endcap [14.1].

The occupancy (number of hits per chip) has been studied in simulated *t\bar{t}* events at $\mu = 200$, as described in Ref. [14.1] and shown for the barrel region in Fig. 14.3. The highest occupancy is expected in Layer 0 of the barrel of the ITk pixel detector, where the average number of hits per chip is expected to be 137 in the nominal inclined geometry. The highest average occupancy with respect to z is 247 hits per chip, which corresponds to a maximum inner pixel channel occupancy of 0.16% and a maximum hit density of 0.64 mm⁻².

The occupancy drives the readout data rate for the ITk pixel detector. The data rate as a function of pixel layer (for both inclined barrel and endcap) is shown in Table 14.1. In the single-level trigger scheme, the innermost pixel layer in the barrel has the highest expected average data rate per link at $\mu = 200$: 3.97 Gb/s. The readout is designed for an average data rate of 70% of the maximum bandwidth, or 3.6 Gb/s, to allow for event-by-event fluctuations. The maximum data rate is 5.12 Gb/s per FE chip. In the evolved scheme, the front-end chips for the inner barrel and endcap layers incorporate logic to remove hits from memory that do not belong to an event accepted by the Level-0 Trigger (the so-called "fast clear"), bringing the rates well below the design specification. Aggregation causes rates near the soft limit in the outer layers as well in the evolved scheme.

However, if the innermost pixel occupancy is dramatically higher than predicted by simulation at the start of Run 4, the event size would increase and the data rate could exceed the

<figure><img src="image_236.png" /><figcaption>Figure 14.2: The four-jet trigger rate per unit $\mu$ as a function of $\mu$. Three pileup values are used for the noise setting in the topological clustering algorithm: $\mu = 60$ (red), $\mu = 140$ (yellow), and $\mu = 200$ (blue).</figcaption></figure>
<figure><img src="image_237.png" /><figcaption>(a) Occupancy distribution for the ITk barrel pixel de-
tector as a function of layer.</figcaption></figure>

<figure><img src="image_238.png" /><figcaption>(b) Average occupancy vs. z for the ITk barrel pixel detector as a function of layer.</figcaption></figure>

Figure 14.3: Distribution of chip occupancies over all events (minimum single hit) and layers for the ITk pixel barrel in the inclined dual layout.

design value of 5.12 Gb/s at the ultimate HL-LHC luminosity. Furthermore, the develop-
ment of the 5.12 Gb/s serial output of the pixel detector aggregator chip carries a moderate
technical risk [14.1]. In either scenario, a lower readout rate of 600 – 800 kHz (compared
to the 1 MHz readout rate for the single-level TDAQ system) or other measures to reduce
bandwidth consumption might be needed for the pixel detector. The lower readout rate
can be accomplished in the evolved architecture through the additional rejection provided
by the Level-1 hardware trigger. In this scheme, the longer latency can be accommodated
by the pixel readout system.

**14.2 Requirements for the Evolved System**

The main requirements for the evolved system are as follows:

* Calorimeter and muon data shall be sent to the Level-0 hardware trigger every bunch crossing, as in the baseline system.

* To minimise the amount of effort and avoid problems in the commissioning phase, the Level-0 latency shall be kept to the value of the baseline Level-0-only hardware trigger system (10 µs).

* Regions of interest for tracking shall be provided by the Global Trigger sub-system.

* Upon a L0A, regional ITk data (no more than 10% of the detector data volume) shall be provided to the Level-1 hardware-based trigger system, at an event rate of at least 2 MHz. For the ITk strip detector, the regional data will be produced by FE electronics then sent out via FELIX, while for the ITk pixel detector outer layers (Layers 2-4) and the forward rings, the regional data will be extracted from the full Level-0 data stream
<table><caption>Table 14.1: Expected data rates per link for Pixel (inclined) barrel and endcap layers for the single-level trigger (1 MHz) and evolved (4 MHz) L0A rates. The rates are based on simulation and are averages per event; for each layer the value for the module with the highest data rate is indicated. The maximum data rate per front-end chip is 5.12 Gb/s [14.1].</caption><thead><tr><th>Layer</th><th>Baseline Data Rate<br>Per Link (Gb/s)</th><th>Evolved Data Rate<br>Per Link (Gb/s)</th></tr></thead><tbody><tr><td>Barrel Layer 0</td><td>3.97</td><td>-</td></tr><tr><td>Barrel Layer 1</td><td>1.78</td><td>-</td></tr><tr><td>Barrel Layer 2</td><td>2.08</td><td>2.08</td></tr><tr><td>Barrel Layer 3</td><td>1.28</td><td>2.56</td></tr><tr><td>Barrel Layer 4</td><td>0.88</td><td>3.52</td></tr><tr><td>Endcap Layer 0</td><td>2.15</td><td>-</td></tr><tr><td>Endcap Layer 1</td><td>2.14</td><td>-</td></tr><tr><td>Endcap Layer 2</td><td>2.60</td><td>2.60</td></tr><tr><td>Endcap Layer 3</td><td>1.56</td><td>3.12</td></tr><tr><td>Endcap Layer 4</td><td>1.08</td><td>4.32</td></tr></tbody></table>

in FELIX, then sent out. Other ITk pixel layers and rings will not be used in the Level-1 decision.

* The HTT hardware shall be designed to minimise the effort required to become part of the real-time trigger path.

* Regional charged tracks with $p_T > 4$ GeV shall be reconstructed in the Level-1 Trigger system within the Level-1 latency budget.

* A rejection factor of 2.5 (7) shall be provided by the Level-1 Trigger to achieve a L1A rate of 800 kHz (600 kHz).

* Upon a L1A, all of the remaining ITk data shall be read out at a rate of 800 kHz assuming a L0A rate of 2 MHz or 600 kHz assuming a L0A rate of 4 MHz.

* The two inner pixel layers (Layers 0-1) shall be read out to the DAQ system at the L1A rate.

* The total latency of the Level-1 hardware-based trigger system (including the Level-0 latency) shall be less than 30 µs (35 µs) for the 4 MHz (2 MHz) L0A scheme.

The required data rates for the evolved system are summarised in Table 14.2. The parameters for the 4 MHz scheme are driven by the ITk strip detector readout constraints, while the parameters for the 2 MHz scheme are driven by NSW detector readout constraints. These and other considerations with respect to the readout of the ATLAS detectors are described in the following sub-sections.
<table><caption>Table 14.2: The relevant data rates and Level-1 latencies for the 4 MHz and 2 MHz L0A rate schemes. The regional data fraction is assumed to be 10% of the L0A rate, and the total allowed data rate is 1 MHz. The data rate for the full readout of the ITk detectors is set by the difference between these two quantities.</caption><thead><tr><th></th><th>4 MHz L0A Scheme</th><th>2 MHz L0A Scheme</th></tr></thead><tbody><tr><td>Maximum Level-1 Latency</td><td>30 µs</td><td>35 µs</td></tr><tr><td>Fraction of Regional Data</td><td>10%</td><td>10%</td></tr><tr><td>Effective Regional Readout Rate</td><td>400 kHz</td><td>200 kHz</td></tr><tr><td>Full ITk Detector Readout Rate</td><td>600 kHz</td><td>800 kHz</td></tr><tr><td>Total Data Rate</td><td>1 MHz</td><td>1 MHz</td></tr></tbody></table>

### 14.2.1 ITk Pixel Detector

The design of the FE chip for the inner pixel layers (0-1) accommodates the reception of both Level-0 and Level-1 signals, and incorporates logic to remove hits from memory that do not belong to an event accepted by the Level-0 Trigger (described as the "Fast Clear" logic in Ref. [14.1]). This design dictates that if the TDAQ L0A latency is within 10 µs and the L1A signal arrives less than 35 µs later, the hit losses will be < 1%. The total memory in the pixel FE corresponds to a maximum latency of 12.5 µs.

The outer pixel layers (2-4) shall be read out at the maximum Level-0 rate of 4 MHz, with the installation of additional cables and fibres<fnref n="1" />. In the case of the outer pixel layers, the selection of data for the Level-1 hardware-based trigger system will be done off-detector, in the Readout System, based on Regional Readout Requests received from the Global Trigger via the corresponding FELIX units.

### 14.2.2 ITk Strip Detector

The ITk strip detector readout shall be able to accommodate a 4 MHz Level-0 rate without any design modifications, provided that the RoIs requested cover at most 10% of the detector. This maximum readout rate is derived from the 128-event depth of the FE buffer. At a 4 (2) MHz Level-0 rate, and 600 kHz (800 kHz) Level-1, Level-1 accept signals shall arrive at the FE chip within 20 (40) µs after the Level-0 signal. Thus the ITk strip detector FE buffer determines the maximum Level-1 latency (30 µs) in the 4 MHz scheme.

<footnote n="1">¹ The installation of these additional cables and fibres leads to an increased material budget of the order of a few percent; this additional material will not have a sizeable impact on the detector performance.</footnote>
### 14.2.3 NSW

The evolved scheme differs from the specifications used to design the NSW system; an in-depth study has been carried out to understand the implications. In the Level-0/Level-1 scenario the NSW would only be read out on L1A, effectively increasing the bandwidth headroom. A Level-1 latency of 25 µs (and potentially up to 35 µs, depending on the dead-time incurred by the NSW front-end electronics) will not lead to any data loss. The dead-time incurred by the NSW front-end electronics determines the maximum latency (35 µs) for the 2 MHz L0A scheme.

## 14.3 Overview of the Evolved System Architecture

If the criteria outlined in Section 14.1 are met, the TDAQ architecture will evolve as shown in Fig. 14.4, to a two-level Level-0/Level-1 system with at least 2 MHz and up to 4 MHz L0A rate. The data rates for these two schemes are summarised in Table 14.2.

The functional overview of the evolved TDAQ system is shown in Fig. 14.4. Unlike the single-level trigger configuration, the hardware trigger is split into a two-level hardware trigger system, where the HTT performs the primary reduction of the Level-0 rate for an affordable EF farm size. The basic Level-0 Trigger functionality is unchanged. However, a Region of Interest Engine (RoIE) is added to the Global Trigger to calculate RoIs upon a L0A. The RoIs are used to generate R3s for the ITk strip detector and the equivalent off-detector data selection for the outer layers of the ITk pixel detector. In the case of the ITk strip detector, the R3s are distributed to the front-end electronics via FELIX, initiating the transfer of data for use in L1Track. Due to bandwidth limitations on the front-end electronics, the selected data cannot represent more than about 10% of the total detector data on average. In the case of the ITk outer pixel layers, the regional readout requests are used in the Readout System to select the relevant data for L1Track. L1Track is a hardware-based regional tracking system that reconstructs tracks in these RoIs; it is composed of the same hardware and firmware as the HTT components, but is subject to an additional latency constraint of 10 µs.

The Global Trigger receives the regional charged track objects and combines these with the same calorimeter- and muon-based objects as foreseen in the single-level scheme. Additional algorithms to ensure jets in hadronic events arise from the same primary vertex may be implemented. The output is sent to the Level-1 CTP, which is responsible for the final Level-1 trigger decision and for distributing the L1A signal to the detectors to transmit their full event data. The overall latencies are summarised in Table 14.2.

The Level-0-only TDAQ design is capable of operation at 1 MHz and luminosities up to $\mathcal{L} = 7.5 \times 10^{34} \text{ cm}^{-2}\text{s}^{-1}$, but all so-called “hooks” necessary to evolve to a Level-0/Level-1 system at a later stage are implemented in the design. The implications for the design of the individual TDAQ sub-systems are described in Section 14.4.
<figure><img src="image_239.png" /><figcaption>(a) Baseline TDAQ System with a single-level hard- (b) Evolved TDAQ System with a two-level hardware
trigger.                      trigger.</figcaption></figure>

<caption>Figure 14.4: Diagram of the evolved TDAQ System in Phase-II with a two-level hardware trigger (b), compared to the single-level hardware trigger configuration (a). The additional components in the evolved system are shown in light blue: the RoIE within the Global Trigger, L1Track, and L1CTP. For the ITk strip detector, the regional data will be produced by FE electronics then sent out via FELIX, while for the ITk pixel detector outer layers (Layers 2-4) and the forward rings, the regional data will be extracted from the full Level-0 data stream in FELIX, then sent out. The remaining full detector data is sent after a L1A. Direct connections between each Level-0/Level-1 trigger component and the Readout system are suppressed for simplicity.</caption>
## 14.4 Design Implications for the TDAQ Sub-systems

### 14.4.1 Level-0 Calo Trigger

The change from the single-level hardware trigger to a two-level system is transparent for Level-0 Calorimeter trigger hardware. It might imply, however, a few small changes in the firmware depending on the readout scheme. In the single-level system, the readout will be issued upon receiving the L0A signal. In a two-level system it seems advantageous to issue the readout after the L1A signal at lower rate and potentially higher readout bandwidth. Moving from one readout scheme to the other requires minor changes in the firmware for all FEX systems as well as the Hub module and its daughter card.

### 14.4.2 Level-0 Muon Trigger

For the evolution from the single-level to the two-level hardware trigger architecture, no change is required for the Level-0 muon trigger system. The hit data for RPC, TGC, NSW, and MDT are transmitted to FELIX after a L1A signal in the two-level architecture. The RPC, TGC, and MDT hit data are streamed off the chambers to the off-detector electronics; these buffers are compatible with the Level-1 latency of the evolved architecture. The NSW hit data are stored in the buffers in the on-detector electronics before the readout; these buffers are also compatible with the Level-1 latency of the evolved architecture.

### 14.4.3 Global Trigger

The Global Trigger will evolve in the two-level architecture to incorporate additional inputs from the L1Track trigger system. This will allow the GEP modules to include tracking information when refining trigger objects for Level-1 trigger decisions, which has the potential for large rate reductions of pile-up and noise. The multiplexed L1Track data will be provided to the GEP in identical fashion as the other data inputs so that the track information can be included in extended object processing. A new RoIE will receive the pattern of active trigger inputs contributing to the L0A decision (TIC) and use this to suitably seed regional readout of the ITk strip tracker and selection of outer ITk pixel data for transmission to L1Track.

To enable this the Global Trigger will have three additional major functions in the two-level architecture over and above the Level-0 system described in Chapter 9:

* At the end of the Level-0 trigger process, after an L0A, the Global Trigger must identify the RoIs corresponding to accepted trigger hypotheses that require tracking information from the L1Track regional hardware-based tracking system.
* The Global Trigger must initiate the generation of the R3 in the ITk strip tracker to seed the transfer of data to the L1Track, transferring this information to the ITk Readout System along with the corresponding off-detector data selection for the outer ITk pixel layer that is buffered in the FELIX system.

* The Global Trigger shall store Level-0 objects until the L1A; these objects may be updated with L1Track information.

* When the L1Track data are available, the Global Trigger shall use this information to update the Level-0 trigger hypotheses to form the input to L1CTP for the Level-1 trigger decision and transfer this information to the L1CTP system.

These functions will be implemented by updating the Global Trigger as follows:

* the number of GEP nodes will remain the same, with the GEP node that processed an event at Level-0 processing it at Level-1, and the GEP firmware will be updated to include these additional steps:

- produce the compact data structure mapping the TIP onto the TOBs which caused those TIP bits to be fired every event as the TIP are being transferred to the CTP Interface, which in the single-level architecture is only produced after the L0A decision, and transfer this to the new RoIE described below

- buffer the Level-0 event information pending the receipt of tracking information

- when the Level-1 track information is available, use this to update the Level-0 trigger hypotheses and transfer the resultant Level-1 TIP to the CTP Interface

* additional MUX modules will be added to receive the tracking information from L1Track in a manner analogous to the Level-0 MUX modules

* additional MUX variant modules will be added, running the new RoIE firmware, as described in more detail below

* the CTP Interface will be updated to transmit the Level-1 TIP to L1CTP

The updated system processing and trigger decision flow including these new steps within the GEP and the new modules are illustrated in Fig. 14.5.

## Impact on the GEP

The step to produce the data structure mapping the TIP onto the TOBs which caused those TIP bits to be fired will not significantly impact on the resource usage on the GEP as this functionality is required in the single-level scheme to generate the readout data - it just moves from running after the L0A to during the CTP Interface and L0CTP processing.

The same GEP node that processed an event at Level-0 will process the event at Level-1. The buffering step will therefore consist of the GEP node streaming the event information into its local RAM after the L0A and retrieving it after a suitable interval for amalgamation with the tracking information from L1Track.
<figure><img src="image_240.png" /><figcaption>Figure 14.5: Illustration of the Global Trigger system processing and trigger decision flow in the evolution to a two-level architecture.</figcaption></figure>

In the single-level scheme the GEP node would have streamed its readout information to the Zynq chip after L0A; this step now occurs on the retrieved information after L1A.

The algorithms and hypotheses using the tracking information are not expected to be as intensive as the jet-finding algorithm and have a nominal allocation of 10% of the device, taking resource utilisation to 60%.

### The RoIE

The RoIE consists of a set of MUX nodes with point-to-point links to each FELIX module in the ITk strip and outer pixel readout. Every bunch crossing the RoIE receives the TIC trigger decision from L0CTP and identifies those TOBs requiring tracking information by comparing this with the complete information from the respective GEP node about those TOBs contributing to accepted trigger items. It then transmits the requisite information down the correct point-to-point links to generate the R3 in the ITk strip tracker and corresponding off-detector data selection in the outer ITk pixel readout to seed the transfer of data to the L1Track.

The RoIE firmware essentially consists of two lookup steps, TIC to TOBs followed by TOB to R3 seed, which are well suited to FPGA implementation.

### 14.4.4 Trigger, Timing, and Control

The TTC paths for the evolved system are shown in Fig. 14.6. In comparison to Fig. 5.3, a GBT connection between the Level-1 Trigger Processors and FELIX and a connection
between the RoIE in the Global Trigger and FELIX are added. Additionally, the CTP is
divided into a L0CTP and L1CTP, as described in Section 14.4.5.

<figure><img src="image_241.png" /><figcaption>Figure 14.6: *The evolutionary two-level system timing and control paths.*</figcaption></figure>

**14.4.5 Central Trigger System**

There is one CTP for the Level-0 trigger system (L0CTP). For the evolved two-level trigger system, it is planned to use an additional CTP dedicated to the Level-1 trigger system (L1CTP). The L1CTP would use the same hardware as the L0CTP, but with different firmware for the Level-1 functions.

Figure 14.7 shows the Central Trigger part of the evolved Level-0/Level-1 architecture. Compared to the single-level scheme, the L1CTP is added in the same ATCA shelf. The L0CTP calculates, for every L0A, the pattern of active trigger inputs contributing to the L0A decision (TIC). This is transmitted to the RoIE of the Global Trigger, where it is used in identifying regions of the detector to be used in the Level-1 rHTT. For sending the TIC pattern (along with associated information), a 12-way ribbon fibre transmitter module [14.2] is foreseen which allows to connect 12 serial optical links with line rates of 9.6 Gb/s. In addition, it has to provide, for each L0A, the pattern of active trigger items after veto (TAV) and transmit it to the L1CTP. For sending the TAV pattern to the L1CTP, another 12-way ribbon fibre transmitter module is foreseen. The inputs from L0CTP can be used for triggers that
<figure><img src="image_242.png" /><figcaption>Figure 14.7: Context diagram showing the L0CTP and L1CTP systems interfacing to the MUCTPI, the Global Trigger, the RoIE, and the FELIX systems (via the LTI modules). Only the trigger path is depicted; control, monitoring, and readout are not shown.</figcaption></figure>

do not require validation at Level-1 and to provide pre-scaled triggers without the Level-1 requirement for monitoring and efficiency measurements.

Figure 14.8 shows the L1CTP in more detail. It uses a CTPIN module to receive the TAV word from the L0CTP. A CTPCORE module with Level-1-specific firmware receives the Level-1 trigger inputs from the Global Trigger via one of its two 12-way ribbon fibre receiver modules, while the other receiver module is used to receive the Level-0 TAV word via the CTPIN. The signals from the Global Trigger are synchronous with the LHC clock and arrive with a fixed latency after the collision. The inputs from L0CTP will be delayed in the CTPIN to match the later arrival time of the Global Trigger inputs. The trigger processing in the CTPCORE includes a synchronisation and time alignment step, the application of trigger logic and pre-scaling. If necessary, veto gating and deadtime generation can be applied. The resulting L1A and associated TTC information will be fanned out to the LTI modules via the TTC outputs of the CTPCORE board. The second SFP input of the LTI is used for this purpose. The LTI merges the Level-1 TTC information with the Level-0 information and makes it available on its TTC-PON outputs.

Table 14.3 summarises the estimated latency contributions of the critical path through the central trigger system.

For the evolved Level-0/Level-1 architecture, the TIC word calculation will take an addi-
tional 200 ns, which is counted in the Level-1 latency. The link for the TIC word from the
CTP to the RoIE is estimated to 200 ns, including a 15 m long fibre. The processing step of
the L1CTP is estimated to be 0.275 ns. The uncertainty on the latency estimates is quoted as
contingency in the rightmost column of Table 14.3.
<figure><img src="image_243.png" /><figcaption>Figure 14.8: Basic architecture of the L1CTP, showing its main components, inputs, outputs, and the main information flow.</figcaption></figure>

<table><caption>Table 14.3: Summary of the expected latency of the CTP in the evolved two-level system.</caption><thead><tr><td>Item</td><td>Partial Current Best Estimate (μs)</td><td>Uncertainty/ Contingency (μs)</td></tr></thead><tbody><tr><td>Calculation of TIC in the L0CTP</td><td>0.200</td><td>0.050</td></tr><tr><td>Link TIC to RoIE (5 BC + 15 m fibre)</td><td>0.200</td><td>0.025</td></tr><tr><td>L1CTP Processing</td><td>0.275</td><td>0.100</td></tr></tbody></table>
## 14.4.6 Summary of Hardware Trigger Latency Estimates in the Evolved System

The latency constraints for the evolved system, as defined by the time elapsed between collision time and the arrival of the trigger decision at the FELIX outputs. The Level-0 latency is 10 µs for the 4 MHz rate, and is constrained by the readout of the inner pixel layer. Given a Level-0 rate of 4 MHz (2 MHz), the corresponding Level-1 rate is 600 kHz (800 kHz) and the maximum latency is 30 µs (35 µs), which is constrained by the ITk strip (NSW and ITk pixel) readout.

An early attempt in estimating of the L1A TDAQ Latency is presented in Table 14.4. The integral Maximum Possible Value of 25.5 µs is within the worst case requirement of 30 µs.

## 14.4.7 Data Acquisition

The overall architecture of the TDAQ system places several dependencies on the design of the DAQ system. In general terms these can be categorised using the throughput and propagation of trigger data. The specific impact of a change depends on the trigger rates which will need to be supported by the system. For the purposes of this assessment two scenarios have been used: a Level-0 rate of 2 MHz with a Level-1 rate of 800 kHz and a Level-0 rate of 4 MHz with a Level-1 rate of 600 kHz.

### Readout

Moving to a two-level TDAQ architecture, with a hardware track trigger (L1Track) running as part of Level-1, would significantly increase the amount of input data arriving at FELIX, as well as the number of links to be supported. The main contributor to this will be ITk, where an extra ~7000 links will be needed to readout the strip and outer Pixel layers at 2 or 4 MHz. There will also be ~3500 links from to L1Track from strips and Pixel via FELIX. The result of this will be an increase in the number FELIX systems needed, as presented in Table 14.5. The FELIX implementation will also become more complicated because of the need to service regional readout requests for ITk data and to transfer the corresponding data via low-latency point-to-point links to L1Track. The number of Data Handler systems will also have to increase to satisfy the increasing number of readout links, as will the scale of the readout network between the these components and FELIX. Each Data Handler will have to support larger local buffers to store data between a L0A and a L1A. One reduced requirement compared to the single-level system would be the data rate between the Data Handlers and the Storage Handler. This would drop from 1 MHz to either of the two lower Level-1 rates. The structure of the Readout system with the split-level hardware trigger, along with the new interfaces, is presented in Fig. 14.9.
<table><caption>Table 14.4: Phase-II Hardware Trigger Level-1 Latency for the Evolutionary scenario; values are expressed in µs.</caption><thead><tr><th>System</th><th>Item</th><th>BEV</th><th>Unc.</th><th>∫ BEV</th><th>∫ MPV</th></tr></thead><tbody><tr><td rowspan="3">CTP</td><td>L0CTP Processing</td><td></td><td></td><td rowspan="3">6.1</td><td rowspan="3">8.0</td></tr><tr><td>Calculation of TIC in the L0CTP</td><td>0.200</td><td>0.05</td></tr><tr><td>Link TIC to RoIE (5BC + 15m fibre)</td><td>0.200</td><td>0.03</td></tr><tr><td rowspan="2">RoIE</td><td>RoIE processing</td><td>0.225</td><td>0.05</td><td rowspan="2">6.8</td><td rowspan="2">8.9</td></tr><tr><td>Link to FELIX (5BC + 20m fibre)</td><td>0.225</td><td>0.03</td></tr><tr><td rowspan="3">FELIX R3</td><td>R3 processing</td><td>0.200</td><td>0.05</td><td rowspan="3">7.0</td><td rowspan="3">9.2</td></tr><tr><td>GBT TX</td><td>0.100</td><td>0.03</td></tr><tr><td>R3 to ITk Stave (110m fibre)</td><td>0.500</td><td>0.05</td></tr><tr><td rowspan="5">ITk Strips</td><td>GBT RX</td><td>0.125</td><td>0.05</td><td rowspan="5">7.6</td><td rowspan="5">10.0</td></tr><tr><td>HCC receive and decode</td><td>0.125</td><td>0.03</td></tr><tr><td>R3 Request to Last hit leaves HCC</td><td>3.000</td><td>0.50</td></tr><tr><td>GBTX</td><td>0.125</td><td>0.05</td></tr><tr><td>Stave to FELIX (110m fibre)</td><td>0.550</td><td>0.03</td></tr><tr><td rowspan="3">FELIX L1</td><td>GBT-FPGA RX</td><td>0.050</td><td>0.03</td><td rowspan="3">11.6</td><td rowspan="3">15.2</td></tr><tr><td>transit time</td><td>0.100</td><td>0.10</td></tr><tr><td>Link to L1Track (5BC + 20m fibre)</td><td>0.225</td><td>0.03</td></tr><tr><td rowspan="3">L1Track</td><td>Header unpacking</td><td>0.100</td><td>0.10</td><td rowspan="3">11.9</td><td rowspan="3">15.8</td></tr><tr><td>Processing</td><td>4.800</td><td>0.60</td></tr><tr><td>Result aggregation</td><td>0.250</td><td>0.25</td></tr><tr><td rowspan="2">Global Trigger</td><td>Processing</td><td>0.400</td><td>0.20</td><td rowspan="2">17.1</td><td rowspan="2">22.9</td></tr><tr><td>Link to L1CTP (5BC + 15m fibre)</td><td>0.225</td><td>0.03</td></tr><tr><td rowspan="5">CTP</td><td>L1CTP processing</td><td>0.275</td><td>0.10</td><td rowspan="5">18.5</td><td rowspan="5">25.2</td></tr><tr><td>link to LTI (5BC + 30m fibre)</td><td>0.275</td><td>0.03</td></tr><tr><td>LTI internal processing</td><td>0.100</td><td>0.03</td></tr><tr><td>Link to FELIX (5BC + 10m fibre)</td><td>0.175</td><td>0.05</td></tr><tr><td>Processing</td><td>0.100</td><td>0.05</td></tr><tr><td rowspan="2">FELIX</td><td>GBT-FPGA TX</td><td>0.100</td><td>0.03</td><td rowspan="2">18.7</td><td rowspan="2">25.5</td></tr><tr><td>L1A at FELIX output (L1 Latency)</td><td></td><td></td><td>18.7</td><td>25.5</td></tr></tbody></table>

<table><caption>Table 14.5: Additional Readout units needed to serve a split level system. The increased network throughput may also require an increase in the number of NIC ports, but the precise number would be a subject of further study.</caption><thead><tr><th>Component</th><th>Additional Number of Units</th></tr></thead><tbody><tr><td>FELIX I/O Cards</td><td>236</td></tr><tr><td>FELIX Servers</td><td>120</td></tr><tr><td>Data Handlers</td><td>260</td></tr></tbody></table>
<figure><img src="image_244.png" /><figcaption>Figure 14.9: Readout system architecture for Phase-II in the Evolved System.</figcaption></figure>

## Dataflow

The structure of the Dataflow system will not have to change in order to satisfy the requirements of a split-level system. L1Track and rHTT have a comparable rejection power, but the higher input rate in the evolved system means that the bandwidth requirements for transfer of data to the EF will be larger, with the need to transfer 600 or 800 kHz of full events instead of 400 kHz in the single-level scheme. The reduced input bandwidth to the system overall means that for a L1A of 600 kHz the total throughput required of the Storage Handler is reduced by 1.4 TB/s but for a L1A of 800 kHz the total throughput increases by 1.8 TB/s. The rates and data volumes to be transferred to the Event Aggregator and gHTT remain unchanged from the single-level case. A summary of the bandwidth requirements for each component of the system is presented in Table 14.6.

## 14.4.8 Event Filter

Evolution of the TDAQ architecture affects the EF in two fundamental ways: first it changes the input rate to the EF and potentially calls for a modified trigger strategy to be applied to
<table><caption>Table 14.6: The Phase-II Dataflow traffic required, assuming a split level system with Level-1 rate of 600-800 kHz. The data volumes coming in to the system will vary strongly by subdetector, with the minimum aggregate rate exceeding that for the single-level trigger scheme.</caption><thead><tr><th>Component Connection</th><th>4 MHz/600 kHz</th><th>2 MHz/800 kHz</th></tr></thead><tbody><tr><td>Detector Front-ends to FELIX</td><td>&gt;6 TB/s</td><td>&gt;6 TB/s</td></tr><tr><td>FELIX to Data Handlers</td><td>&gt;6 TB/s</td><td>&gt;6 TB/s</td></tr><tr><td>Data Handlers to Event Builder/Storage Handler</td><td>3.2 TB/s</td><td>4.8 TB/s</td></tr><tr><td>Storage Handler to Event Filter</td><td>3.2 TB/s</td><td>4.8 TB/s</td></tr><tr><td>EF to gHTT</td><td>563 GB/s</td><td>563 GB/s</td></tr><tr><td>EF to Event Aggregator/Permanent Storage</td><td>60 GB/s</td><td>60 GB/s</td></tr></tbody></table>

those events, and second the rHTT has moved to the Level-1 hardware-based trigger, with the associated event rejection now taking place before the EF.

No specific hardware changes are necessary for the EF processors, but there is likely to be a significant increase in the level of computing power required due to the higher rate remaining after regional tracking in the evolved system. The quantitative estimate of the additional computing power needed can be determined only with a specific trigger menu and strategy in mind; the purchase cost would strongly depend on the timing of the planned evolution. These issues will be investigated in detail if the system evolution is deemed necessary according to the criteria described in Section 14.1.

Two scenarios have been developed to mitigate readout limitations, resulting in input rates to the EF of 800 and 600 kHz, in both cases less than the 1 MHz of the single-level trigger architecture. However, the rates are higher than the 400 kHz expected in the single-level scheme once regional tracking (rHTT) has been used. With regional tracking having already taken place upstream in Level-1 in the evolved architecture, this will place a greater computing burden on the EF to achieve rejection through software-based regional tracking. This will be kept to a minimum by using the output of L1Track to seed precision software tracking.

The evolved menu (Table 14.12) shows a scenario in which some hadronic signatures are given lower thresholds to boost acceptance for key physics signals, resulting in an additional EF input rate of around 170 kHz. These signatures would all require additional processing from the EF and gHTT. If necessary the input rate to gHTT can be reduced by pre-selections on some signatures such as the di-tau, and use of regional rather than global tracking in the case of the 4 b-jet signature. The gHTT can be tuned to handle a greater input rate by raising the $p_T$ threshold from 1 to 2 GeV, and/or limiting global tracking to a z-vertex found by L1Track. These more advanced uses of HTT could be commissioned after the baseline functionality for use with or without the evolved architecture. Some analysis-specific selection may be needed to limit the EF output rate, but this would not be expected to impose significant computing requirements.
## 14.4.9 HTT

In the evolution of the TDAQ system to the two-level architecture, the hardware that supports the regional hardware-based tracking system (rHTT) is relocated from the EF to the Level-1 trigger system, reconfigured, and renamed L1Track. In order to compensate for the latency requirement on L1Track, the track $p_T$ threshold is raised to 4 GeV (from 2 GeV in rHTT). The specifications for the rHTT and L1Track are compared in Table 14.7.

<table><caption>Table 14.7: Comparison of specifications for the rHTT in the single-level trigger scheme and L1Track in the evolved design.</caption><thead><tr><td>Trigger</td><td>Latency requirement</td><td>Level-0 rate [MHz]</td><td>Trigger threshold [GeV]</td></tr></thead><tbody><tr><td>rHTT</td><td>No</td><td>1</td><td>2</td></tr><tr><td>L1Track</td><td>6.0 μs</td><td>2–4</td><td>4</td></tr></tbody></table>

Regional hits from the three outer layers of the ITk pixel detector and all layers of the ITk strip detector are transmitted to L1Track upon a L0A based on regions identified by the Global Trigger. These regions can compose up to 10% of the ITk detector volume for a given L0A rate scheme. The L1Track coverage in the forward region relies on a sufficient number of Pixel layers being read out upon a L0A.

In the 2 MHz L0A rate scheme, 192 of the AMTPs will be configured as the L1Track system. For a 4 MHz L0A rate, with the most simple extrapolation, 384 AMTPs are configured as L1Track. Further optimisation for the 4 MHz case should be performed to reduce the required AMTPs modules. The gHTT function will remain as a separate system that works as a co-processor to the EFPU in the EF. In the single-level scheme just described, the split of L1Track and gHTT functions into two separate system requires duplication of AM patterns and it is likely that additional hardware will be required. The need for additional hardware can be removed either relaxing the requirements on gHTT or using the full HTT system to implement both L1Track and gHTT functions together. This latter case would allow to share AM patterns among the two functions as in the Level-0-only TDAQ design. Further developments are needed to evaluate the performance of a combined L1Track and gHTT system, where L1Track processing is prioritised. System diagrams for L1Track and gHTT are shown in Figs. 14.10 and 14.11, respectively.

The simulated track fitting efficiency and the average number of fits for first stage track fitting is shown for single jets embedded in minimum bias of pileup 200 in Tables 14.8. Because L1Track latency depends on the tail of distributions, both average number of fits and the 99% - point of the distribution are given. Track parameter resolutions for L1Track are presented in Section 13.5.
<figure><img src="image_245.png" /><figcaption>Figure 14.10: Overview diagram of the L1Track system showing interconnections within L1Track units and with the FELIX.</figcaption></figure>

<table><caption>Table 14.8: First stage track fitting performance for single jets ($p_T$ > 4 GeV) in PU200.</caption><thead><tr><th>η range</th><th># fits (99%)</th><th># fits χ<sup>2</sup> < 40 (99%)</th><th># tracks HW (99%)</th></tr></thead><tbody><tr><td>0.1 &lt; η &lt; 0.3</td><td>614 (3287)</td><td>27 (158)</td><td>1.5 (10)</td></tr><tr><td>0.7 &lt; η &lt; 0.9</td><td>323 (2094)</td><td>19 (129)</td><td>1.1 (6)</td></tr><tr><td>1.2 &lt; η &lt; 1.4</td><td>581 (3038)</td><td>32 (200)</td><td>2.2 (13)</td></tr><tr><td>2.0 &lt; η &lt; 2.2</td><td>99 (551)</td><td>29 (153)</td><td>1.7 (7)</td></tr></tbody></table>

## Latency of L1Track

The latency of the HTT is not a critical parameter in the single-level trigger scheme since data is sent to the HTT from storage. In contrast, the latency is a critical parameter for L1Track.

The latency of data flow through the PRM has been studied using a simple discrete event simulation written in Python using the SimPy package. The simulation assumes a L1Track configuration with 2 MHz L0A accept rate and a 10% detector volume processed by the regional tracking per L0A. Realistic distributions for the number of input hits, the number of roads identified for subsequent processing, and the resulting number of track fits required, are taken from the pattern recognition studies described in Section 13.5.1. The model for
<figure><img src="image_246.png" /><figcaption>Figure 14.11: Overview diagram of the gHTT system showing interconnections within HTT units and with the HTTIF.</figcaption></figure>

dataflow and the timings used for each stage are shown in Fig. 14.12. The model do not include dataflow on crate level. In future studies the model will be expanded to take the crate level latency into account.

In the simulation, events are generated at random from realistic distributions of time difference between successive L0A and with the parameters for the occupancy and fit multiplicities from the distributions mentioned, but taken collectively, to ensure that correlations between number of hits, number of fits and the processing times taken are all properly considered. The simulation generates the individual events at random, and so the simulation of expected queuing times into each stage of the simulation is correctly handled. The timeline for processing the events as explained in 13.6.3 is shown in Fig. 14.13. At t1, the AM processing for event 1 is finished and SSIDs from event 2 can be input in parallel with the output of roads from event 1. At t2, SSIDs from event 3 can be input in parallel with the output of roads from event 2. However, the input of SSIDs for event 4 cannot start until t3, when the output of roads from event 2 finishes even though the input and processing of SSIDs from event 3 have both finished earlier. The diagram also shows that the reading of hits on roads from the DO, track fitting and output happen largely in parallel.

The simulation shows that the latency is primarily determined by the input of the superstrips to the AM chips through their shared buses. This is the motivation for having the
<figure><img src="image_247.png" /><figcaption>Figure 14.12: The dataflow within the PRM on the AMTP, including with the timing used in the discrete event simulation.</figcaption></figure>

<figure><img src="image_248.png" /><figcaption>Figure 14.13: Illustrative timeline for event processing in L1Track.</figcaption></figure>

AM chips on a PRM separated into two zones with independent buses for each zone. With two sets of input buses and a cut-off at 2.5 µs for the input stage, the simulation shows that greater than 99% of the super-strips can be read in to the AM chips within the cut-off time. Figure 14.14 shows the time spent waiting for the input bus to become available and the percentage of super-strips that arrive at the AM chips. Current studies indicate that AMTP and PRM algorithms will have sufficiently low latency. More complete post-TDR studies will be needed to confirm that the resources on the FPGAs planned for the L0 only TDAQ architecture will be sufficient to achieve the latency goals. At the time of FDR it
<figure><img src="image_249.png" /><figcaption>Figure 14.14: Simulation results for L1Track running at 2 MHz showing the time spent waiting for the AM chip input buses to become free on the left; and, right, the percentage of hits that arrive at the AM chips within 2.5 µs.</figcaption></figure>

<table><caption>Table 14.9: L1Track input dataflow for a LOA rate of 2 MHz. See text.</caption><thead><tr><th></th><th></th><th>L1Track 2 MHz</th><th>L1Track 4 MHz</th></tr></thead><tbody><tr><td></td><td>Rate (equivalent)</td><td>200 kHz</td><td>400 kHz</td></tr><tr><td></td><td>total - crate - module</td><td>total - crate - module</td><td>total - crate - module</td></tr><tr><td>L1Track regional</td><td># of shelves</td><td>16</td><td>32</td></tr><tr><td>L1Track regional</td><td>no overlap b/w</td><td>3200 - 200 - 17 Gb/s</td><td>6400 - 200 - 17 Gb/s</td></tr></tbody></table>

should be evaluated which FPGAs will be sufficient to implement the single-level and the dual-level TDAQ architecture.

## L1Track Dataflow and Power Consumption

The estimated input dataflow and first-stage dataflow for L1Track in the evolved system is summarised in Tables 14.9 and 14.10. The input dataflow values are approximate because they use the same duplication factors for both 2 MHz and 4 MHz. The 2 MHz configuration in Table 14.10 provides two copies of 0.7M patterns per 0.2 × 0.2 HTT-region. The 4 MHz configuration provides two copies of 1.4M patterns per 0.2 × 0.2 HTT-region. Further studies optimising the usage of patterns and FPGA processing as a function of rate are needed to optimise the L1Track system size. In Table 14.10 the relevant parameter for the cluster rate is shown for half a PRM since each half PRM will perform AM pattern matching independently. The calculation of the output bandwidth assumes only 160 bits per track, which include the track parameters and basic information. The transmission of the clusters on tracks uses an additional 480 bits that are not needed for the Level-1 decision but are useful for regional tracks in the EFPUs.

The power consumption for L1Track operated at both 2 MHz and 4 MHz is similar to that of the baseline HTT.
<table><caption>Table 14.10: L1Track first-stage dataflow summary for a L0A rate of 2 MHz and 4 MHz.</caption><thead><tr><th></th><th>L1Track 2 MHz</th><th>L1Track 4 MHz</th></tr></thead><tbody><tr><td>Rate(equivalent)</td><td>200 kHz</td><td>400 kHz</td></tr><tr><td>#PRMs</td><td>384</td><td>768</td></tr><tr><td># HTT-regions/PRM</td><td>3.3</td><td>1.7</td></tr><tr><td>#clusters/0.2×0.2</td><td>210<sup>a</sup> (&lt;150&gt;<sup>b</sup>)</td><td>210<sup>a</sup> (&lt;150&gt;<sup>b</sup>)</td></tr><tr><td>#clusters/half PRM (per event)</td><td>350<sup>c</sup> (&lt;250&gt;<sup>d</sup>)</td><td>210<sup>c</sup> (&lt;150&gt;<sup>d</sup>)</td></tr><tr><td>Cluster rate/half PRM</td><td>50 MHz<sup>a</sup></td><td>60 MHz<sup>a</sup></td></tr><tr><td>#roads/PRM(per event)</td><td>330</td><td>170</td></tr><tr><td>Fit rate/PRM</td><td>400 MHz</td><td>400 MHz</td></tr><tr><td>Tracks out per PRM(per event)</td><td>8</td><td>4</td></tr><tr><td>Bandwidth out/PRM</td><td>260 Mbps</td><td>130 Mbps</td></tr><tr><td>Bandwidth out/shelf</td><td>4 Gb/s</td><td>4 Gb/s</td></tr><tr><td>#ATCA shelves</td><td>16</td><td>32</td></tr><tr><td>Total Bandwidth out</td><td>100 Gb/s</td><td>200 Gb/s</td></tr></tbody></table>

a Busiest layer.

b Average layer occupancy.

c Busiest layer with worst case overlap estimate.

d Average layer occupancy with worst case overlap estimate.

<table><caption>Table 14.11: Power summary for L1Track for 2 MHz and 4 MHz L0A rate schemes.</caption><thead><tr><th></th><th>L1Track 2 MHz</th><th>L1Track 4 MHz</th></tr></thead><tbody><tr><td>number of AMTPs</td><td>192</td><td>384</td></tr><tr><td>Main card (including DC/DCs)</td><td>100 W</td><td>100 W</td></tr><tr><td>PRM FPGA</td><td>30 W</td><td>30 W</td></tr><tr><td>PRM 12 AM ASIC</td><td>60 W</td><td>60 W</td></tr><tr><td>PRM others (RAMs, IO fanout, AM core DC/DC)</td><td>25 W</td><td>25 W</td></tr><tr><td>Total/AMTP</td><td>330 W</td><td>330 W</td></tr></tbody></table>

## 14.5 Physics Opportunities with the Evolved System

The evolved system provides a mitigation strategy for detector readout bandwidth limitations and hadronic trigger rate uncertainties, but it also provides opportunities to reduce thresholds and improve acceptance for physics signatures. One use case of this option is to address a new physics discovery or important theoretical progress. Here we present how the additional rate available could be used to substantially improve acceptance for a range of known Standard Model and BSM signatures. Table 14.12 shows the gains achievable by distributing the additional Level-0 rate across four prospective trigger selections: $E_T^{\text{miss}}$, di-hadronic tau, 4-jet, and inclusive VBF Higgs. The $E_T^{\text{miss}}$ trigger would deliver $\approx 2.4 \times$ the acceptance for $ZH \to \nu\bar{\nu}b\bar{b}$ and $\approx 2 \times$ acceptance for the example compressed SUSY model presented in Section 2.6. The di-hadronic tau trigger would increase the signal acceptance
from 30% to 55% for the VBF $H \to \tau\tau$ channel and 32% to 54% for the $HH \to b\bar{b}\tau\tau$ channel. The four-jet trigger would improve the limit on $HH \to 4b$ from 1.85 $\sigma/\sigma_{SM}$ to 1.65 $\sigma/\sigma_{SM}$. Finally, the inclusive VBF Higgs acceptance would increase from 6.6% to 10%. These are the acceptance gains that would be available at the Event Filter, but the corresponding events could not be recorded in the planned 10 kHz Event Filter output limit. Instead, analysis-specific selections would need to be applied. For example, a requirement of a soft lepton in addition to $E_T^{\text{miss}}$ would be appropriate for the SUSY models, and the exotic Higgs searches could require additional soft jets in addition to a $E_T^{\text{miss}}$ requirement. These are possible in the Event Filter but not in the Level-0 trigger because of the low thresholds of these objects.

<table><caption>Table 14.12: Prospective additional triggers for an evolved system. The gains for example physics channels are described in the last column.</caption><thead><tr><th>Signature</th><th>Single-Level Scheme Threshold</th><th>Dual-Level Scheme Threshold</th><th>Level-0 (kHz)</th><th>Level-1 (kHz)</th><th>EF before analysis specific cuts (kHz)</th><th>Gain</th></tr></thead><tbody><tr><td>E<sub>T</sub><sup>miss</sup></td><td>210 GeV</td><td>160 GeV</td><td>800</td><td>80</td><td>3</td><td>2&times; acceptance for compressed SUSY model and 2.4&times; for ZH &rarr; vवb b</td></tr><tr><td>di-&tau;</td><td>40, 30 GeV</td><td>30, 20 GeV</td><td>800</td><td>80</td><td>2.2</td><td>increased acceptance from 30% to 55% for VBF H &rarr; &tau;&tau; and 32% to 54% for HH &rarr; b\bar{b}\tau\tau</td></tr><tr><td>4 jet w/ 2-btags</td><td>65 GeV</td><td>55 GeV</td><td>800</td><td>100</td><td>0.4</td><td>improved limit in HH &rarr; 4b from 1.85 to 1.65 &sigma;/&sigma;<sub>SM</sub></td></tr><tr><td>VBF Higgs</td><td>75 GeV + topological</td><td>60 GeV + topological</td><td>280</td><td>40</td><td>40</td><td>increased acceptance from 6.6% to 10% for inclusive VBF Higgs production</td></tr><tr><td>Total</td><td></td><td></td><td>2680</td><td>300</td><td>-</td><td></td></tr></tbody></table>

The example evolved system menu in Table 14.12 allows 3 MHz more events to be passed from Level-0 to Level-1, but that rate must then be reduced to 600 kHz using regional tracking and algorithms similar to those described in Section 6. The data fraction of an event needed is similar to the fractions shown in Table 6.5. The increased rate into the Event Filter leads to an increased need for precision tracking with gHTT for use in b-tagging and track-based pile-up corrections and calibrations. In order to mitigate this increased need, the HTT co-processor can be used in a regional or a global capacity. Because of the length of the beam spot, a large amount of detector data is required to reconstruct charged particles arising from a given vertex. However, in the evolved system charged particles have already
been reconstructed in L1Track, along with the possibility for coarse determination of the z-position of the primary vertex. This can be used to restrict the detector data used so that only tracks that originate from a region along the beam line near the triggered collision are reconstructed. A similar algorithm is used in the Run 2 HLT for b-tagging; first, a vertex is reconstructed with a fast tracking algorithm, and then the precision tracking is restricted to a selected region within the beam spot. This aggressive algorithm is not assumed for the single-level configuration in order to ensure its robustness and flexibility.

## 14.6 Challenges of the Evolved System

The single-level and dual-level trigger configurations chosen for the baseline trigger architecture have been carefully evaluated. The single-level hardware trigger architecture was chosen in order to minimise the complexity of the TDAQ system and TDAQ-detector interactions, the cost, and the challenges of commissioning L1Track. These considerations are described in detail in the following paragraphs.

The evolved architecture configuration is considerably more complicated than the single-level architecture. In addition to performing all of the capabilities of the single-level hardware trigger, the dual-level hardware trigger in the evolved system must accommodate the fast collection of RoI information and the formation of R3 signals, as well as the processing of L1Track events once they are received by the Global Trigger. The data handling in FELIX and the Data Handler is significantly more complex, as is the signalling from the Central Trigger via the LTIs to the FELIX. More complex signal handling is also used for detectors that make use of the Level-1 signal before reading out, notably the strip and inner-pixel front-end ASICs.

There are additional costs associated with the evolved scheme, including those associated with reading out the detectors at the higher Level-0 rates (especially the outer pixels that contribute to L1Track); the higher rate L1Track processing; additional modules, patch panels, and fibres for L1CTP, an additional MUX module to handle inputs from L1Track for the Global Trigger, the RoIE for the Global Trigger; the RoI distribution; the additional FELIX-L1Track interface; and finally, additional AMTPs needed to split the HTT functionality. In addition, there are other costs associated to the detector systems that are described in the relevant TDRs. Foreseeing the Level-1 functionality in the baseline scenario leads to a very small impact on the overall cost of the system. These considerations are summarised in Table 14.13.

The single-level trigger configuration has the distinct advantage of a simpler commissioning scheme than that of the evolved system. Furthermore, the baseline system will need to cope with trigger rates that are an order of magnitude higher than those in Run 3. However, in commissioning the single-level trigger system, there is no need for disruption to the detector readout while the HTT is being commissioned. It will be possible to operate the rHTT passively, sending data to and recording results from the rHTT without using
the results in the EF trigger decision<fnref n="2" />. In contrast, commissioning the L1Track trigger for the evolved scheme requires intervention at the level of the FELIX for the ITk outer pixels and at the level of the on-detector ASICs for the strips. If these actions provoke synchronisation errors, for example, there could be a serious impact on the data-taking, including the possibility of data loss and downtime. Should ATLAS determine it to be necessary or beneficial to the physics programme to move to the evolved architecture at a later stage, the operations team would be integrating well-understood HTT hardware into the Level-1 trigger, with mature FELIX and ITk electronics systems.

Experience from ATLAS operations during Run 1 and Run 2 dictates that reliability, robustness, and stability are most easily achieved when systems are kept as simple as possible.

<table><caption>Table 14.13: Summary of hooks put into the baseline design in order to accommodate a potential evolution and additional hardware and firmware needed for the evolved system.</caption><thead><tr><th>System Component</th><th>Hooks for Evolved System</th><th>Additional Hardware/Firmware Needed for Evolved System</th></tr></thead><tbody><tr><td>Level-0 Calo</td><td>sufficient FPGA resources</td><td>minor firmware changes</td></tr><tr><td>Level-0 Muon</td><td></td><td></td></tr><tr><td>Global Trigger</td><td>extra transceivers</td><td>additional MUX modules to receive information from L1Track; extra GCM modules configured as RoIEs; extra GCM module for L1CTP interface; additional GEP firmware</td></tr><tr><td>Central Trigger</td><td>extra optical connectivity</td><td>add L1CTP plus additional patch panels and fibres</td></tr><tr><td>Readout</td><td>none</td><td>additional FELIX I/O Cards, servers, and Data Handlers; new FELIX/Data Handler firmware and software; low-latency links to L1Track;</td></tr><tr><td>Dataflow</td><td>none</td><td>larger bandwidth requirements</td></tr><tr><td>Event Filter</td><td>none</td><td>significant increase in computing power</td></tr><tr><td>HTT</td><td>rHTT hardware and firmware must meet L1Track latency requirement</td><td>separation of regional and global functionality; additional AMTPs; new firmware</td></tr></tbody></table>

References

[14.1] ATLAS Collaboration, Technical Design Report for the ATLAS Inner Tracker Pixel Detector, Tech. Rep. CERN-LHCC-2017-021, CERN, Geneva, (in preparation).

<footnote n="2">² This is similar to the commissioning strategy used for FTK.</footnote>
[14.2] *MiniPOD AFBR-814VxyZ, AFBR-824VxyZ, 14 Gbps/Channel Twelve Channel, Parallel Fiber Optics Modules*, Tech. Rep. AV02-4039EN, Avago Technologies, 2013.
# 15 DCS and TDAQ interfaces

The DCS [15.1] has the task to permit coherent and safe operation of ATLAS and to serve as a homogeneous interface to all sub-detectors and to the technical infrastructure of the experiment. The DCS must be able to bring the detector into any desired operational state, to continuously monitor and archive the operational parameters, to signal any abnormal behaviour to the operator, and to allow manual or automatic actions to be taken. In order to synchronise the state of the detector with the operation of the physics data acquisition system, bi-directional communication between DCS and run control must be provided. Finally, the DCS has to handle the communication between the ATLAS sub-detectors and other systems that are controlled externally, such as the LHC accelerator, the CERN technical services, the ATLAS magnets and the Detector Safety System (DSS).

<figure><img src="image_250.png" /><figcaption>Figure 15.1: Front-end and back-end components within a DCS system.</figcaption></figure>

Figure 15.1 shows the architectural components defined for a given system within DCS which is divided into Front-End (FE) and Back-End (Back-End (BE)). The detector hardware is equipped with FE interface equipment which can be purpose-built electronics and their associated services such as power supplies or use general purpose I/O components such as the Embedded Local Monitor Board (ELMB) [15.2] or ELMB++ [15.3]. The connection to the BE is provided by a communication layer such as Ethernet or CAN bus. The BE is formed by Local Control Stations (LCSs) (COTS rack servers) and their communication interfaces (e.g. Ethernet Network Interface Card (NIC) or CAN bus interface).

The BE is organised in three layers (see Fig. 15.2): the LCSs for process control of sub-systems, the Sub-detector Control Stations (SCSs) for high-level control of a sub-detector allowing stand-alone operation, and the Global Control Stations (GCSs) with server applications for common functions and human interfaces in the ATLAS control room for the overall operation. The BE consists of control stations, interconnected as a distributed system of Supervisory Control And Data Acquisition (SCADA) applications based on Simatic WinCC Open Architecture (WinCC OA). Communication to the FE interfaces is performed
using a middleware layer based on the industry standard Open Platform Communications
Unified Architecture (OPC UA).

<figure><img src="image_251.png" /><figcaption>Figure 15.2: *Architecture of the DCS Back-End.*</figcaption></figure>

The full BE hierarchy from the operator interface down to the level of individual devices
is represented by a distributed Finite State Machine (FSM) allowing for standardised op-
eration and error handling in each functional layer. The SCS layer also serves as primary
interface to the TDAQ run control system while the GCS layer integrates with the external
control systems (LHC, Cryo, Cooling etc.). All control stations may archive operational
parameters to a global database.

**15.1 Interfaces for On-Detector FE Components**

Monitoring and control of on-detector components will follow two different paths with re-
spect to communication and powering. The first path is entirely dedicated to DCS functions
with independent communication and power lines. This serves as the primary path for im-
portant DCS data which require a high degree of reliability and availability at all times,
including when the detector and its integrated electronics are not fully powered. An ex-
ample for this path would be an implementation based on the proposed ELMB++ [15.3].

The secondary path is using the communication infrastructure of the on-detector electronics
usually based on the GBT or lpGBT chipsets which are interfaced with the off-detector back-
end via Versatile Links (optical) [15.4] and the FELIX system. The controls data in this path
may in some cases need to share resources with other types of data such as readout and