trigger data. This path is foreseen for controls data with less critical use cases or if an
independent path is impossible to implement, e.g. due to constraints on physical space or
available services.

<figure><img src="image_252.png" /><figcaption>Figure 15.3: On-detector front-end interface schema showing the two paths for controls data and associated powering. The primary path shown is an implementation based on the ELMB++ chain. The secondary path shown is for front-ends such as the GBT Slow Control Adapter (GBT-SCA) or custom ASICs interfaced with Versatile Links and the FELIX system. The handling of DCS data shown separates the data paths in firmware only. Another possibility is to separate the data in software on the FELIX host.</figcaption></figure>

Figure 15.3 gives an overview of these two paths for which details are given in the following
two sections.

15.1.1 Exclusive DCS On-Detector Control Path

Control and supervision of on-detector hardware components in the ATLAS cavern usu-
ally covers situations when these components are not powered or the powering has issues
which need to be diagnosed. Independent (remote) powering of the DCS FE interface hard-
ware is thus vital along with an independent communication path.

In analogy to the current ATLAS solution based on the ELMB [15.2] for which the com-
munication is based on CAN bus and associated powering is provided by dedicated CAN
power supplies, the ELMB++ was proposed [15.3] as the standard solution to implement the
primary path for DCS systems. Apart from the critical improvement in radiation hardness
required by the HL-LHC environmental conditions, the ELMB++ provides general perform-
ance enhancements and additional analog and digital interface standards. The functionality
covers:
* ADC channels with effective conversion rate on the kHz level and 12 bit precision,

* DAC channels (12 bit),

* digital I/O support for SPI, JTAG, I²C, GPIO,

and is provided by an *ELMB++* satellite containing for example *GBT-SCA* ASICs with associated *Electrical chip-to-chip interconnect (e-link)* connectivity and a DC-DC converter for remote powering. Several satellites can be interfaced in a star-point topology directly via *e-links* to a *mobile receiver* if the link distance does not exceed a few meters or to an intermediate *ELMB++ hub*. The hubs feature full *GBT* (with associated *GBT-SCA*) or *lp-GBT* functionality together with a DC-DC converter for remote powering. Towards the back-end, several hubs can be connected via Versatile Links to the mobile receiver. These individual components of the ELMB++ variant are shown in Fig. 15.4 while the topology architecture is illustrated in Fig. 15.5.

### 15.1.2 On-Detector Control via the Readout Path

For the secondary control path where ASIC components with DCS related functions are interfaced via Versatile Links (e.g. via the GBT chipset), the communication on the off-detector side is handled by the FELIX system. Two categories of functions should be distinguished: controls data for higher priority control functions which are important for the regular operation of the detector component such as powering control and monitoring and secondly lower priority detector diagnostics data.

If affordable, a dedicated Versatile Link for DCS data is foreseen to avoid interference with other data types which could lead to untimely or unsuccessful delivery of controls data. However, in many systems this separation is not feasible, e.g. due to space or power constraints, and the data streams are only possible to be separated within the FELIX system.

Within a given Versatile Link, the controls data should be always handled via the GBT protocol and associated exclusively to one or several *e-links* and thus controls data can be identified within FELIX by the *GBT e-link* identifier. Data for these *e-links* is separated by FELIX in firmware and/or software and transferred to an OPC UA server which is connected to the DCS back-end via a separate physical network interface unit (cf. Fig. 15.3).

<figure><img src="image_253.png" /><figcaption>Figure 15.4: *Components of the ELMB++ chain.*</figcaption></figure>
<figure><img src="image_254.png" /><figcaption>Figure 15.5: *Possible topology for the use of the ELMB++ components.*</figcaption></figure>

If a separation of controls data is achieved in firmware only, sharing the *PCIe* bus and software with the non-*DCS* data could be avoided. Further, this approach would allow a closed control loop between the OPC UA server on the *FELIX* I/O card and the detector *FE* for important *DCS* functions. This would minimise downtime by not depending on host software or operating systems of the *FELIX* or *DCS BE* system hosts. For example in cases where the *FELIX* host needs to be rebooted, software processing is completely unavailable or processing suffers from back-pressure of other data types, the *DCS* path becomes quickly available again when the *FELIX* I/O card is available for data processing.

The lower priority diagnostics data are often merged into the event data streams, either within event data e-links if *GBT* is used or within customised protocols. On the off-detector side, the point of separation of these data from the event data still needs to be established.

**15.2 Interfaces for Off-Detector Front-End Components**

Off-detector *FE* components can be divided into the following categories:

A. Commercial power supplies or similar devices with proprietary communication interface,

B. Off-detector electronics crates and their commercial or purpose-built boards following the *ATCA* or *VME* standards,

C. Other purpose-built devices.

For all device types the standard middleware solution *OPC UA* is used to integrate them into the *DCS BE*. As a general principle, this middleware should be used as close to the actual device as possible in order to profit from a standard communication interface on the physical as well as the logical level. However, this principle cannot be applied equally to the device types A-C and thus different solutions are applied as noted below.
### 15.2.1 Devices with Proprietary Interfaces

Standardisation of proprietary components such as power supplies is provided in the frame of the Joint Controls Project (JCOP). For each supported hardware component, Ethernet is the communication layer of choice and integration software based on OPC UA is provided along with framework components for WinCC OA. The equipment manufacturers provide a hardware abstraction layer which is then used inside the OPC UA server implementation supported by JCOP.

Similarly, for non-standard equipment using proprietary protocols over Ethernet, dedicated OPC UA servers will be implemented using a standardised server generation framework detailed in Sec. 15.3 which is also used for the OPC UA servers provided by JCOP.

### 15.2.2 Off-detector Electronics based on ATCA and VME Standards

Most off-detector electronics will be based on the ATCA standard together with legacy equipment based on VME. For both standards, DCS monitoring and control is provided at the level of the crates/shelves and the individual boards.

The proprietary VME crate interface is covered by a JCOP OPC UA solution (see also previous section) via Ethernet or CAN bus.

<figure><img src="image_255.png" /><figcaption>Figure 15.6: Off-detector front-end interface schema for ATCA-based electronics. Two main paths for DCS integration are shown: via the ATCA shelf manager (top) and via a SoC embedded into the ATCA board (bottom). The third path in the middle (grey shades and dashed lines) is based on an IPBus interface used in Phase-I upgrades, which is deprecated for Phase-II designs though.</figcaption></figure>
For ATCA equipment, DCS integration is achieved using two complementary control paths as shown in Fig. 15.6. The first path is based on interfacing the CERN standard ATCA shelf manager via Ethernet. The shelf manager SNMP interface is then used for an OPC UA server implementation in the DCS BE for WinCC OA integration. In contrast to VME, the ATCA standard clearly defines the monitoring and control parameters which are available for the shelf. Further, the standard defines the interface for ATCA boards for which control and monitoring is implemented using communication between an IPMC on the shelf manager and the board respectively. Thus the standardised board parameters such as health, presence, power, reset and custom sensor data are then also available via the shelf manager SNMP interface. However, this solution does not scale well above a few hundred parameters and thus cannot be the only DCS path for ATCA boards.

The second control path for individual boards is based on embedded processing units such as SoC components or FPGA softcore solutions which allow embedding an operating system together with an OPC UA server directly in the hardware. The OPC UA server allows a more scalable implementation of monitoring and control functions based on the digital or analog interfaces provided by the associated FPGA or SoC firmware.

If an on-board processor element is not available, data exchange via IPBus is used in some cases as fallback solution for Phase-I upgrades although it is less flexible and requires more complex software on the DCS BE (an IPBus OPC UA server). For Phase-II designs this path is considered deprecated and the SoC approach is preferred.

### 15.2.3 Interfaces for Other Purpose-Built Off-Detector Front-Ends

Any other type of (non-standard) front-end hardware component is integrated using the methods described above. In case a processor element is directly built into the hardware, an embedded OPC UA server serves as DCS FE interface connected via Ethernet. Alternatively, an ELMB++serves as I/O concentrator and allows direct monitoring or control of analog or digital I/O channels.

For FELIX systems which are used for DCS functions, monitoring and control of the respective FELIX hardware and software components themselves are provided directly within the OPC UA server embedded into FELIX.

## 15.3 Back-End and Middleware

### 15.3.1 Back-End Hardware and Software

The hardware platform for the BE system is a set of industrial, rack-mounted server machines balancing maximum reliability with low cost, providing redundancy for many components such as power supplies, hard disks and network interfaces. With Ethernet as the
standard communication layer for FE integration, the replacement of a BE server should
incur minimal downtime by relocating the affected BE application to a spare server.

The SCADA package WinCC OA (formerly known as PVSS) is the base for the BE applications, extended by the JCOP framework [15.5]. Since all FE systems are interfaced via the network, virtualisation of server operating systems and encapsulation of BE applications in software containers could ensure reliability and maintainability of the BE software on all levels.

The bi-directional interface between DCS BE and TDAQ run control applications will be
provided by means of a software implementation with a generic API.

15.3.2 Standard Middleware OPC UA

The middleware standard for DCS systems is OPC UA, an industry standard for machine-
to-machine communication in the controls domain, featuring:

* independence from operating systems, availability for a variety of platforms and allowing development in various programming environments. The OPC UA reference stack (providing lower layers of OPC UA internally to OPC UA toolkits) is delivered in standard C. OPC UA toolkits are available in C++, Java, .NET, python and others. Moreover, the ubiquitous TCP/IP is the communication technology used by OPC UA allowing for full portability of OPC UA to any modern network-aware operating system,

* robust data modelling capabilities: thanks to graph-based information model, data can be organised in virtually any fashion, from simple variables (of built-in or custom types) through structures (simple and nested) and trees, up to possibly cyclic graphs representing not only the current state of the system but also its meta-model and associated knowledge [15.6]. In addition, a number of techniques from object-oriented modelling is available, like method invocations, as well as techniques known from relational data modelling like queries and views. Type information is fully exposed to the client(s) and may be accessed as type instances.

* embedding into custom hardware – thanks to limited dependency on standard C and C++, some common cryptography functions and the TCP/IP stack, one can run OPC UA servers on any embedded computing element that has enough memory (few MB) and TCP/IP connectivity. A number of successful miniaturised and embedded controllers with OPC UA servers are available on the market,

* secure communication: thanks to PKI (Public Key Infrastructure) support in OPC UA, one can use state-of-the-art security in OPC UA based control infrastructure.

In order to reduce development and maintenance efforts, a framework for OPC UA server
creation is available – the Quick OPC UA Server Generation Framework (quasar)
[15.7]. Development starts with creation of a design file, in XML format, describing an
object-oriented information model of the target system or device. Using this model, the
framework generates an executable OPC UA server application, which exposes the per-design OPC UA address space, without the developer writing a single line of code. Furthermore, the framework generates skeleton code into which the developer adds the required target device/system integration logic. This approach allows both developers unfamiliar with the OPC UA standard, and advanced OPC UA developers, to create servers for the systems they are experts in while greatly reducing design and development effort as compared to developments based purely on COTS OPC UA toolkits. Higher level software may further benefit from the explicit device model by using the XML design description as the basis for generating client connectivity configuration and server data representation. Moreover, having the XML design description at hand facilitates automatic generation of validation tools.

<figure><img src="image_256.png" /><figcaption>Figure 15.7: OPC UA server architecture within the *quasar* framework.</figcaption></figure>

Figure 15.7 gives an overview of the different layers of *quasar* put into context. Controllable devices or systems are accessed using their specific access layer – often provided together with the specific device. The device logic layer functions as interface with the high level layers of *quasar* which comes in several modules covering different functional aspects. The address space module sits in the OPC UA end of the server, exposing data towards OPC UA clients, and is implemented using a OPC UA back-end layer with ex-changeable back-end implementations. A configuration module facilitates address space and device instantiation and the definition of their relations. XML is used as the configura-tion format backed by XML schema definitions. A XML schema to C++ mapping generator (here: xsd-cxx) is used to build actual instances from configuration files. An additional subsystem called Calculated Items, operating entirely in the address space, enables cre-ation of new variables which are derived from existing ones using mathematical functions. *quasar* further comes with optional modules such as component based logging, certific-ate handling, server meta-data, embedded python processing, WinCC OA integration tools and SQL/NoSQL archiving with historic data access. A ready-to-use build system based on CMake along with pre-configured tool-chains for several platforms such as x86_64 or
ARM-based Linux and Microsoft Windows are provided. Finally, an [OPC UA](https://www.opc.org/) client generation facility called UaoForQuasar is available for building C++ clients for *quasar*-based servers.

### 15.3.3 Network

The ATLAS Control Network (ATCN) integration of any DCS related network interface is implemented based on Ethernet with two main requirements. Firstly, DCS data should be exchanged between network devices with a minimum of network resources shared with other types of data by using dedicated switches exclusively assigned to DCS devices. Further, routing of data between DCS switches should be done with dedicated router hardware components. Secondly, individual DCS devices should be secured at the network level, e.g. by using Virtual Local Area Network (LAN) (VLAN) or private network gateways, allowing only connections to their respective DCS BE control stations and thus preventing unauthorised access.

## 15.4 DCS of TDAQ Sub-Systems

Introducing the new TDAQ hardware into P1 requires a well configured monitoring and control interface to ensure the system is in an operational state. Integrating these systems into the ATLAS DCS allows virtual access to the devices installed underground and in the counting rooms. Apart from the clear advantage of monitoring the state of the hardware and its hosting environment, the ability to keep track of the on-board sensors can help to detect a potentially upcoming hardware failure, and allow action to be taken prior to the system triggering a safety system interlock.

The TDAQ DCS includes all Level-0 Trigger subsystems – i.e. L0Calo (eFEX, jFEX, gFEX and fFEX), L0Muon (Barrel Sector Logic, Endcap Sector Logic, MDT Trigger Processor and NSW Trigger Processor), MUCTPI, Global and CTP – as well as the HTT under the EF subsystem.

The TDAQ DCS strategy includes monitoring and controlling of the system environment, e.g. the crates in terms of power and cooling, as well as monitoring their critical parameters including: voltages, currents, power-consumption and temperature. Some control options are also included and can be executed with special expert access rights.

All TDAQ front-end subsystems are based on ATCA technology consisting of shelves and blades. With an ATCA shelf-manager, information is exchanged between IPMCs implemented on both the board and the shelf-manager allowing communication between the two over IPMI. This covers basic shelf monitoring and control of e.g. fan speed, crate-slot occupancy, power etc. The ATCA blade IPMC carries out the minimal board management functions required by the ATCA standard, such as payload power control and indication
of general board health. The blades contain either FPGAs or ASIC chips (or both) for data processing, optical transceivers for data IO, power modules, DC-DC converters, temperature sensors and power-management chips. Most of the systems also contain a SoC for control and monitoring. Thus integration into the DCS BE is achieved through two paths, SoC-to-DCS and IPMC-to-DCS, as described in Fig. 15.6:

* IPMC-to-DCS interface:

A dedicated path to DCS is configured for traffic of IPMCs to the DCS BE. Either board or shelf information is sent to and from the ATCA shelf-manager through a SNMP-to-OPC UA bridge configured on the LCS machine via SNMP communication.

* SoC-to-DCS interface:

An OPC UA server constructed on the SoC using the quasar tool developed by the central-DCS team. Monitoring is established by communication between that server and an OPC UA client configured on the LCS machine where the information is transferred over TCP IP.

On-board hardware component supervision is done via the monitoring device configured as master of the I2C (I²C) bus of the board. Three possible scenarios are considered: in the first the SoC is defined as the master, in the second both the SoC and the on-board IPMC are defined as masters of two separate I²C networks, and in the rare third case that a SoC is not embedded, only the IPMC is defined as master. For the latter case, if additional monitoring of individual board components is needed, an IPBus interface is required. In all cases, the master retrieves the monitoring information over I²C and transfers it to the DCS BE via the two paths outlined above. Using a SoC is considered a more flexible approach in terms of possible extension of parameters to be monitored or controlled. Both the firmware and the software remains under full control of the developers; there are no third-party component dependencies and there are fewer limitations on the possible number of monitored parameters.

On the DCS BE, the configuration of LCS clients, one per monitoring device is set: an SNMP-to-OPC UA bridge for communication with the ATCA shelf-manager and an OPC UA client for the SoC.

The two paths to the DCS BE require several separate Ethernet ports for a single ATCA shelf, one for the shelf manager and several for the aggregated blade interfaces (SoC or IPBus), c.f. Fig. 15.6. All network devices for a given subsystem and the corresponding LCS are connected to at least one ATCN network switch reserved for the respective subsystem DCS. In total, hundreds of network devices and a few ATCN network switches are required.

The DCS provides complete monitoring and control of the blades with human-machine interfaces for the respective hardware. In addition, the ATCA blades are equipped with power-management features, controlling the power distribution on the board, designed to take immediate action in case of over-current/voltage incidents. A dedicated interlock procedure is foreseen depending on the board functionality.
References

[15.1] A. Barriuso Poy, H. Boterenbrood, H. Burckhart, J. Cook, V. Filimonov, S. Franz, O. Gutzwiller, B. Hallgren, V. Khomutnikov, S. Schlenker, and F. Varela, *The Detector Control System of the ATLAS Experiment*, JINST 3 (2008) P05006.

[15.2] B. Hallgren, H. Boterenbrood, H. J. Burckhart, and H. Kvedalen, *The embedded local monitor board (ELMB) in the LHC front-end I/O control system*, proceedings of the 7th Workshop on Electronics for LHC Experiments, Stockholm, Sweden (2001).

[15.3] P. Farthouat, *ELMB++ Proposal, Expert meeting on ELMB successor requirements*, 2015. https://indico.cern.ch/event/535628/contributions/2176605/attachments/1278838/1898759/ELMB_Proposal.pdf.

[15.4] *Versatile Link*, (online). https://espace.cern.ch/project-versatile-link/public/default.aspx.

[15.5] O. Holme, M. Gonzalez Berges, P. Golonka, and S. Schmeling, *The JCOP framework*, . [Conf. Proc.C051010,WE2.1(2005)].

[15.6] W. Mahnke, S. Leitner, and M. Damm, *OPC Unified Architecture*. Springer-Verlag, 2009.

[15.7] S. Schlenker, D. Abalo Miron, B. Farnham, V. Filimonov, P. Nikiel, and C.-V. Soare, *quasar - A Generic Framework for Rapid Development of OPC UA Servers*, in *Proceedings, 15th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS 2015): Melbourne, Australia, October 17-23, 2015*, p. WEB3O02. 2015. http://jacow.org/icalepcs2015/papers/web3o02.pdf.
# 16 System Integration, Installation and Commissioning

In this Chapter initial considerations and planning on installation and commissioning are presented. The plans will mature into a set of documents that will define policies and procedures for each stage of the Installation and Commissioning:

* Documents describing Quality Assurance Policy and Quality Control Procedures, to be adopted during construction by each sub-system. The documents shall be part of the documentation to be presented in the PRR reviews.

* Documents describing System Integration and Validation Requirements and Specifications for each sub-system of the integration tests at CERN.

* A detailed Installation and Commissioning Plan document shall be prepared before the PRRs of the sub-systems, see Chapter 19, and a review of the installation plan will be coordinated with the Upgrade Project Office in Technical Coordination.

The chapter is organised as follows:

* Section 16.1 overviews general aspects of the installation and commissioning of the TDAQ upgrade elements in ATLAS.

* Section 16.4 focuses on validation tests and on the initial system integration of the custom hardware components of the three systems: specifically all the L0 trigger components, the HTT modules of the EF system, the FELIX I/O cards in the DAQ system. The Section briefly covers QA/QC policies at the production sites, acceptance tests and initial commissioning on the surface in a dedicated setup hosted at a TDAQ Maintenance Facility, production firmware deployment and system integration tests.

* Section 16.5 briefly describes the installation aspects of the TDAQ Phase-II hardware in USA15.

* The integration of the TDAQ components with the ATLAS detector systems in USA15 and the early commissioning with beam is summarised in Section 16.6.

## 16.1 Overview of the TDAQ Phase-II Integration and Installation

The installation and integration of the TDAQ electronics in ATLAS is the conclusion of several years of development and construction. In general terms, the production of the different components spans over the years 2021-2024. Deliverables will be arriving at CERN at
different times and fully characterised in setups installed on the surface (see Section 16.4.3) before final installation. The sequence of operations for the integration, installation and commissioning of the TDAQ systems can be summarised in the following:

* Single board/unit acceptance tests

* System and/or sub-system integration tests and final validation on the surface

* Installation in ATLAS

* Standalone commissioning

* Detector integration validation and commissioning via dedicated calibration procedures

* Cosmic rays commissioning

* Early beam commissioning

* Operations

### 16.1.1 Location of the TDAQ Phase-II Upgrade Project's Components

Figure 16.1 is a schematic view of the buildings at Point-1 and the access shafts to the ATLAS cavern. The installation of the TDAQ Phase-II upgrade components is in the same locations as the existing Run 2 electronics:

* The counting room and service cavern USA15 houses the electronics racks and the services that needs to be installed close to the detectors. The room is accessible during data-taking operations and hosts all the L0 Trigger sub-systems components, the first elements of the DAQ readout and the HTT hardware.

* SDX1, located on top of the personnel access shaft and used for the personnel and material access control to the underground areas. It also contains the uninterruptible power supplies for the detector services, an electrical sub-station, and a TDAQ room. The TDAQ room is laid out on two floors and houses the Dataflow, the Networking components, and the EF Processing Farm units.

Table 16.1 details the exact location of all the TDAQ components. The motivation for selecting USA15 as baseline location of the HTT sub-system is twofold: (i) compatibility with a possible L0/L1 evolved architecture, where HTT may be used as a Level-1 Track Trigger, and, (ii) the limited space, power, and cooling available in SDX1. Data volume exchange between the EF servers in SDX1 and the HTT electronics in USA15 is relatively limited if compared with the network traffic required to stream the data from the Data Handlers to the Storage Handlers, and, therefore is not a significant concern.

### 16.1.2 Installation and Commissioning Planning

Installation and early commissioning will occur in the years 2025-2026. The installation procedures are relatively simple, as all the electronics is either implemented as ATCA blades or
<figure><img src="image_257.png" /><figcaption>Figure 16.1: Layout of ATLAS Point-1 buildings on the surface and of the access shafts to the ATLAS cavern (from [16.1]). The TDAQ Phase-II upgrade systems are installed in the main counting room and service cavern (USA15) and in the surface.</figcaption></figure>

as pluggable cards in standard rack-mounted PC server-based units. The installation and commissioning is relatively independent on the activities occurring in the detector cavern in USA15. However, there are a few constraints to schedule installation tasks and work packages, which will require coordination with Technical Coordination, ultimately responsible for any installation task in USA15. Namely:

* **Refurbishing of the counting rooms:** Both USA15 and SDX1 are planned to be renovated: (i) in USA15 interventions to isolate acoustically the area with ATCA racks from the rest of USA15 will be required by safety regulations; (ii) in addition, cooling, ventilation, and power distribution will also need to be upgraded; (iii) in SDX1 floor support reinforcement will be likely needed as more servers and racks are to be installed.

* **Decommissioning of the legacy electronics racks:** Several of the legacy electronics of Run 2 and Run 3 will be decommissioned and disposed following ATLAS and CERN regulation.

* **Installation of new racks:** The number of crates and of PC server-based units required by all the ATLAS systems to comply with the new trigger and readout requirements.
<table><caption>Table 16.1: Location in ATLAS of the TDAQ systems' and sub-systems' components</caption><thead><tr><th>System</th><th>Sub-system</th><th>Component</th><th>Location</th></tr></thead><tbody><tr><td rowspan="7">Level-0 Trigger</td><td rowspan="4">L0Calo</td><td>eFEX</td><td rowspan="7">USA15</td></tr><tr><td>jFEX</td></tr><tr><td>gFEX</td></tr><tr><td>fFEX</td></tr><tr><td rowspan="2">L0Muon</td><td>Barrel (RPC) Sector Logic</td></tr><tr><td>Endcap (TGC) Sector Logic</td></tr><tr><td>MUCTPI</td><td>NSW Trigger Processor</td></tr><tr><td>Global Trigger</td><td>MDT Trigger Processor</td></tr><tr><td>CTP</td><td>–</td></tr><tr><td>TTC</td><td>–</td></tr><tr><td rowspan="2">DAQ</td><td>Readout</td><td>FELIX<br>Data Handler</td><td>USA15</td></tr><tr><td>Dataflow</td><td>Event Builder<br>Storage Handler<br>Event Aggregator</td><td>SDX1</td></tr><tr><td rowspan="2">EF</td><td>Processor Farm</td><td></td><td>SDX1</td></tr><tr><td>HTT</td><td></td><td>USA15</td></tr></tbody></table>

ments will exceed the current available rack space in USA15. ATLAS Technical Coordination is considering the installation of taller racks housing up to 3 ATCA shelves. In addition, deeper PC server racks may have to be installed to enable a wider choice of computing platforms. In fact the current racks in use in Run 2 are not compatible with all commodity-server form-factors.

* **Routing of fibres and cables:** New cable plants for most of the detectors will be required. Removal of the old cables, and the installation and routing of new cables and optical fibres will be a significant activity of Technical Coordination during Long Shutdown 3.

* **Rack Layout Configuration:** Optimisation of the rack layout and of the rack configuration is critical for the L0 Trigger system. TDAQ and Technical Coordination will coordinate the allocation of the ATCA racks to the systems, in particular to minimise contribution of cable routing and interconnects to the L0 trigger latency.

Analysis of the required interventions is on-going by Technical Coordination, and a detailed plan of the operations in USA15 will be available at the beginning of 2018. Figure 16.2 represents a preliminary top-level chart of the tasks to be completed by Technical Coordination before TDAQ can start the installation of the hardware in USA15. It assumes a worst-case scenario, in which the renovation of part of USA15 completes in approximately one year from the end of the Run 3 operations. It would limit the time for installation and commis-
sioning of the TDAQ systems, in particular for the Level-0 Trigger modular electronics in the tall ATCA racks, to 18 months. Technical Coordination is studying the possibility of advance in Long Shutdown 2 some of the tasks listed in Fig. 16.2, which would provide additional schedule float for installation, integration and commissioning.

<figure><img src="image_258.png" /><figcaption>Figure 16.2: Initial planning by Technical Coordination for work packages and tasks related to the installation of racks and modular electronics in USA15. This preliminary chart assumes a worst case scenario where all the tasks will be executed at the beginning of the Long Shutdown 3 shutdown, and no access to the detector systems is allowed for approximately one year.</figcaption></figure>

## 16.2 Laboratories and Facilities

Logistics at CERN will allow the TDAQ Phase-II Upgrade Project to conduct system integration tests in different laboratories and facilities. Two main laboratory spaces are foreseen for integration and validation of hardware components in ATCA format (Section 16.2.1) and for commodity server units (Section 16.2.2).

### 16.2.1 Building 4 surface testing facility

A new laboratory space is being refurbished under the responsibility of ATLAS Technical Coordination (see Figure 16.3). The laboratory is initially meant to be a surface test facility for the integration of the L1Calo Phase-I upgrade components, i.e. FEXs modules, RODs and HUBs. The initial configuration of the laboratory foresees 4 racks for a total of 50kW power distribution and proper water cooling/ventilation.

### 16.2.2 Lab4 Testbed Infrastructure

Figure 16.4 shows the rack layout in the Lab4 facility on Bldg. 4 in the Meyrin site. The laboratory is entirely dedicated to TDAQ developments: currently 16 racks are being equip-
<figure><img src="image_259.png" /><figcaption>Figure 16.3: Floor view in CERN Bldg. 4 and the 20 m² lab space for Phase-I L1Calo integration tests</figcaption></figure>

ped for testbeds of DAQ and EF tests. The facility may be expanded up to 24 racks, and it is being considered as an option for system integration and vertical slice tests of those TDAQ Phase-II components based on rack mountable commodity servers.

### 16.2.3 TDAQ Maintenance Facility

The Building 4 laboratory is a potential candidate to host the integration tests of all the L0 Phase-II sub-systems before installation and commissioning in USA15. The lab space should allow to install up to 4-5 racks, each housing 2 ATCA shelves.

After completion of the installation in USA15 the space will be converted to a Slice Test and Maintenance Facility (TMF) with minimal required setup for each system and sub-system. The size of each slice will be of a size sufficient to carry all the required sub-system tests, probably shelf-sized. The facility will be maintained alive for the duration of the HL-LHC operations to run long term tests, to develop and validate firmware and software before their deployment in ATLAS - see Section 16.3 and, ultimately, to troubleshoot hardware modules.

### 16.2.4 Safety

ATCA cooling systems, both in the horizontal and vertical cooling configuration, are extremely noisy. CERN HSE group, in collaboration with the EP-ESE group and ATLAS Tech-
<figure><img src="image_260.png" /><figcaption>Figure 16.4: Current Lab4 rack configuration for DAQ/EF testbeds [16.2].</figcaption></figure>

nical Coordination has conducted during 2016-2017 measurements in USA15 with racks equipped with two fully loaded shelves [16.3]. Measurements have been made with fans working in two different configurations: 75% of the maximum speed and full speed. Measurements in excess of 85 db(A) were recorded, above the safety level.

At the TMF, worse conditions are expected. Safety precautions become necessary and co-ordinated with the ATLAS Technical Coordination groups, as similar precautions will be needed also in USA15:

* Isolation walls to reduce the acoustic level noise and the health risks to the users need to be installed

* Access will be regulated and allowed for a limited time only to personnel who has followed proper safety training

* Wearing personal protection equipment will be required

* Yearly medical visits will be mandatory.
## 16.3 Production Firmware Deployment

In most of the TDAQ Phase-II sub-systems the level of complexity of firmware will increase significantly cf. to what experienced by the TDAQ for the Run 2 hardware or to what projected for the Phase-I upgrades. Firmware complexity is mainly driven by the need of implementing and integrating in FPGAs more sophisticated algorithms that reproduce the current offline performance. This is the main mitigation measure against the increased pileup.

Specific project management regulation shall be defined to ensure firmware is properly delivered well in advance for commissioning on the surface and in USA15.

* Firmware shall be considered a CORE deliverable alike hardware components. A zero value will be assigned to firmware deliverable.

* Institutional responsibilities will be accounted for in the MoU document.

* Tracking of the Firmware developments shall be possible through the definition of milestones throughout the lifetime of the sub-system.

* Firmware design and implementation shall be reviewed in dedicated reviews, to be carried along the main hardware reviews, i.e. Specification, Preliminary (PDRs), Final (FDR) and Production Readiness (PRR) Reviews. The tight coupling between Firmware and Hardware reviews is required to both have a firmware development which allows to do all tests required at the time of the hardware review and to force similar time profiles for effort in hardware and firmware, aimed to reach a high level of overall readiness at the time of commissioning.

* No sub-system component shall be approved for progressing along the hardware design milestones without a full validation of all the firmware functionality required for that specific milestone, see Section 19.2.

It is imperative to establish very clear and rigorous procedures on firmware management and revision control. This is clear for example in the Global Trigger sub-system, because of its complexity in terms of architecture, and in terms of algorithmic firmware integration on both the MUX and GEP modules. About 20 firmware working packages have been identified by the level-3 managers, and, responsibilities are shared among several groups in different countries.

* The TMF infrastructure will be used as validation tool for any firmware block upgrade before it is deployed on the final boards and used during operations on the hardware in USA15.

* For all the Level-0 sub-systems, a Production Firmware Deployment system will be designed and installed in TMF permanently. As an example, for Global, Production Firmware modules (PFM), see Section 9.4.6, will be designed with same hardware devices as the final MUX and GEP blades.
- Each firmware block, developed at a remote site, will need to be commissioned and validated through extensive integration system tests in the production firmware deployment system.

- Only after validation through the Production Firmware Deployment system, firmware builds integrating the validated blocks will be uploaded on the USA15 modules and final commissioning tests will be conducted in USA15. The same policy shall be adopted on pre-production boards for all the other TDAQ sub-systems.

The aforementioned guidelines will be adopted as general policy for firmware and software maintenance for the duration of the operations in the HL-LHC era, as it is currently being done with the HLT testbeds in Lab4.

## 16.4 Validation and Initial System Integration at CERN of custom Hardware sub-systems

### 16.4.1 Quality Assurance and Quality Control at the production sites

In several sub-systems multiple production sites are being planned. Each site will implement a local test facility to fully characterise their production share. By the time of the sub-systems' PRR a Quality Assurance Policy and Quality Control Procedures should be established and will be included in the documentation to be reviewed by the PRR panel.

The main objective of the QA/QC documentation is to ensure that each production site conforms to the agreed policies for what concerns the setup, infrastructure, material handling and test sequences for the characterisation of the production modules.

QC procedures shall include:

* Visual inspection of the hardware delivered by assembly sub-contractors

* Basic functionality tests, e.g. powering sequence, configuration and monitoring

* Extensive behavioural characterisation of the hardware core and of its input/output interfaces, e.g. BER tests or eye-diagram verification of conformity to ATLAS-wide approved standards like IEEE 802.3bs or similar

* Predetermined test-vectors' validation to demonstrate functionality, or to test fault conditions

* Burn-in stress testing to minimise infant-mortality effects in the early commissioning phase
### 16.4.2 Acceptance validation tests at CERN

Acceptance tests, regulated by the same QA/QC policy, shall be repeated in the TMF facility once the hardware is delivered at CERN. The objective of these tests is to ensure reproducibility of the results, reliability of the testing procedures executed at the production sites, and of the deliverables' handling (including storage and transport). In addition, validation tests at receiving may need to include also system integration procedures see Section 16.4.3, in particular for sub-systems whose functionality rely on critical communication between modules.

### 16.4.3 Level-0 Trigger Sub-system Integration Tests

Before installation in USA15 each sub-system and each component in a sub-system shall undergo system integration tests. The tests are required to verify and validate communication protocols, interfaces and algorithms that require communication between modules in the system. The general strategy is to define a minimal required setup that reproduce a vertical slice of the sub-system. Such a vertical slice shall be built with production first articles and shall be kept available for the entire duration of the production, installation, commissioning and later for testing further developments of the L0 algorithms. Each production module will be tested through the vertical slice validation with pre-programmed test vector patterns that validate the module's functionality.

#### Phase-I FEXs

Legacy L0Calo processors (eFEX, jFEX, gFEX) will undergo at most firmware upgrades. The modules will be fully retested on surface, and system integration validation shall be required for the part of the systems which has interfaces to new LAr or Tile pre-processors and new optical fibres.

#### Forward FEXs

fFEX production is made only of a few modules in a single ATCA shelf. System integration on the surface shall be made with output interfaces to a dedicated readout unit and connections to the required Global MUXs modules.

#### L0Muon Sector Logic and MDT Trigger Processors

The vertical slice setup shall be composed by one fully loaded SL ATCA shelf, one MDT Trigger Processor shelf and with MUCTPI, CTP, FELIX first articles or pre-production units.
## Global

Global is a challenging project based on a design that requires integration of several complex algorithms in a single FPGA device. Different stages to validate the system integration are foreseen:

* Firmware will be validated through the PFM modules (Sections 9.4.6, 16.3) in the TFM facility

* A standalone vertical slice with two fully loaded ATCA shelves, for MUXs modules and for GEPs' will be used to validate the time-multiplexing architecture.

* Integrated setups with a minimal set of L0Calo processors, and with calorimeter pre-processors will validate the high-speed communication and the reconstruction algorithms of those

### 16.4.4 HTT specific Integration and Initial Commissioning

Integration of the HTT system before the installation of the production modules in USA15 shall be realised through a vertical slice of the system, corresponding to two tracking units described in Section 13.1. The commissioning of the HTT slice will also require enough EFPU nodes to feed tracking requests at the expected rate.

The HTT integration will use:

* a group of EF nodes requesting tracking for HL-LHC events from offline Monte-Carlo simulations.

* two HTTIF modules routing the simulated tracking data to the HTT processors

* a fully loaded ATCA shelf with 12 AMTP blades

* a partially loaded ATCA shelf with 2 SSTP blades

The HTT described here is 2% of the overall system. It will become available with pre-production. An estimate for the required EF nodes is also 2% of the overall EF. Two racks will be needed to contain the above HTT and EF equipment.

Track reconstruction performance and latency shall be among the validation parameters to be measured during the integration tests, before final acceptance and installation in USA15.

### 16.4.5 DAQ Readout Sub-system: FELIX and Data Handlers

System integration tests will be organised for each detector system, as the FELIX I/O cards may require detector-specific firmware blocks, and data-handlers certainly require detector-specific software. Detector systems are responsible for the organisation of these tests and TDAQ will collaborate and support to ensure a full validation of their readout.
In general terms, no large system integration of the FELIX production I/O cards is planned. Validation tests will be carried at the level of individual readout servers after they will be equipped with the two I/O cards. In general a minimal vertical slice to validate system integration will include:

* one or more detector front-end electronics boards (up to a fully loaded shelf);

* a FELIX system;

* up to two Data Handler nodes;

* a monitoring node to analyse performance.

An exception is needed for ITk. The detector system will be fully assembled and integrated in the SR1 surface building at Point-1 before being transported and installed in the ATLAS UX15 cavern. The integration and standalone commissioning of the ITk readout system will be done in parallel, using 1/8, i.e. ~ 12.5% of the final required FELIX cards, in SR1 together with the integration of the ITk detectors [16.4].

## 16.5 Installation in USA15

The TDAQ Phase-II installation in USA15 will be preceded by two main tasks: (i) cabling of the fibres from the detector systems to the racks of the trigger processors and of the readout units, (ii) installation and configuration of the TDAQ racks.

### Cabling

All the cables and optical fibres from UX15 are in the scope of the detector system projects, while all optical fibres and optical plants internal to TDAQ are within this scope. Technical Coordination personnel will lay down and route all the cables up to the final destination point.

A cabling database will record each cables, the source and receiving terminal points, and its final length after connectorisation. Measurements of cable lengths, i.e. propagation delays will be conducted in coordination between the detector systems, Technical Coordination and TDAQ for all the cases where latency is relevant.

### USA15 Rack Layout

USA15 racks are distributed in several rows over the two floors, as shown in Fig. 16.5, extracted from the Rack Wizard tool on Glance [16.5].

The front rows, shown in both Figs 16.5a, 16.5b below the thick blue line that represent the noise isolation wall, house a total 67 racks in the two floors of USA15. Most of these
<figure><img src="image_261.png" /><figcaption>(b) Level-2 (bottom) floor of USA15</figcaption></figure>

<caption>Figure 16.5: Current layout and rack configuration of the USA15 floors. Encircled in red the possible locations of the Level-0 Trigger electronics. In blue the noise-cutting wall to isolate acoustically the USA15 area with the ATCA racks</caption>
racks will be available in Phase-II for allocating the L0 Trigger sub-systems. As the readout electronics of many detectors will be decommissioned during Long Shutdown 3, most of the racks in the back rows will be reused for installing detector FELIX and Data Handler nodes.

The exact configuration of the racks is under discussion with Technical Coordination and all the ATLAS detector systems and it will be fully specified in the Installation and Commissioning Plan document.

## TDAQ sub-systems' configuration in USA15

Table 16.2 summarises the numbers of ATCA racks allocated in USA15 to all the TDAQ Trigger sub-systems. A total of 65 racks are required: 9 for the L0 Trigger system, 27 for the DAQ Readout nodes, and 28 racks for the HTT sub-system. A first estimate of the maximum power consumption per rack and the total power consumption of each TDAQ sub-system racks is given in the last four columns. The table indicates a range of power consumption: a current best estimate (CBE), based on an approximate bottom-up analysis of the components on a board, and a maximum possible value (MPV) that takes into account possible uncertainties on those estimates and margins assigned. The total power consumptions of those 61 racks is estimated to be between approximately 670 and 820 kW.

<table><caption>Table 16.2: Summary table with number of shelves, server units and racks in USA15 for the TDAQ system. First estimates of maximum power consumption per rack and the total of power consumption of the TDAQ sub-systems. Current Best Estimates (CBE) and Maximum Possible Values (MPV), which take into account known uncertainties and margins to add</caption><thead><tr><th rowspan="2">Sub-system</th><th rowspan="2">No. of ATCA<br>shelves or<br>commodity servers</th><th rowspan="2">No. of<br>racks</th><th colspan="2">Max. Power<br>per rack [kW]</th><th colspan="2">Total Power/system<br>/system [kW]</th></tr><tr><th>CBE</th><th>MPV</th><th>CBE</th><th>MPV</th></tr></thead><tbody><tr><td>Level-0 Trigger System</td><td>25</td><td>9</td><td></td><td></td><td></td><td></td></tr><tr><td>L0Calo</td><td>4</td><td></td><td>13</td><td>16</td><td>18</td><td>25</td></tr><tr><td>L0Muon</td><td>15</td><td></td><td>13</td><td>15</td><td>56</td><td>68</td></tr><tr><td>Global Trigger</td><td>4</td><td></td><td>13</td><td>16</td><td>24</td><td>29</td></tr><tr><td>CTP,MUCTPI</td><td>2</td><td></td><td>5</td><td>6</td><td>5</td><td>6</td></tr><tr><td>HTT</td><td>56</td><td>28</td><td>11</td><td>16</td><td>314</td><td>392</td></tr><tr><td>Readout</td><td>812</td><td>27</td><td>11</td><td>13</td><td>245</td><td>298</td></tr><tr><td><strong>TOTAL</strong></td><td></td><td><strong>65</strong></td><td></td><td></td><td><strong>662</strong></td><td><strong>818</strong></td></tr></tbody></table>

Such a layout assumes each Level-0 Trigger System rack to be configured with three (3) ATCA shelves, as shown in Figure 16.6, and a shelf with 12 user-blades, a shelf manager and, in some cases, an extra-switch. The HTT sub-system uses standard racks with two (2) ATCA shelves.
<figure><img src="image_262.png" /><figcaption>Figure 16.6: Rack layout with three ATCA shelves and three heat exchangers.</figcaption></figure>

**L0Muon sub-system configuration** A possible arrangement of racks and configuration is shown, as example, in Tables 16.3 and 16.4 for the L0Muon sub-system.

<table><caption>Table 16.3: Example of Racks' configuration in the L0Muon sub-system. TP stands for Trigger Processor.</caption><thead><tr><th></th><th>SL Shelves</th><th>MDT TP Shelves</th><th>NSW TP Shelves</th><th colspan="2">Rack Power</th></tr><tr><th></th><th></th><th></th><th></th><th>CBE [kW]</th><th>MPV [kW]</th></tr></thead><tbody><tr><td>L0Muon Endcap</td><td></td><td></td><td></td><td>34</td><td>41</td></tr><tr><td>Rack #1</td><td>2</td><td>0</td><td>1</td><td>11</td><td>13</td></tr><tr><td>Rack #2</td><td>0</td><td>2 Large</td><td>1</td><td>13</td><td>15</td></tr><tr><td>Rack #3</td><td>2</td><td>1 Small</td><td>0</td><td>10</td><td>13</td></tr><tr><td>L0Muon Barrel</td><td></td><td></td><td></td><td>22</td><td>27</td></tr><tr><td>Rack #1</td><td>2 Large</td><td>1 Small</td><td>0</td><td>10</td><td>13</td></tr><tr><td>Rack #2</td><td>1 Small</td><td>2 Large</td><td>0</td><td>12</td><td>14</td></tr><tr><td>TOTAL</td><td></td><td></td><td></td><td>56</td><td>68</td></tr></tbody></table>
<table><caption>Table 16.4: Possible configuration of the ATCA shelves in the L0Muon sub-system</caption><thead><tr><th></th><th>No. of blades<br>or shelves</th><th>Power/Unit<br>CBE [kW]</th><th>Unc.<br>[%]</th><th>Margin<br>[%]</th><th>Power[kW] /Unit<br>MPV [kW]</th></tr></thead><tbody><tr><td>L0Muon Endcap/TGC Shelf</td><td></td><td>3.5</td><td></td><td></td><td>4.3</td></tr><tr><td>SL</td><td>12</td><td>0.20</td><td>10</td><td>10</td><td>0.24</td></tr><tr><td>Shelf Manager</td><td>1</td><td>0.15</td><td>10</td><td>10</td><td>0.36</td></tr><tr><td>Fan Tray</td><td>1</td><td>1.00</td><td>10</td><td>10</td><td>1.21</td></tr><tr><td>L0Muon Endcap/NSW TP Shelf</td><td></td><td>3.9</td><td></td><td></td><td>4.7</td></tr><tr><td>NSW TP</td><td>8</td><td>0.35</td><td>10</td><td>10</td><td>0.42</td></tr><tr><td>Shelf Manager</td><td>1</td><td>0.15</td><td>10</td><td>10</td><td>0.18</td></tr><tr><td>Fan Tray</td><td>1</td><td>1.00</td><td>10</td><td>10</td><td>1.21</td></tr><tr><td>L0Muon Barrel/RPC Large Shelf</td><td></td><td>3.5</td><td></td><td></td><td>4.3</td></tr><tr><td>SL</td><td>12</td><td>0.20</td><td>10</td><td>10</td><td>0.24</td></tr><tr><td>Shelf Manager</td><td>1</td><td>0.15</td><td>10</td><td>10</td><td>0.18</td></tr><tr><td>Fan Tray</td><td>1</td><td>1.00</td><td>10</td><td>10</td><td>1.21</td></tr><tr><td>L0Muon Barrel/RPC Small Shelf</td><td></td><td>2.7</td><td></td><td></td><td>3.3</td></tr><tr><td>SL</td><td>8</td><td>0.20</td><td>10</td><td>10</td><td>0.24</td></tr><tr><td>Shelf Manager</td><td>1</td><td>0.15</td><td>10</td><td>10</td><td>0.18</td></tr><tr><td>Fan Tray</td><td>1</td><td>1.00</td><td>10</td><td>10</td><td>1.21</td></tr><tr><td>L0Muon /MDT Large Shelf</td><td></td><td>4.4</td><td></td><td></td><td>5.3</td></tr><tr><td>MDT TP</td><td>12</td><td>0.20</td><td>10</td><td>10</td><td>0.24</td></tr><tr><td>MDT TP RTM</td><td>12</td><td>0.20</td><td>10</td><td>10</td><td>0.24</td></tr><tr><td>Shelf Manager</td><td>1</td><td>0.15</td><td>10</td><td>10</td><td>0.36</td></tr><tr><td>Fan Tray</td><td>1</td><td>1.00</td><td>10</td><td>10</td><td>1.21</td></tr><tr><td>L0Muon /MDT Small Shelf</td><td></td><td>3.3</td><td></td><td></td><td>4.0</td></tr><tr><td>MDT TP</td><td>8</td><td>0.20</td><td>10</td><td>10</td><td>0.24</td></tr><tr><td>MDT TP RTM</td><td>8</td><td>0.20</td><td>10</td><td>10</td><td>0.24</td></tr><tr><td>Shelf Manager</td><td>1</td><td>0.15</td><td>10</td><td>10</td><td>0.36</td></tr><tr><td>Fan Tray</td><td>1</td><td>1.00</td><td>10</td><td>10</td><td>1.21</td></tr></tbody></table>
## 16.6 Integration with the Detector Systems and Early Commissioning

The main commissioning phase begins when all infrastructure in USA15 is installed and connected (including ATCA shelves, blades, fibre plant, TTC and readout fibres). Integration tests executed in the TMF laboratory will be repeated to assess the correct functions of the individual hardware components. Joint commissioning tests with the detector systems or among different sub-systems will be conducted to validate the correct interfaces and proper mapping between modules.

A common plan for schedule of the detectors' installation, of the commissioning of their electronics, and on their needs of early availability of the relevant TDAQ components will need to be defined early 2018, through dedicated meetings with Upgrade and Technical Coordination.

### Detector system calibrations

As part of the initial commissioning, the detectors will be organised in independent partitions and calibration scans will be executed on each partition. Calibration runs will obviously allow TDAQ and the ATLAS detectors to commission all the components of Readout and Dataflow sub-systems associated to a particular detector partition.

They will also allow commissioning of L0 Trigger elements. This is particularly true for the calibrations of the LAr and Tile calorimeters. Precision calibration loops allow determining the energy scale at the calorimeter's cell-level, and, consequently, of the L0Calo - i.e. of the FEXs - modules and of the Global Trigger processors.

### Early commissioning with cosmics and with beams

Final validation of the systems in USA15 will use cosmic rays triggers and high-rate stress tests to probe operational regions otherwise untested with calibrations. In general, cosmics data are sufficient to measure individual signals and overall system timings to the bunch-crossing levels. However, final tuning and verification of the trigger and readout chain will require LHC beam data, for example to achieve the ultimate timing performance.

Finally, once the commissioning strategies aforementioned and the parameters of the systems are well established and tuned, early data enable the study of the system performance and of the reconstruction algorithms deployed in the trigger processors. Rates and efficiencies of trigger object candidates can be studied in detail for example by tag and probe methods, and comparing online vs. offline object reconstructions.
## 16.7 Installation and Commissioning Plans of DAQ/EF components in SDX1

As outlined in Section 16.1.1 SDX1 is the surface building near the personnel access shaft. It hosts the underground access infrastructure, the TDAQ room and the Uninterruptible Power Supply (UPS) system. Technical Coordination is planning a renovation of the infrastructure to provide additional cooling capabilities to the ATLAS detector systems in U15 and U17, detaching the circuits dedicated to the ventilation of the surface building and the rack cooling in the TDAQ room, from the existing chilled and mixed water production stations, providing new cooling capability also to the racks that hosts the EFPU farms and the DAQ storage nodes.

The TDAQ room is laid out on two floors, equipped with approximately 100 standard racks for server applications, as shown in Figure 16.7:

A typical configuration of the SDX1 rack is shown in Figure 16.8. The front and rear views of the racks in the figure show 40 motherboards installed and the corresponding network interconnections.

### Installation and Integration of the EFPU and Dataflow nodes

The installation sequence and all the steps to integrate PC servers in SDX1 follow the well defined procedures used for the Run 1 and Run 2 operations, for example during the rolling replacement of the HLT PC servers.

For the processing nodes the procedure takes into account that the upgraded EFPU integrates to pre-existing nodes. It is summarised in the following steps:

* Units installed and cabled in the SDX1 racks

* A sys-admin burn-in tests of 24 hours

* A 24 hour burn-in with DAQ common software running

* A vertical slice tests simulating dataflow between a few racks commissions the nodes and the associated network

* Vertical slices tests gradually scaling up to the full system

Storage nodes in the Dataflow sub-system undergo similar procedures, but as storage will be completely replaced with new devices, a configuration step is required:

* Units installed and cabled in the SDX1 racks

* A sys-admin burn-in tests of 24 hours

* Configuration of the Storage handlers

* Vertical slice tests together with the EFPU nodes, scaling gradually to the full system.
<figure><img src="image_263.png" /><figcaption>Figure 16.7: Current layout and rack configuration of the SDX1 Level-1 (top) and Level-2 (bottom) floors</figcaption></figure>
<figure><img src="image_264.png" /><figcaption>Figure 16.8: Photographs of a typical event processing rack configuration in SDX1. The left photographs shows the front side of the servers, while right one the rear network interconnections to the rack-level switches.</figcaption></figure>
## Software Integration

Integration of the software and installation or upgrades of the operating systems of all the TDAQ nodes follow also a standard procedure:

* Development and pre-commissioning will be conducted in dedicated test nodes in the Lab4 facility

* On the SDX1 nodes Technical run periods are foreseen:

A. Technical runs will be conducted every 6-8 weeks for approximately one week.

B. During a technical run all the DAQ and EF resources will be available only for TDAQ commissioning, i.e. there will be no involvement of detector systems, and all the nodes will be run in emulation mode.

C. Software commissioning will be run on small partition initially to test the data flow.

D. Tests will be repeated scaling up the dimensions of a partition gradually to the full system.

## References

[16.1] ATLAS Collaboration, *The ATLAS Experiment at the CERN Large Hadron Collider*, JINST 3 (2008) S08003.

[16.2] TDAQ TestBed, (online).
https://twiki.cern.ch/twiki/bin/view/Atlas/Lab4Testbed.

[16.3] C. Bortolin, *ATCA Cooling Report*, Tech. Rep. ATU-GE-ER-0015 v.1 / EDMS Doc. No 1609794. https://edms.cern.ch/document/1609794/1.

[16.4] ATLAS Collaboration, *Technical Design Report for the ATLAS Inner Tracker Strip Detector*, Tech. Rep. CERN-LHCC-2017-005, CERN, Geneva, Apr, 2017.
https://cds.cern.ch/record/2257755.

[16.5] ATLS RackWizard, (online).
https://glance.cern.ch/atlas/rackwizard/menu.
<figure><img src="image_265.png" /></figure>
# Part III

## Project Management and Organisation
<figure><img src="image_266.png" /></figure>
# 17 Project Management and Organisation

This chapter focuses on the TDAQ Phase-II Upgrade Project (UPR) organisational structure and summarises the basic project governance framework in which the project management plan is executed. The TDAQ Phase-II upgrades are developed and executed by a group of ATLAS Institutions that commit formally through a MoU document to design, construct, install and commission new components in the ATLAS Trigger and DAQ system. As such, the upgrade activities are embedded in the TDAQ system organisation, and ultimately integrated in its operation model. However, to control and administer the activities and the tasks in an effective and timely manner, the Phase-II upgrades are operated through a dedicated organisation, the TDAQ Phase-II UPR, with a relatively autonomous governance and management structure, well defined responsibilities and reporting lines to the TDAQ system and, ultimately, to ATLAS.

The following sections describe the relationship between the TDAQ UPR, TDAQ, and ATLAS management entities in detail. Section 17.1 is an overview of the ATLAS organisation as it relates to the management of the upgrade activities. The structure and operation of TDAQ is summarised in Section 17.2. Finally, Section 17.3 details the structure, the governance and the management of the UPR at the top three levels of the organisation.

## 17.1 Overview of the ATLAS Upgrade Organisation

The overall organisation, the management and the integration of the ATLAS Phase-II upgrade projects is completely defined in an internal report, the ATLAS Upgrade Organisation document [17.1], released in December 2016. This section summarises briefly the essential points needed to understand the relationship of the UPR with ATLAS management and with the ATLAS detector system with which the UPR project is integrated.

For reference, a simplified view of the ATLAS Upgrade organisation is provided in Figure 17.1, extracted from [17.2] and updated to reflect the new ATLAS management organisation in place since March 2017.
<figure><img src="image_267.png" /><figcaption>Figure 17.1: A schematic view of the ATLAS Upgrade organisation.</figcaption></figure>

### 17.1.1 ATLAS management

The overall execution of ATLAS is the responsibility of the ATLAS Management, led by the Spokesperson, two Deputy Spokespersons, the Technical Coordinator, the Resource Coordinator, and the Upgrade Coordinator [17.3]. The Spokesperson and her/his Deputies have the overall responsibility to globally manage all the aspects of the ATLAS experiment. The Spokesperson represents ATLAS with respect to CERN, funding agencies and other outside bodies.

### 17.1.2 Executive Board

The Executive Board (EB) is the highest-level body for directing the execution of the ATLAS project and for direct communication between the ATLAS management and the systems. It is chaired by the Spokesperson with the Technical Coordinator as deputy chair. The EB includes: the ATLAS management, the System Project Leaders (Inner Detector, Liquid Argon and Tile Calorimeters, Muon, TDAQ, ITk Project, Forward Detectors), the Activity Coordinators (Run, Trigger, Data Preparation, Computing, Physics), and the Publication Committee Chair. Ex-officio members are the ATLAS Collaboration Board (CB) Chair, and
the CB Deputy-Chair. In addition, the Spokesperson appoints additional members-at-large to ensure an overall balance and competence in the EB.

### 17.1.3 Upgrade Coordinator

The overall steering and monitoring of the upgrade activities are delegated to the Upgrade Steering Committee (USC), which is a sub-committee of the EB, with an extended membership. The USC is chaired by the Upgrade Coordinator (UC), who is elected by the Collaboration Board (CB) with a renewable two-year appointment.

The UC follows each phase of the UPRs, from their initial R&D phase, through construction and into the system integration phase. The UC also appoints committees to assist with major decisions that cut across upgrade projects, or that have global impacts on the performance of any given upgrade project. The Spokesperson and the UC work closely together in preparing upgrade-related interactions with the CERN LHCC and UCG review committees.

### 17.1.4 Upgrade Steering Committee

The UC chairs the Upgrade Steering Committee (USC), the committee which is formally responsible for the oversight of the entire Upgrade programme. The USC operates in close collaboration with the TC and the TC PO, which links to the technical coordination and operation of the entire experiment.

The USC consists of: the UC as USC Chair, and a Deputy Chair designated by the TC, ATLAS management, the Upgrade Project Leaders (UPLs), the conveners of the Sub-committees, the Upgrade Physics Conveners, the Upgrade Software Coordinators, the Electronics Coordinator, the Review Office Chair, the Run Coordinator, the Computing Coordinator, the Physics Coordinator, and the Project Leaders of the main ATLAS systems.

The involvement of the System Project Leaders within the main decision-making body should ensure appropriate interaction with any corresponding UPRs affecting that System.

### 17.1.5 Upgrade Advisory Board

An Upgrade Advisory Board (UAB) is formed of ATLAS members who act as upgrade representatives of the communities supported by each Funding Agency (FA) contributing to ATLAS. The UAB is chaired by the RC and discusses primarily the availability of resources, and planning across the full spectrum of upgrades, including financial and workforce planning. The UAB member for a given FA may be either the National Contact Physicist (NCP) or a separate ATLAS collaborator representing the community supported by the FA.
### 17.1.6 Interactions with External Committees and Organisations

Formal reporting of ATLAS status and planning to the CERN Directorate, to the LHCC and to the RRB is among the responsibilities of the ATLAS Spokesperson. In the case of discussions with the LHCC related to upgrades, the Spokesperson may delegate or share the responsibility with the UC, Contacts with individual ATLAS FAs are primarily the responsibility of the NCPs and/or handled by the UAB member for that FA. However, the Spokesperson shall handle directly discussions with FAs and CERN Management whenever required, with the presence of the NCP and/or UAB member if possible.

### 17.1.7 Upgrade Projects

The upgrades of ATLAS are formally organised through Upgrade Projects. An Upgrade Project (UPR) comprises in general the plans and the activities of one of the existing ATLAS systems listed in Section 17.1.2. An UPR may be structured, managed and operated by an autonomous organisation, but its activities and plans should be developed in consultation and agreement with the parent system's management.

#### UPR approval steps

UPRs are approved in several stages. When a project has reached the level of maturity agreed by the parent system, with the UC and the USC, the project passes first through initial stages of approvals as shown in Fig. 17.2: (i) an Initial Design Review (IDR) verifies that the technical requirements have been identified and that the initial design meets those requirements; (ii) a kick-off meeting is held by the Institute Board (IB) of the parent system to collect expression of interests from any institutes in ATLAS that wish to participate; (iii) based on the IDR committee's report and a recommendation by the USC members, the UC requests approval of the project by the EB; (iv) the EB approval decision is endorsed by the ATLAS Collaboration Board (CB).

<figure><img src="image_268.png" /><figcaption>Figure 17.2: Internal approval process of an Upgrade Project (UPR)</figcaption></figure>

#### UPR Organisation and relationship to the parent system

With the exception of ITk, each UPR is assisted by the parent system's Institute Board (IB). The IB monitors the activities of the UPR, in line with the UPR's MoU. The IB takes decisions
in matters of resources and policy, and endorses technical decisions pertaining to the UPR recommended by the UPR.

## Upgrade Project Leader

The UPR activities are coordinated by an Upgrade Project Leader (UPL), initially appointed by the System Project Leader and endorsed by the System IB until the approval of the project by the CERN RB. After approval the UPL is elected by the System IB through the standard ATLAS election rules. The UPL reports on progress and achievements to the USC, and to the parent System's management body.

## UPR Management

The UPL may propose a new management structure in consultation with the parent System's Project Leader, the UC and the System's IB Chair. Modifications to the UPR management structure shall be agreed with the UC, and formally endorsed by the System's IB.

## Completion of the UPR

After completion of the integration of the UPR into ATLAS, the UPR organisation dissolves and the parent system takes over the exploitation and maintenance of the delivered detector elements.

## 17.2 Organisation and management of the TDAQ systems and the TDAQ Upgrades

The TDAQ system covers the design, development, construction, installation, commissioning and operation of all hardware and software components that perform the selection and acquisition of events from the ATLAS detector. The system organisation and its management structure is described extensively in Ref. [17.4] and the essential points are summarised in this Section. The following sections describe the continuing roles of the TDAQ Institute Board (17.2.1), the TDAQ Management Team (17.2.2), the TDAQ Steering Group (17.2.3), and how resources are controlled. In Section 17.2.5 the relationship of the Upgrade Project to parent system TDAQ system is described. The upgrade organisation and management plan is presented in Section 17.3.

The organigram of the ATLAS TDAQ system and its relationship to ATLAS management are shown in Figure 17.3.
<figure><img src="image_269.png" /><figcaption>Figure 17.3: Organisation and management structure of the ATLAS TDAQ system and relationships to ATLAS.</figcaption></figure>

### 17.2.1 TDAQ Institutional Board

The TDAQ Institute Board (TDIB) is the policy and decision making body of the ATLAS TDAQ system. It is subordinate to the ATLAS Collaboration Board. The TDIB takes decisions on sharing of resources and responsibilities. Major issues affecting the overall performance of ATLAS must be brought forward to the collaboration as a whole for decision in the ATLAS Collaboration Board. The institutions are the source of money and effort [17.5], and therefore all major questions involving sharing of responsibilities and contribution of resources have to be agreed upon by the TDIB. In major matters concerning resources the Institute Board shall consult the ATLAS Resources Coordinator.

Members of the TDIB are:

* The TDIB Chair

* A representative per institution contributing to the TDAQ system operations

* The members of the regular TDAQ Steering Group (ex-officio) - See Section 17.2.3

* The Chair(s) of sub-committee initiated by TDIB (ex-officio)

* The ATLAS Management (ex-officio)
## 17.2.2 TDAQ Management Team

The TDAQ Management Team (TDMT) is responsible to the TDIB for the execution of the TDAQ system. The TDMT is formed of the TDAQ Project Leader and her/his deputies. The Project Leader is elected by the TDIB, after nomination of candidates and due consultation within the TDAQ community, for a term of two years, and he/she can be re-elected with a 2/3 majority [17.5]. The Project Leader will appoint at least two deputies to share part of her/his responsibilities, and concentrate in complementary areas of TDAQ. These appointments will be approved by the TDIB. As indicated in Figure 17.3, at the time of drafting this TDR, the Project Leader has two appointed deputies, one for L1 Trigger operations, the other for the operation of the DAQ, HLT and FTK activities.

## 17.2.3 TDAQ Steering Group

The TDAQ Steering Group (TDSG) encompasses all the activities of the TDAQ system, and is composed of the people leading those efforts. The TDSG is the main body where executive decisions are taken on technical matters and it proposes recommendations to the TDIB on matters of sharing resources and responsibilities. The TDSG is chaired by the TDAQ Project Leader or by a delegated member of the TDMT. The frequency of TDSG meetings shall be sufficiently high to ensure the effective leadership of TDAQ system execution. The TDMT shall routinely consult the TDSG whenever significant issues arise.

The following itemised list defines TDSG membership:

* The members of the TDMT

* The leaders of TDAQ activities

* The TDIB Chair (ex-officio)

* The Chairs of sub-committees initiated by the TDIB

* The members at large chosen to provide balance and competence in the TDSG

* The ATLAS Run coordinators (ex-officio)

* The ATLAS Trigger coordinators<fnref n="1" /> (ex-officio)

* The ATLAS Software & Computing coordinators (ex-officio)

* ATLAS Management (ex-officio)

## 17.2.4 TDAQ Resources

The TDIB Chair and the TDMT are responsible for resources matters, particularly the use of TDAQ funds and the sharing of financial responsibilities. The TDAQ funds include funds for maintenance and operation, including replacements, and non-CORE funds for supporting the ATLAS system. The use of past, current and foreseen TDAQ funds, as well

<footnote n="1">¹ In ATLAS, the Trigger Activity Area has primary responsibility for the Trigger Menu that is used during operations and the algorithmic software that is run in the HLT.</footnote>
as the development of proposals regarding sharing of financial responsibilities need to be brought to the TDAQ Institute Board for approval. The TDIB Chair and the TDMT can decide together to appoint a person to assist in their task on resources matters. The appointed person needs to be endorsed by the TDIB. As most TDAQ M&O funds are provided by the full ATLAS Collaboration as Category A M&O, the ATLAS Resources Coordinator is consulted in all related matters.

### 17.2.5 TDAQ system and TDAQ Upgrade Projects

The TDAQ upgrade tasks will be executed by one more more ATLAS Institutions that commit to constructing, installing and commissioning new hardware and software components. These Institutions are, to a large extent, already members of the TDAQ. However, in compliance with ATLAS rules, any other ATLAS Institutions are eligible to join the UPR project activities and assume responsibilities within. Non-TDAQ Institutions are not required to formally join the system, but are invited to attend the regular TDIB meetings, to participate in the discussions and in the decision making process related to any upgrade activity, including for example the election of the Upgrade Project Leader (UPL).

The oversight and coordination of the ATLAS upgrade programme is detailed in Ref. [17.1] and outlined in Section 17.1. TDAQ upgrade R&D and project tasks form an integral part of the TDAQ system planning and are embedded within the TDAQ system. However, different strategies for the Phase-I and Phase-II upgrades have been employed.

The major construction work in the TDAQ Phase-I upgrade is related to the Level-1 Trigger, and falls under the responsibility of the Level-1 Upgrade Coordinator, appointed by the TDAQ Project Leader as described in the Phase-I TDR [17.6]. This person reports to the ATLAS USC together with the TDAQ Project Leader, and also to the TDAQ Steering Group. The TDAQ Project Leader directly reports to the TDIB on the status of the Phase-I upgrades.

The Phase-II TDAQ upgrade project has an elected Upgrade Project Leader (UPL), a dedicated internal organisation and an autonomous management structure for the duration of the project lifetime. Phase-II resources management is handled directly by the UPL and will be formalised through an independent MoU document. The UPL reports directly to the TDIB on the technical progress in the project and on any key resources aspect. The UPL will consult regularly with the TDAQ Project Leader and keep him or her apprised of any upgrade issues that impact operations, the Phase-I upgrade or which are relevant for the ATLAS Executive Board. An extended TDSG, which includes the regular TDSG members and the UPR management, is the main body where executive decisions are taken on technical matters, and is chaired by the UPL as documented in Section 17.3. After completion, the UPR organisation dissolves and the M&O of the delivered system, hardware, firmware and software is integrated into the overall TDAQ M&O activities. The ATLAS Management shall verify the institutional commitments of the non-TDAQ Institutions that have
joined the UPR after its completion, for example to the long-term M&O programme during
operations in Run 4 and beyond.

## 17.3 TDAQ UPR Organisation and Upgrade Project Management Plan

The current (interim) management structure of the TDAQ HL-LHC UPR is shown in Fig. 17.4. This structure is responsible for directing and managing the UPR activities starting from the ATLAS approval of the UPR.

The UPR is managed through the TDAQ Upgrade Project Leader (UPL) and her/his Management Team (UPMT); this team is responsible for the execution and the control of the tasks and of the resources required to ensure completion of the project activities. The executive decisions are made through the extended TDSG (eTDSG) group, which facilitates communications between the UPR management and the Level-3 managers, and ensures full integration of the UPR in the TDAQ system.

The UPR comprises three systems at Level-2 of the organisation structure: the Level-0 (L0) trigger, the Data Acquisition (DAQ), and the Event Filter (EF) & Performance systems. The organisational structure reflects the main TDAQ operation activities and the project's system-wide deliverables. Each system is organised at Level-3 of the management structure in sub-systems, which are functionally and organisationally independent sub-units corresponding to specific deliverables and tasks.

### 17.3.1 UPR Management Team

The UPMT is responsible for executing and directing all the UPR activities and coordination areas until all deliverables of the project have completed construction, installation and initial commissioning in ATLAS. Individual items will be turned over to the main TDAQ system as they are completed. Upon project completion all deliverables, the UPR is dissolved into the main TDAQ system operations.

**Composition:** The UPL is assisted upon appointment by four Level-2 manager/coordinates, who form together the UPMT:

* Upgrade Project Leader UPL (chair)

* UPR Level-0 Trigger Coordinator (Deputy UPL)

* UPR Data Acquisition Coordinator (Deputy UPL)

* UPR Event Filter & Performance Coordinator (Deputy UPL)

* UPR Project Office Chair
<figure><img src="image_270.png" /><figcaption>Figure 17.4: Organisation and management structure of the TDAQ HL-LHC UPR until CERN approval of the TDR, expected by April 2018. The detailed structure is subject to (minor) changes as the scope of the project finalises in the early phases of the TDR preparation.</figcaption></figure>
**Mandate and Responsibilities:** The UPMT is the central element of the UPR organisation and is responsible for managing the overall project. It monitors the activities and the execution of all the project elements and of all coordination areas, and resolves all the integration aspects across the sub-projects. The responsibilities of the UPMT span all phases of the project, from R&D through detailed design, production, system assembly and integration, installation and commissioning.

## 17.3.2 Upgrade Project Leader (UPL)

**Term:** The UPL is elected by the TDIB for a renewable two-year term. The elections are regulated by the ATLAS and TDAQ system policy.

**Interim UPL (iUPL):** A first interim appointment by the TDAQ Project Leader was endorsed by the TDIB in September 2016 for the duration of the approval process of the UPR, i.e. until the TDR approval by the CERN RB. The expected completion of the term is April 2018.

**Mandate and Responsibilities:** Ultimately responsible for the UPR, and manages, in consultation with the eTDSG, its execution at CERN and with collaborating institutions to ensure that the project is completed within approved cost, schedule, and technical scope.

**Reporting Lines** There are three well defined reporting lines of the UPR management to TDAQ and ATLAS management:

* The UPL represents the TDAQ Phase-II UPR in the TDSG where he/she reports on the status of the project. The UPL and the TDAQ system's Project Leader will have frequent meetings to discuss planning, and technical status and progress.

* The UPL shall report to the TDIB on the status of the project, seek approval as needed on specific decisions proposed after a discussion within the eTDSG, and on any resources and financial matter of the project.

* The UPL shall also report regularly to ATLAS management, and, in particular, to the ATLAS UC through the USC and to the ATLAS RC on technical and resources-related matters, respectively, as outlined in Section 17.1.4.

## 17.3.3 Deputy Upgrade Project Leaders

One or more DUPLs are appointed by the UPL to assist in all matters relating to Project's activities and resources, including the planning, procurement, disposition of institutional
resources, progress reports on the executed activities, liaison with Institutional Representatives, fabrication and QA of TDAQ components, and their timely delivery for installation in ATLAS during LS3.

The DUPLs may be delegated, on a case by case basis, to represent the project to the ATLAS management or in CERN-wide review processes.

**Interim Mandate and Term:** In January 2017, the iUPL appointed the UPR L0 Trigger Coordinator, the DAQ Coordinator and the EF & Performance Coordinator as DUPLs for the whole duration of the interim mandate of the iUPL.

### 17.3.4 Extended TDAQ Steering Group

The Extended TDAQ Steering Group (eTDSG) is an extension of the TDSG (see Section 17.2.3); it is chaired by the UPL or by a delegated member of the UPMT. The eTDSG is the main body where executive decisions are taken on technical matters and it proposes recommendations to the TDIB on matters of sharing resources and responsibilities.

The following are members of the eTDSG:

* UPR Level-3 Coordinators/Managers

* TDAQ Upgrade Project Office chair

* TDAQ Upgrade Project Office Activities' Coordinators (see Section 17.3.8)

* TDIB Chair (ex-officio)

* Members of the TDMT (ex-officio)

* Members of the regular TDSG

* ATLAS Upgrade Coordinator (ex-officio)

* Technical Coordination's and Upgrade Project Office representative (ex-officio)

The UPL shall routinely consult the eTDSG whenever significant issues arise, and organise regular meetings whose frequency shall be sufficiently high to ensure an effective leadership of the UPR execution.

### 17.3.5 UPR Level-0 Trigger Coordinator

The UPR L0 Trigger Coordinator is directly responsible for the definition and the management of all the Trigger deliverables, as identified in the UPR PBS and WBS and related software activities.

He/she will oversee the R&D programme, design, fabrication, assembly and construction, installation and commissioning of the hardware L0 Calorimeter Trigger (forward FEX), the L0 Muon Trigger (RPC and TGC Sector Logic, MDT and NSW Trigger Processors), the
Global Trigger (Aggregating stages, Global Processing Units), and the Central Trigger systems (CTP units, MUCTPI, TTC network distribution and interfaces).

He/she is also expected to manage a UPR Level-0 Trigger Coordination body as specified in Section 17.3.9.

### 17.3.6 UPR DAQ Coordinator

The UPR DAQ Coordinator is directly responsible for the definition and the management of all the DAQ deliverables and associated software activities, as identified in the UPR PBS and WBS specified in Sections 18.4 and 19.3, respectively.

The DAQ Coordination Group will work closely with the existing DAQ/HLT Coordination Group. The two coordination groups will have some common members. This choice is justified by several factors: (i) the limited size of the groups, (ii) the synergies with the existing activities in TDAQ operations and the tasks defined within and for the TDAQ Phase-I upgrades, and, most importantly, (iii) the fact that the DAQ/EF architecture and resources will evolve smoothly from the current Run 2 system through the Phase-I and Phase-II upgrades and the lifetime of the HL-LHC.

### 17.3.7 UPR EF & Performance Coordinator

The EF & Performance Coordinator is directly responsible for the definition and the management of all the EF deliverables, as identified in the UPR PBS and WBS, specified in Sections 18.4 and 19.3 respectively.

He/she will oversee as applicable the R&D programme, design, fabrication, assembly and construction, specification, procurement, installation and commissioning of the Event Filter Processing Units (EFPUs), possibly including Accelerator Processing Units, and of the Hardware-based Tracking for the Trigger (HTT). He/she will also oversee the design and development of the EF software, and of the Physics performance and Event Selection activity

The EF & Performance Coordination Group will work closely with the existing DAQ/HLT Coordination Group and Trigger Coordination Group. These coordination groups may have some members in common. This choice is justified by several factors: (i) the limited size of the groups, (ii) the synergies with the existing M&O activities and TDAQ Phase-I upgrades, and (iii) the fact that the DAQ/EF architecture and resources will evolve smoothly from the current Run 2 system through the Phase-I and Phase-II upgrades and the lifetime of the HL-LHC.
### 17.3.8 TDAQ Upgrade Project Office

The TDAQ Upgrade Project Office assists the UPMT with specific technical aspects of the project, to ensure a smooth integration with the TDAQ system and with ATLAS.

#### UPR Technical Coordinator

The principal responsibility of the UPR Technical Coordinator (UPR-TC) is to act as Technical Engineering Manager and Architect, focusing on global technical aspects of the UPR. The UPR-TC will assist the UPL chairing dedicated meetings of the eTDSG as the need arises. He/she will represent the UPR in the ATLAS UPO and, in general, with ATLAS TC and in discussion with other ATLAS detector systems. He/she will coordinate with the ATLAS Review Office for ATLAS-wide reviews.

#### UPR Project Resource Coordinator and Risk Manager

The principal responsibility is to assist the UPL on the Project Management aspects of the UPR, maintaining and monitoring the CORE costs, schedule and risk register. He/she will assist the UPL chairing dedicated meetings of the eTDSG or dedicated eTDSG sessions as the need arises. He/she will, in consultation with the TDIB, survey and monitor that the available person-power at each institute responsible for any CORE deliverables matches what is required.

#### UPR DCS Coordinator

The principal responsibility of the UPR DCS Coordinator is to guarantee the consistency of the Control, Configuration and Monitoring functions of the different hardware deliverables, and consistency with ATLAS central DCS strategies and plans.

### 17.3.9 Organisation of the TDAQ UPR Systems and Sub-systems

The Level-2 system (Level-0 Trigger, DAQ and EF) coordinators, and the Level-3 sub-system coordinators oversee the design, construction, installation, and commissioning of their sub-systems. Each system and sub-system is organised according to an individual structure proposed by the coordinators in consultation with the UPL. Figures 17.5, 17.6 and 17.7 show the organisation charts for the three systems in the TDAQ UPR.

Each Level-2 system has a coordination body chaired by the System Coordinator, and where each Level-3 sub-system is represented by its coordinator. In addition, the DCS and the Technical Coordinators are ex-officio members of each System coordination group to ensure
coherency across the whole TDAQ UPR of the implementation of the DCS functions in the hardware modular electronics, and to guarantee consistency with ATLAS guidelines regarding installation and integration in both USA15 and SDX1.

Each sub-system is divided into specific activity areas, or tasks, that are mapped onto Level-4 and Level-5 items of the WBS. Coordinator Areas may have one or more conveners.

## Organisation of the Level-0 Trigger System

The Level-0 Trigger system is organised in four sub-systems and several coordination areas, as shown in the organigram in Figure 17.5.

<figure><img src="image_271.png" /><figcaption>Figure 17.5: Organisation of the sub-systems in the Level-0 Trigger system</figcaption></figure>

**Level-0 Calorimeter Trigger** The existing L1Calo group coordinates and organises meetings to discuss topics related to either Operations, Phase-I upgrades, or Phase-II upgrades of this sub-system.

**Level-0 Muon Trigger** Regular meetings are organised by the Level-3 coordinators among the conveners of each activity area, the experts, as well as the colleagues from the Muon Phase-II Upgrade Project to discuss issues related to the muon trigger that are relevant for both upgrade projects.
**Global Trigger** Two Level-3 coordinators manage the activity of the sub-system, together coordinating the hardware developments, the algorithmic firmware and software, and the core framework firmware. The sub-system also manages integration meetings with other sub-systems within the TDAQ UPR organisation and with other ATLAS upgrade projects on a regular basis.

**Central Trigger** The Central Trigger group manages and coordinates the activities for the CTP, the MUCTPI, the LTI and the TTC distribution. The group, also in charge of the existing central trigger system and its Phase-I upgrade, has tight connections with the CERN micro-electronics group, which, for example, has a key role in TTC developments.

## Organisation of the Data Acquisition system

The DAQ system comprises the Readout, Dataflow, Network and Online software sub-systems. The activities are managed by the associated Level-3 coordinators who work closely together with the existing DAQ/HLT coordination group, which is responsible for the Operations and Phase-I upgrades.

**Readout** The Level-3 coordinators manage the activities of the sub-system, including custom hardware development and commodity hardware procurement, firmware development, software development, and the integration and testing with detector systems. The Phase-I FELIX and swROD group will contribute to the R&D needed and integrate into the group.

**Dataflow** The Level-3 coordinator manages the activities of the sub-system and works with the existing Dataflow group for using the current infrastructure to perform evaluation and testing.

**Network** The Level-3 coordinator manages the activities of the sub-system and works with the existing network group for R&D activities, such as technology watching and technology evaluation.

**Online Software** The Level-3 coordinator manages the activities of the sub-system and works with the existing Configuration and Control group, and Monitoring group, for the long term development and the evolution strategy. Most of the online software elements will be maintained and upgraded adiabatically in the next several years by the DAQ/HLT coordination group. However, some of the core software infrastructure, may be significantly redesigned to benefit from the availability of new technologies. The chart in Figure 17.6 shows only those specific Phase-II upgrade activities.
<figure><img src="image_272.png" /><figcaption>Figure 17.6: Organisation of the sub-systems in the DAQ system</figcaption></figure>

## Organisation of the Event Filter & Performance System

The EF & Performance system comprises the HTT and EF sub-systems, and the Physics, Performance and Event Selection (PPES) activity. The Level-3 coordinators manage these three areas.

**Event Filter Software** Developments of the EF Tracking, Calorimeter, Muon and Core Software are organised within the relevant sub-groups of the existing Trigger activity area and its coordination group. Software developments and performance studies for conditions in Run 2, Run 3 and Run 4 are discussed in those forums. The Level-2 and Level-3 coordinators are active members of the Trigger groups, and shall be responsible (i) to guarantee that the developments of the trigger upgrade software are consistent with the UPR schedule, and (ii) to facilitate the contacts required with other upgrade subsystems.

**Event Filter processors** The computing requirements of the EF, calculated using a model described in Section 12.4, rely on performance information from all of the sub-groups. They will be periodically updated to check and if necessary revise the original estimates and ultimately as input to the procurement process.

Topics related to the EFPUs are discussed within the existing DAQ/HLT coordination group.

**Hardware Tracking for Trigger** The HTT sub-system has an established group with regular meetings where both hardware development, simulations and studies of the regional tracking and the global tracking scan are reported.
**Physics, Performance and Event Selection** The following activities are coordinated:

* Physics requirements capture

* Coordination of simulation of the TDAQ upgrades

* Performance studies of the overall upgrade and its subsystems

* Design of event selection strategies, including trigger menus and rate estimates

The group works closely with subsystems of the Level-0 Trigger and with the HTT and EF sub-systems to provide physics-motivated requirements and give performance feedback on their designs. These activities will continue to be organised periodically post-TDR, as required either by the UPR or external requests and initiatives such as a forthcoming CERN Yellow Report on HL-LHC. The group works also closely with members of the Trigger Activity area and its Coordination group for what concerns the strategies of the trigger menus in Run 4. It is expected that definite plans for the Run 4 trigger menus will come within the purview of the Trigger Coordination Group toward the end of Run 3.

<figure><img src="image_273.png" /><figcaption>Figure 17.7: Organisation of the sub-systems in the EF & Performance system</figcaption></figure>

## References

[17.1] ATLAS Collaboration, *ATLAS Upgrade Organisation*, Tech. Rep. ATU-ORG-MG-0009 v.1 / EDMS Doc. No. 1815123, CERN, Geneva, Jun, 2017.
https://edms.cern.ch/document/1815123/1.

[17.2] ATLAS Collaboration, *ATLAS Phase-II Upgrade Scoping Document*, Tech. Rep. CERN-LHCC-2015-020. LHCC-G-166, CERN, Geneva, Sep, 2015.
https://cds.cern.ch/record/2055248.
[17.3] ATLAS Collaboration, ATLAS Organisation, Tech. Rep. ATLAS Internal Note GEN-NO-009, CERN, Geneva, Dec, 2007.
https://atlas.web.cern.ch/Atlas/private/ATLAS_CB/CB_Approved_Documents/A59_ATLAS_Organisation%5B1.2%5D.pdf.

[17.4] The TDAQ Group Collaboration, Constitution of the ATLAS Trigger/DAQ Institute Board, https://twiki.cern.ch/twiki/bin/view/Atlas/AtlasTdaqIbConstitution.

[17.5] ATLAS Collaboration, ATLAS System Organisation, Tech. Rep. ATLAS Internal Note GEN-NO-015, CERN, Geneva, Nov, 2006.
https://atlas.web.cern.ch/Atlas/private/ATLAS_CB/CB_Approved_Documents/A56_ATLAS_System_Organisation%5B2.1%5D.pdf.

[17.6] ATLAS Collaboration, Technical Design Report for the Phase-I Upgrade of the ATLAS TDAQ System, Tech. Rep. CERN-LHCC-2013-018. ATLAS-TDR-023, CERN, Geneva, Sep, 2013. https://cds.cern.ch/record/1602235.
<figure><img src="image_274.png" /></figure>
# 18 Project Cost Estimates

In this chapter the cost estimates of the TDAQ upgrades are presented. The project is organised in a set of deliverables that are summarised in the Product Breakdown Structure table in Section 18.4. The chapter is logically organised in two parts. A first general part introduces: (i) the Cost Management Plan in Section 18.1 summarising the functions and the processes adopted by the project to estimate the costs, define budgets, and to manage and monitor the costs during the UPR's lifetime; and (ii) the cost policy and methodology, described in Sections 18.2 and 18.3, used for the TDAQ deliverables. The second part of the chapter is a collection of tables and figures presenting the cost estimate of each of the systems of the UPR (Section 18.5) and the expected spending profiles (Section 18.6).

## 18.1 Overview of the Cost Management Plan

The Cost Management Plan of the TDAQ UPR is based on the following fundamental functions/processes that are used to manage costs and budgets within the project.

**Cost Estimation** Cost estimation is the iterative process of developing, assembling and predicting, by successive approximations, the financial resources needed to complete the production of the project's deliverables. The cost estimating process prepares the cost tables and spending profiles, which are reviewed by the UCG group and are the baseline for the preparation of the TDAQ Memorandum of Understanding (MoU) document.

**Cost Budgeting** Cost budgeting is the set of accounting functions that establishes budgets and define the processes and the procedures for measuring, analysing and reviewing the forecasted and the actual costs. Budgeting is executed after the approval by the RB of the project and, once the MoU process has been completed.

**Cost Control** Cost control is the process concerning the collection, the analysis, the monitoring, the reporting and the managing of the costs on a regular basis. The UPR Resource Coordinator (UPR-RC), who reports directly the UPR's Project Office chair, and, ultimately to the UPL, will communicate and interact with the Institute's leaders to ensure, for example, that cost overruns do not endanger schedule or quality. The UPR-RC is also responsible
for controlling the **CORE** (see Section 18.2) cost estimates during the initial phases of the
project, until a MoU document has been approved. Once an MoU document is in force,
the UPR management will collect reports, for comparison with the CORE cost profile and
help the institutes take corrective actions to achieve minimum variances. Changes to the
CORE cost baseline profile will be documented along with an updated forecast for project
completion.

**Internal Scrutiny and reporting to TDAQ** The UPR Resource Coordinator (UPR-RC) is responsible, within the UPR Project Office, for executing these functions during the lifetime of the UPR. Within the UPR Project Office, the UPR-RC reports on the Cost/Scheduling status of the project, which will be reviewed on a quarterly basis by the UPL, who ultimately approves the report within the project.

The UPL reports to the TDIB on a yearly basis on the status of costs and spending of the project. The TDIB formally approves the budget and the spending profile of the project within TDAQ. In addition, the TDIB chair may appoint a special Scrutiny Group within TDAQ an annual independent internal review of the financial status of the UPR.

**Reporting to ATLAS** Frequent interactions with the ATLAS management are used to monitor the status of the project. The TDAQ UPR report will be used by the ATLAS Resources Coordinator in his/her interaction with the CERN RRB and with the RRB's Scrutiny Group.

**18.2 CORE Costing Policy**

At the time of the original construction, the LHCC established a COsting REview (CORE) committee charged with reviewing the experiment's costs. A costing policy was established to define a metric that assigns cost values to each deliverable. Since then CORE equivalent costs are used to estimate and evaluate the effective costs of upgrade projects.

**CORE** cost does not represent the entire cost of the TDAQ UPR. There are non-CORE costs that are required to complete successful production and delivery of the components of the TDAQ project. It is the responsibility of each Institution participating in the UPR to obtain financial support from its Funding Agency for both CORE and non-CORE expenditure on the deliverables where the institution is responsible.

The policy for the evaluation of the TDAQ UPR CORE costs is based on a few key ele-
ments:
* All production units that need to be installed in either USA15 or SDX1, and/or are required for the operations of the TDAQ system during HL-LHC shall be considered as ATLAS deliverables and their cost estimates shall be included in the CORE value of the project.

* Specialised infrastructure, tooling and equipment required to maintain production hardware, firmware and software of the TDAQ system during operations at the HL-LHC are also deliverables and shall be included in the CORE costs of the project.

* Evaluation of the CORE costs of a deliverable shall include, but not be limited to:

- Cost value of the materials and components that define a hardware deliverable.

- Non-recurring engineering (NRE) costs associated for example with ASIC production, PCB manufacturing or equivalent.

- Allowances for production yields in both manufacturing, assembly, and QA/QC testing.

* Pre-production items, manufactured and assembled with final tooling and labour to demonstrate production readiness, are to be considered CORE items and part of production.

* Generic infrastructure costs, such as prototype and development test stands costs, shall not be part of CORE.

* Services and labour costs for manufacturing and QA/QC procedures performed by industrial subcontractors shall be included in the CORE estimates.

* Labour and infrastructure at a production site of a participating Institute shall not be accounted in CORE.

* In general spares are not to be considered to be CORE items. As they represent failures occurring during operations, in general spares are to be considered part of the Maintenance and Operation (M&O) programme. Exception is made for those deliverables whose failure would cause a significant impact to the operations of ATLAS. In this case, spare units shall be included in the production and accounted for in the CORE estimates. These units shall be available at CERN for a possible emergency replacement in USA15 for any failing module.

* For applications where spares are accounted in CORE, the fraction of spare units to be manufactured in addition to the regular production shall not exceed 5% of the total quantity, except in cases where a very limited number of units are produced.

* No contingency shall be included in the CORE value of any deliverable. Institutions responsible for a deliverable are also responsible for the financial coverage and the additional costs that a deliverable might incur beyond what is specified in its CORE value.

## 18.3 Costing Methodology

A bottom-up methodology has been adopted for the estimate of the CORE values of the elements of the TDAQ UPR. Detailed checking and scrutiny, firstly within TDAQ, and, in
a second phase, by the Upgrade Project Office (UPO), has been performed on the estimates for each individual element of the UPR's systems and sub-systems.

* The quantities for each element of the upgrade have been calculated based on the architecture and design presented in this Technical Design Report.

* Each quantity is assigned an uncertainty reflecting possible changes of configurations, such as changes in interfaces because of the design evolution of a detector, new upgrade elements in the ATLAS upgrade scope, etc.

* Yield accounts for failure and loss during all the phases of production up to and including the installation of deliverables in the USA15 and SDX1 areas. A preliminary model of the production process is defined for a given deliverable, and a yield factor is estimated for each phase of construction based on past experience with projects of similar complexity and/or equivalent deliverables.

* Based on the policy given in Section 18.2, for mission-critical deliverables of the Level-0 Trigger system, some spare units are accounted in the CORE estimates. The fraction of spares to be considered in the production depends on the total production quantity for each sub-system, and shall be reported case by case in the Basis of Estimates (BOEs) documents associated to the UCG review that is scheduled in Q1 2018.

* Cost estimates for each item are quoted in the currency expected for the final procurements. Exchange rates to Swiss Francs (CHF) have been calculated as the average over the period of September 1st to November 30th, 2016, following the ATLAS convention used first in the ITk-strip TDR [18.1].

<table><caption>Table 18.1: Exchange rates used for the cost estimates. They represent the 3-month average over the period of September 1st to November 30th 2016, following the convention of the ITk-strip TDR [18.1].</caption><thead><tr><td>Currency</td><td>Swiss Francs [CHF]</td></tr></thead><tbody><tr><td>Euro [EUR]</td><td>1.085</td></tr><tr><td>British Pound [GBP]</td><td>1.246</td></tr><tr><td>U.S. Dollar [USD]</td><td>0.986</td></tr><tr><td>Japanese Yen [100 JPY]</td><td>0.942</td></tr><tr><td>Israeli Shekel [ILS]</td><td>0.2588</td></tr><tr><td>Chinese Yuan [CNY]</td><td>0.1461</td></tr></tbody></table>

* In general, values reported in this TDR reflect the estimates of a deliverable as if it were to be manufactured today. No de-rating factors are applied for procurements that will be executed in 3-5 years unless explicitly stated.

* An exception is made for the costs of the CPU-based commodity servers of the Event Filter system, as detailed in Section 18.5.3, where a performance improvement model is assumed as a function of time at a fixed cost.
* Similarly, assumptions are made on performance as function of purchase date when estimating costs of commercial off-the-shelf (COTS) integrated circuits (e.g. FPGAs), or high-speed serial computer expansion bus standards and related devices (e.g. speed of PCIe controllers).

### 18.3.1 Estimate Uncertainties

CORE costing is based on the Current Best Estimate (CBE) concept, i.e. on the information available at the time of the estimate, and it is naturally associated with an uncertainty. Several factors determine the level of cost uncertainty of an item: (i) the maturity of the technical development and design of a particular item, (ii) recent experience in other construction projects of items with similar technical complexity, (iii) the availability of vendor quotes through tendering processes or standard catalogues for commercial-off-the-shelf (COTS) components, (iv) understanding the procurement processes by the Institutions responsible for the deliverable item. To describe the level of uncertainty of an estimate, a quality factor ranging from 1 to 5 is used. The quality factors and the criteria are shown in Table 18.2: **QF1** has the highest certainty and is based on a vendor quote for the final item or a catalogue price; **QF5** has the lowest amount of certainty and is based on a rough estimate without any detailed design. For assembled items, i.e. comprising of several sub-assemblies or components with different quality factors, a cost-weighted average quality factor is calculated.

<table><caption>Table 18.2: Quality Factor (QF) definitions used to estimate uncertainties on the CORE values of the project elements. **QF1** has the lowest uncertainty, while **QF5** has the highest one.</caption><thead><tr><th>Factor</th><th>Definition of the criteria based upon</th></tr></thead><tbody><tr><td>QF1</td><td>(i) Items for which there is a recent (1 year max.) quote or catalogue price, based on a nearly completed design and for which there is more than one potential vendor<br>(ii) Items that are a copy or are almost identical to an existing design for which there is a recent catalogue price or quote and for which there is more than one potential vendor.</td></tr><tr><td>QF2</td><td>Items that just fall short of satisfying the QF1 criteria:<br>(i) Items that have only one potential vendor.<br>(ii) Estimates based on a detailed, but not completed, design.<br>(iii) Items adapted from an existing design with minor modifications.<br>(iv) Items having quotes >1 year old, but deemed still to be sufficiently reliable based on experience.</td></tr><tr><td>QF3</td><td>(i) Items with quotes > 2 years old.</td></tr></tbody></table>

continued ...
<table><thead><tr><th>Factor</th><th>Definition of the criteria based upon</th></tr></thead><tbody><tr><td></td><td>(ii) Items whose cost estimates are based on a conceptual design or adapted from an existing design with extensive modifications.<br>(iii) Items whose costs are estimated using physicist or engineering experience regardless of the maturity of the design.</td></tr><tr><td>QF4</td><td>Items that have unproven fabrication yields or for which there are unique issues, e.g. a special-order item and/or a single preferred vendor</td></tr><tr><td>QF5</td><td>Items that are still in a conceptual stage with no detailed specifications or design</td></tr></tbody></table>

## 18.4 UPR Project Breakdown Structure (PBS)

The UPR comprises a set of hardware, firmware and software deliverables that are organised in a hierarchically structured multi-levelled tree, the Product Breakdown Structure (PBS). The PBS mirrors the functional partition of the project into three (3) systems, twelve (12) sub-systems, and thirty-nine (39) components. The overall TDAQ UPR corresponds to the first level of the PBS, the three systems, i.e. Level-0, DAQ and EF, correspond to the Level-2 items in the PBS structure, while Level-3 items corresponds to sub-systems, e.g. the Level-0 Muon Trigger sub-system. In general, each Level-4 item in the PBS corresponds to one or more hardware deliverables. Firmware and software tasks related to the configuration, monitoring, control and operation of the hardware are included in the same PBS item.

Table 18.3 shows the PBS, expanded to the Level-4 of the tree structure, with a short dictionary entry for each item. A full PBS dictionary [18.2] down to Level-5/Level-6 of the structure is maintained in the "Project Management" section of the Electronics Data Management System (EDMS) folders dedicated to the TDAQ UPR, and is made available to the LHCC and UCG review panels through the UCG confidential material package.

The PBS seeds a Work Breakdown Structure (WBS), which defines the tasks and the processes to be executed for the design, prototyping, construction and the installation/ commissioning phases of the UPR, as described in Chapter 19.

<table><caption>Table 18.3: The TDAQ UPR Product Breakdown Structure (PBS).</caption><thead><tr><th>PBS Code</th><th>Item</th><th>Description</th></tr></thead><tbody><tr><td>1</td><td>TDAQ</td><td></td></tr><tr><td>1.1</td><td>Level-0 Trigger</td><td>Lowest-level, hardware-based Trigger system</td></tr><tr><td>1.1.1</td><td>CTP sub-system</td><td>Central Trigger Processor that forms the overall trigger decision and distributes timing information (TTC) to all sub-detectors</td></tr></tbody></table>

continued ...
<table><thead><tr><th>PBS Code</th><th>Item</th><th>Description</th></tr></thead><tbody><tr><td>1.1.1.1</td><td>CTPMI module</td><td>Interface of the CTP to LHC timing signals. It makes available timing signals on the ATCA back-plane for distribution to other CTP modules</td></tr><tr><td>1.1.1.2</td><td>CTPIN module</td><td>ATCA blade that accepts electrical and optical inputs for non-latency-critical signals to the CTP</td></tr><tr><td>1.1.1.3</td><td>CTPCORE module<br>Processor</td><td>ATCA blade, implementing the core functions of the Central Trigger and is the source of signals to the TTC system</td></tr><tr><td>1.1.1.4</td><td>Infrastructure</td><td>ATCA shelf, patch panels and optical fibres</td></tr><tr><td>1.1.2</td><td>MUCTPI sub-system</td><td>Interface that combines and processes muon trigger information from the Barrel and Endcap Muon Trigger systems, and interfaces to the Global Trigger and to the CTP</td></tr><tr><td>1.1.2.1</td><td>MUCTPI module</td><td>ATCA blade handling one quarter of the muon trigger geometric coverage</td></tr><tr><td>1.1.3</td><td>TTC system</td><td>Trigger Timing and Control infrastructure</td></tr><tr><td>1.1.3.1</td><td>LTI</td><td>Local Trigger Interface module: ATCA blade that is an integral part of TTC and provides an interface between the CTP sub-system and FELIX or sub-detector-specific electronics through a Passive Optical Network (PON).</td></tr><tr><td>1.1.3.2</td><td>Infrastructure</td><td>ATCA shelves for LTIs, Passive Optical Network (PON) trees elements (splitters and fibres)</td></tr><tr><td>1.1.4</td><td>Calorimeter Trigger</td><td>This system defines Level-0 Trigger Objects comprising electron/gamma, tau, jet, large-R jet objects and global variable like $E_T^{miss}$, and provide this information to Global trigger.</td></tr><tr><td>1.1.4.1</td><td>High-eta feature extraction</td><td>ATCA blade-based system, which defines electron/gamma, tau and jet Level-0 candidates in the Forward region</td></tr><tr><td>1.1.4.2</td><td>Fibre management</td><td>Optical fibre plant to handle Phase-2 interfaces to the calorimeters and to the Global trigger</td></tr><tr><td>1.1.5</td><td>Muon Trigger</td><td>System that defines Muon Level-0 Trigger Objects and provides this information to the MUCTPI.</td></tr><tr><td>1.1.5.1</td><td>TGC SL</td><td>TGC Sector Logic Board: an ATCA blade that defines Muon candidates for a Muon Endcap trigger sector and provides a seed for the MDT Trigger Processor; it subsequently combines the MDT trigger result with the RPC/TGC one and provides the information to the MUCTPI.</td></tr></tbody></table>

continued ...
<table><thead><tr><th>PBS Code</th><th>Item</th><th>Description</th></tr></thead><tbody><tr><td>1.1.5.2</td><td>RPC SL</td><td>RPC Sector Logic Board: an ATCA blade that defines Muon candidates for a Muon Barrel trigger sector and provides a seed for the MDT Trigger Processor; it subsequently combines the MDT trigger result with the RPC/TGC one and provides the information to the MUCTPI.</td></tr><tr><td>1.1.5.3</td><td>MDT Processor Main Board</td><td>ATCA blade that processes MDT hits in regions identified by Barrel or Endcap SL Board and provides Muon candidates transverse momentum and coordinates measurements through precision tracking to MUCTPI.</td></tr><tr><td>1.1.5.4</td><td>MDT Rear Transition module</td><td>Auxiliary module to an MDT Processor Main Board</td></tr><tr><td>1.1.5.5</td><td>MDT Sector processor mezzanine</td><td>Auxiliary module to an MDT Processor main board; it performs the segment finding part of the tracking algorithm.</td></tr><tr><td>1.1.5.6</td><td>NSW Trigger carrier board</td><td>ATCA blade that is the main carrier board providing all services to the system.</td></tr><tr><td>1.1.5.7</td><td>NSW Trigger Mezzanine</td><td>Auxiliary module to the NSW Trigger Carrier Board, it contains all the MultiGigaBit interfaces and implements the Muon identification using the New Small Wheel detector data. It provides muon candidates to the Endcap Sector Logic.</td></tr><tr><td>1.1.5.8</td><td>Infrastructure</td><td>ATCA shelves for all Muon Trigger Processors</td></tr><tr><td>1.1.6</td><td>Global Trigger subsystem</td><td>The Global Trigger receives and processes trigger information from the calorimeter and muon triggers. It also uses full-granularity calorimeter information to refine calorimeter objects and performs topology-based multi-object selections.</td></tr><tr><td>1.1.6.1</td><td>Common Module</td><td>An ATCA blade, it is the basic hardware platform used for all Global trigger functional elements.</td></tr><tr><td>1.1.6.2</td><td>PFM</td><td>Production Firmware deployment Module: ATCA blade used as hardware platform for all algorithm and infrastructure developments</td></tr><tr><td>1.1.6.3</td><td>Infrastructure</td><td>ATCA shelves and fibre plant needed to interface Global trigger elements</td></tr><tr><td>1.2</td><td>DAQ</td><td>The DAQ system transports and stores data accepted by the Level-0 trigger for Event Filter processing.</td></tr></tbody></table>

continued ...
<table><thead><tr><th>PBS Code</th><th>Item</th><th>Description</th></tr></thead><tbody><tr><td>1.2.1</td><td>Detector Readout</td><td>The Detector Readout subsystem receives event data from detector front-end links and facilitates detector specific processing.</td></tr><tr><td>1.2.1.1</td><td>FELIX</td><td>The Front-End LInk eXchange (FELIX) routes detector data from custom serial links to a commodity network.</td></tr><tr><td>1.2.1.2</td><td>Data Handler</td><td>The Data Handler receives data from FELIX via a commodity multi-gigabit network and hosts detector-specific processing.</td></tr><tr><td>1.2.2</td><td>Dataflow</td><td>The Dataflow subsystem buffers, transports, aggregates and compresses event data.</td></tr><tr><td>1.2.2.1</td><td>Storage Handler</td><td>The Storage Handler buffers data received from the Detector Readout subsystem through the Event Builder to decouple the Readout and Event Filter.</td></tr><tr><td>1.2.2.2</td><td>Event Aggregator</td><td>The Event Aggregator receives events accepted by the Event Filter, performs compression, and then prepares output files for transfer to permanent storage</td></tr><tr><td>1.2.3</td><td>Network</td><td>The network for all TDAQ communications and data transport</td></tr><tr><td>1.2.3.1</td><td>FELIX network</td><td>The network connecting the FELIX servers and the Data Handlers with additional requirements for DCS, control, and monitoring infrastructure interconnection</td></tr><tr><td>1.2.3.2</td><td>Routers and fibres</td><td>The routers and fibres to be used for the data network and control network</td></tr><tr><td>1.2.3.3</td><td>Top-of-Racks (ToR) switches and fibres</td><td>The switch and fibres in an Event Filter Farm rack for connecting to the core router</td></tr><tr><td>1.2.4</td><td>Infrastructure</td><td>Infrastructure to facilitate the operation of all DAQ PC units in SDX1 (racks and coolers for electronics in USA15 are under the responsibility of Technical Coordination.</td></tr><tr><td>1.2.4.1</td><td>Server Rack</td><td>Racks to host the PC servers and network switches</td></tr><tr><td>1.2.4.2</td><td>Rack cooler</td><td>Cooler to provide cooling in the server rack</td></tr><tr><td>1.3</td><td>Event Filter</td><td>The Event Filter (EF) reduces the L0 trigger rate by two orders of magnitude through reconstruction and selection of events with commodity processors assisted by hardware tracking processors</td></tr><tr><td>1.3.1</td><td>Tracking Hardware (HTT)</td><td>The co-processor that will perform track reconstruction for the Event Filter.</td></tr></tbody></table>

continued ...
<table><thead><tr><th>PBS Code</th><th>Item</th><th>Description</th></tr></thead><tbody><tr><td>1.3.1.1</td><td>AM Tracking Processor (AMTP)</td><td>The ATCA main card that will perform first-stage or second-stage processing</td></tr><tr><td>1.3.1.2</td><td>AMTP Rear transition Module (RTM)</td><td>The Rear Transition Module provides optical connections for the external interfaces of the TP</td></tr><tr><td>1.3.1.3</td><td>PRM</td><td>Pattern Recognition Mezzanine: The card where AM-based pattern recognition and the first stage track fitting will be performed</td></tr><tr><td>1.3.1.4</td><td>AM ASIC</td><td>The Associative Memory ASIC that will find the initial track candidates through pattern matching.</td></tr><tr><td>1.3.1.5</td><td>TFM</td><td>Track fitting Mezzanine: The card where first-stage track candidates will be extrapolated to second-stage layers, where track fits using all layers will then be performed.</td></tr><tr><td>1.3.1.6</td><td>Infrastructure</td><td>The ATCA infrastructure dedicated to HTT that will be common within TDAQ.</td></tr><tr><td>1.3.1.7</td><td>Hardware Tracking Interface</td><td>The interface between the EF processors and the HTT units</td></tr><tr><td>1.3.2</td><td>Processing Units</td><td>The commodity compute servers and relative infrastructure.</td></tr><tr><td>1.3.2.1</td><td>Servers</td><td>The commodity compute servers, which will perform the EF event processing.</td></tr></tbody></table>

## 18.5 UPR CORE Costing Tables

Table 18.4 presents the CORE cost estimates of the three systems constituting the TDAQ upgrade project.<fnref n="1" />

<table><caption>Table 18.4: CORE Cost summary of the TDAQ UPR project for each system.</caption><thead><tr><th>PBS Code</th><th>System</th><th>CORE Cost [kCHF]</th></tr></thead><tbody><tr><td>1</td><td>TDAQ Total</td><td>44,880</td></tr><tr><td>1.1</td><td>Level-0 Trigger</td><td>10,393</td></tr><tr><td>1.2</td><td>DAQ</td><td>13,640</td></tr><tr><td>1.3</td><td>Event Filter</td><td>20,847</td></tr></tbody></table>

<footnote n="1">¹ Commodity items in the current system that will not be changed for the upgrade and that are not listed in the tables in this section will be retained for Phase-II and are subject to the rolling M&O replacement policy.</footnote>
## 18.5.1 Level-0 Trigger

The CORE costing of the Level-0 Trigger system is summarised in Table 18.5 and shown as a pie chart in Figure 18.1. The estimates reflect the design detailed in Chapters 7-10. The L0Muon and Global Trigger sub-systems are the main cost drivers within the Level-0 Trigger system because of the large number of ATCA blades and of the large processing FP-GAs required. The highest uncertainty in the cost of the sub-system components is due to the uncertainty in the estimate of the processing resources needed, and consequent FPGA choices. QF3 is in general assigned to all the trigger processors. QF4 is used for the Global Trigger GEP and MUX modules, since new generation FPGAs that are expected to be available on a 2-3 year timescale should provide substantially more processing power at fixed costs. The UCG confidential material includes Basis of Estimates' documents that detail and justify the values shown in Table 18.5 down to the Level-5 of the PBS.

<figure><img src="image_275.png" /><figcaption>Figure 18.1: CORE cost distribution of the Level-0 Trigger System.</figcaption></figure>

<table><caption>Table 18.5: CORE Cost Estimate of the Level-0 Trigger system in the TDAQ UPR. CORE values of the system's components represent Current Best Estimates.</caption><thead><tr><th>PBS Code</th><th>Item</th><th>Cost [kCHF]</th><th>Cost Quality</th></tr></thead><tbody><tr><td>1.1</td><td>Level-0 Trigger System</td><td>10,393</td><td></td></tr><tr><td>1.1.1</td><td>Central Trigger Processor</td><td>726</td><td></td></tr><tr><td>1.1.1.1</td><td>CTPMI</td><td>58</td><td>QF3</td></tr><tr><td>1.1.1.2</td><td>CTPIN</td><td>116</td><td>QF3</td></tr><tr><td>1.1.1.3</td><td>CTPCORE</td><td>501</td><td>QF3</td></tr><tr><td>1.1.1.4</td><td>Infrastructure</td><td>51</td><td>QF1</td></tr><tr><td>1.1.2</td><td>MUCTPI System</td><td>213</td><td></td></tr></tbody></table>

continued ...
<table><thead><tr><th>PBS Code</th><th>Item</th><th>Cost (CBE) [kCHF]</th><th>Cost Quality</th></tr></thead><tbody><tr><td>1.1.2.1</td><td>MUCTPI Modules</td><td>209</td><td>QF2</td></tr><tr><td>1.1.2.2</td><td>Infrastructure</td><td>4</td><td>QF1</td></tr><tr><td>1.1.3</td><td>TTC system</td><td>817</td><td></td></tr><tr><td>1.1.3.1</td><td>LTI</td><td>643</td><td>QF3</td></tr><tr><td>1.1.3.2</td><td>Infrastructure</td><td>174</td><td>QF1</td></tr><tr><td>1.1.4</td><td>Calorimeter Trigger</td><td>389</td><td></td></tr><tr><td>1.1.4.1</td><td>High-eta feature extraction</td><td>357</td><td>QF3</td></tr><tr><td>1.1.4.2</td><td>Fibre management and Infrastructure</td><td>32</td><td>QF2</td></tr><tr><td>1.1.5</td><td>Muon Trigger</td><td>5,111</td><td></td></tr><tr><td>1.1.5.1</td><td>TGC SL Board</td><td>1,479</td><td>QF3</td></tr><tr><td>1.1.5.2</td><td>RPC SL Board</td><td>1,024</td><td>QF3</td></tr><tr><td>1.1.5.3</td><td>MDT Processor Main Board</td><td>939</td><td>QF3</td></tr><tr><td>1.1.5.4</td><td>MDT RTM</td><td>118</td><td>QF3</td></tr><tr><td>1.1.5.5</td><td>MDT Sector Processor Mezzanine</td><td>636</td><td>QF3</td></tr><tr><td>1.1.5.6</td><td>NSW Trigger Carrier Board</td><td>127</td><td>QF3</td></tr><tr><td>1.1.5.7</td><td>NSW Trigger Mezzanine</td><td>607</td><td>QF3</td></tr><tr><td>1.1.5.8</td><td>Infrastructure</td><td>181</td><td>QF3</td></tr><tr><td>1.1.6</td><td>Global Trigger</td><td>3,135</td><td></td></tr><tr><td>1.1.6.1</td><td>Common Module</td><td>2,422</td><td>QF3</td></tr><tr><td>1.1.6.4</td><td>PFM</td><td>493</td><td>QF3</td></tr><tr><td>1.1.6.5</td><td>Infrastructure</td><td>220</td><td>QF2</td></tr></tbody></table>

## 18.5.2 Data Acquisition System

The CORE cost estimate of the DAQ system is shown in Table 18.6 and in Figure 18.2. The estimates reflect the design detailed in Chapter 11. The detector readout costs are driven essentially by the number of links required to read out the ATLAS detector systems at the specified trigger rate of 1 MHz and for an event size including 200 additional events from pileup. Dataflow and network costs are essentially determined by the ATLAS event size expected at the HL-LHC (see Section 11.5). The UCG confidential material includes Basis of Estimates' documents that detail and justify the values shown in Table 18.6 down to the Level-5 of the PBS.

## 18.5.3 Event Filter

The CORE cost estimate of the EF system is shown in Table 18.7 and in Figure 18.3. The estimates reflect the technical design presented in Chapters 12 and 13 for the EFPU servers and for the custom-hardware tracking co-processors, respectively.
<table><caption>Table 18.6: CORE Cost Estimate of the DAQ system in the TDAQ UPR. CORE values of the system's components represent current best estimates.</caption><thead><tr><th>PBS Code</th><th>Item</th><th>Cost [kCHF]</th><th>Cost Quality</th></tr></thead><tbody><tr><td>1.2</td><td>DAQ</td><td>13,640</td><td></td></tr><tr><td>1.2.1</td><td>Detector Readout</td><td>6,186</td><td></td></tr><tr><td>1.2.1.1</td><td>FELIX</td><td>4,262</td><td>QF3</td></tr><tr><td>1.2.1.2</td><td>Data Handler</td><td>1,924</td><td>QF3</td></tr><tr><td>1.2.2</td><td>Dataflow</td><td>4,034</td><td></td></tr><tr><td>1.2.2.1</td><td>Storage Handler</td><td>4,006</td><td>QF3</td></tr><tr><td>1.2.2.2</td><td>Event Aggregator</td><td>28</td><td>QF3</td></tr><tr><td>1.2.3</td><td>Network</td><td>3,315</td><td></td></tr><tr><td>1.2.3.1</td><td>FELIX network</td><td>916</td><td>QF2</td></tr><tr><td>1.2.3.2</td><td>Routers and fibres</td><td>2,109</td><td>QF3</td></tr><tr><td>1.2.3.3</td><td>ToR switches and fibres</td><td>290</td><td>QF3</td></tr><tr><td>1.2.4</td><td>Infrastructure</td><td>105</td><td></td></tr><tr><td>1.2.4.1</td><td>Server Rack</td><td>45</td><td>QF3</td></tr><tr><td>1.2.4.2</td><td>Rack cooler</td><td>60</td><td>QF3</td></tr></tbody></table>

<figure><img src="image_276.png" /><figcaption>Figure 18.2: CORE cost distribution of the DAQ System.</figcaption></figure>
<figure><img src="image_277.png" /><figcaption>Figure 18.3: CORE cost distribution of the EF System.</figcaption></figure>

<table><caption>Table 18.7: CORE Cost Estimate of the EF system in the TDAQ UPR. CORE values of the system's components represent current best estimates.</caption><thead><tr><th>PBS Code</th><th>Item</th><th>Cost [kCHF]</th><th>Cost Quality</th></tr></thead><tbody><tr><td>1.3</td><td>Event Filter</td><td>20,847</td><td></td></tr><tr><td>1.3.1</td><td>Tracking Hardware</td><td>17,448</td><td></td></tr><tr><td>1.3.1.1</td><td>Tracking Processor (TP)</td><td>6,451</td><td>QF3</td></tr><tr><td>1.3.1.2</td><td>TP Rear transition Module (RTM)</td><td>695</td><td>QF2</td></tr><tr><td>1.3.1.3</td><td>Pattern Recognition Mezzanine (PRM)</td><td>4,678</td><td>QF3</td></tr><tr><td>1.3.1.4</td><td>AM ASIC</td><td>3,293</td><td>QF4</td></tr><tr><td>1.3.1.5</td><td>Track fitting Mezzanine (TFM)</td><td>1,261</td><td>QF3</td></tr><tr><td>1.3.1.6</td><td>Infrastructure</td><td>658</td><td>QF2</td></tr><tr><td>1.3.1.7</td><td>Hardware Tracking Interface</td><td>411</td><td>QF2</td></tr><tr><td>1.3.2</td><td>Processing Units</td><td>3,399.0</td><td></td></tr><tr><td>1.3.2.1</td><td>Servers</td><td>3,399.0</td><td>QF3</td></tr></tbody></table>

The UCG confidential material includes Basis of Estimates' and additional documentation that detail and justify the values shown in Table 18.7 down to the Level-5 of the PBS. The remaining sections just briefly outline a few considerations on costs for the different components of the EF sub-systems.

## Tracking Hardware

The dimensions of the system described in Chapter 13 determine the quantities in the first part of Table 18.7. The FPGAs of the 672 TP main boards and in the 1152 PRM mezzanines are the cost drivers of the HTT sub-system. The cost uncertainties are determined by the
choice of the FPGAs in the TP modules and the PRM and TFM mezzanines, where a quality factor QF4 is assigned. The ASIC costs are driven by the NRE costs, which account for two separate mask-sets for pre-production and production, and include a 70% yield production. An overall quality factor QF3 is assigned.

## Event Filter Processing Units

The CORE cost estimates of the EFPU sub-system are based on the technical considerations detailed in Chapter 12. A total of ~4.5 MHS06 have been estimated to be required for the EF system at the ultimate performance configuration of the HL-LHC. Two factors contribute to the CORE cost estimates: (i) the CPU cost extrapolations, together with the projected hardware configuration of the commodity servers in 2026; and (ii) the continuation of the M&O rolling replacement foreseen to continue throughout the lifetime of the experiment.

**CPU cost extrapolations:** The latest CERN IT extrapolations indicate computing costs between 1.4 and 3.3 CHF/HS06 in 2026, as shown in Figure 18.4. The extrapolations are based on past purchases, assuming an exponential decrease. Similar extrapolations, based on the ATLAS TDAQ purchases, provide a compatible figure of roughly 2 CHF/HS06. This type of extrapolations have shown a large variability. For example the 2013 CERN IT analysis indicated a compute cost of 1.1 CHF/HS06 in 2023, significantly below the current figures. The variability probably reflects the market-driven nature of the compute technology sector. Considering the CERN IT and ATLAS TDAQ results, as well as their evolution in the past few years, a value of 2 CHF/HS06 was chosen as estimated compute cost in 2026.

**M&O Rolling-replacement:** The TDAQ M&O policy to partly replace computing resources at the end of the 5-year lifetime of a server defines a rolling-replacement procedure where approximately a third of the EF farm is replaced every 2-3 years. In the event of a long-shutdown of the LHC complex, the rolling-replacement is foreseen for the first year when the LHC operations are resumed. To achieve the projected 4.5 MHS06 required for the operations at the ultimate performance configuration of HL-LHC 1133 servers should be installed by the end of the LS3 shutdown in addition to the rolling replacements, which is the cost reported in Table 18.7.

## 18.6 Costing Profiles

Figure 18.5 shows the spending profile projected in the calendar years 2018-2026. The stacked columns represent the sum of the three systems the upgrade project is comprised of: the Level-0 trigger (blue), DAQ (green) and the EF (yellow) systems.
<figure><img src="image_278.png" /><figcaption>Figure 18.4: Preliminary extrapolation of CPU costs based on CERN procurements [18.3]. Two extrapolation curves are given: a pessimistic scenario with a 10% annual reduction, and a scenario considered realistic with a 20% annual reduction of the CPU cost per HS06 unit. An overall factor 2.2 difference between the two scenarios is calculated at the end of the LS3 shutdown.</figcaption></figure>

<figure><img src="image_279.png" /><figcaption>Figure 18.5: Estimated CORE spending profile of TDAQ upgrade project. The stacked histogram shows the contributions from the PBS Level-2 systems: the Level-0 trigger, DAQ and EF.</figcaption></figure>
The expected spending time profiles, expanded at Level-3 of the PBS (sub-systems) for each of the systems, are shown in Figures 18.6, 18.7, and 18.8, respectively. The profiles reflect the schedule discussed in Chapter 19.

## 18.6.1 Spending profile of the Level-0 Trigger System

The Level-0 Trigger sub-systems components are essentially designed as ATCA modular electronics, and their production spans the years 2022-2024.

<figure><img src="image_280.png" /><figcaption>Figure 18.6: Estimated CORE spending profile for the Level-0 trigger system. The stacked histogram shows the contributions from the PBS Level-3 items.</figcaption></figure>

## 18.6.2 Spending profile of the DAQ System

The cost of the DAQ upgrade is spread between 2024 and 2026. The plan includes the support for the surface tests of the ITk detectors during assembly through pre-production units. Some DAQ elements, in particular for what concerns networking, are COTS components purchased toward the end of LS3 and the beginning of the HL-LHC operations.
<figure><img src="image_281.png" /><figcaption>Figure 18.7: Estimated CORE spending profile for the DAQ system. The stacked histogram shows the contributions from the PBS Level-3 items.</figcaption></figure>

### 18.6.3 Spending profile of the EF Trigger System

The HTT sub-system is based on ATCA boards and contains a challenging and technically complex ASIC. The production peaks in the years 2020-2024, with pre-production ASICs already produced as early as 2020. Procurement of the EFPU servers is carried out conveniently as late as possible, consistently also with the schedule for the M&O rolling replacement programme (see Section 18.5.3).
<figure><img src="image_282.png" /><figcaption>Figure 18.8: Estimated CORE spending profile for the EF system. The stacked histogram shows the contributions from the PBS Level-3 items.</figcaption></figure>

References

[18.1] ATLAS Collaboration, Technical Design Report for the ATLAS Inner Tracker Strip Detector, Tech. Rep. CERN-LHCC-2017-005, CERN, Geneva, Apr, 2017.
https://cds.cern.ch/record/2257755.

[18.2] The TDAQ Group, TDAQ Phase-II PBS dictionary, Tech. Rep. ATL2-D-MC-0012, EDMS Id 1817664 v.2, CERN, Geneva, Oct, 2017.
https://edms.cern.ch/document/1817664/2.

[18.3] H. Meinhard, Technology Evolution, Tech. Rep. CHEP 2016, CERN, Geneva, Oct, 2016. https://indico.cern.ch/event/555063/contributions/2285842/attachments/1350817/2039143/2016-10-08-WLCGWorkshop-TechnologyEvolution.pdf.
<figure><img src="image_283.png" /></figure>
# 19 Planning and Schedule

This chapter focuses on the planning and an initial high-level schedule of the UPR that summarises the information documented in the Schedule Management Plan (SMP) [19.1], which is part of the *Confidential Material* package to be reviewed by the UCG in early 2018, and available on the EDMS area of the project [19.2].

Detailed bottom-up plans of the activities of each sub-system [19.2] are used as the basis for the overall project plan, schedule and milestones. The plan described in this TDR and in the SMP builds on the experience of the TDAQ Phase-I upgrade project [19.3].

The schedule includes all the tasks required until the completion of the project with installation and commissioning finished, and all their interdependencies. The plans will evolve as the project moves forward, including the addition of more detailed breakdown of tasks as they approach. This will enhance the ability of the UPR management to track progress, detect delays as early as possible, and take pre-emptive actions.

The chapter is organised as follows: Section 19.1 summarises the principal concepts and guidelines extracted from the project's SMP; Section 19.2 describes the production plan and the reviewing process that controls and monitors the technical progress of the project for a successful and timely delivery to ATLAS. Finally, Section 19.3 summarises the initial plans of the project's systems and sub-systems, describing their top-level production schedule, and presenting in full detail one reference example, the L0Muon Trigger Barrel and Endcap SL.

## 19.1 Overview of the Schedule Management Plan

The complete plan to monitor, control, and revise, if necessary, the schedule of the project is documented in a separated document [19.1]. The basic elements of the Plan are:

* The systems' and sub-systems' coordinators are responsible for identifying all the activities and the tasks required for the design, construction, installation and commissioning of a given deliverable. The tasks and their duration are documented in a structured hierarchical work breakdown document (WBS) which goes down to Level-5 or Level-6.
* The coordinators shall agree with the project's UPR-RC on the consistency and completeness of the WBS's tasks required to deliver successfully to ATLAS the system/sub-systems' elements.

* The coordinators prepare the schedule for each Level-2 and Level-3 deliverable. The schedules are approved by the UPR-RC through an internal scrutiny process, that is chaired by the UPR-RC. The UPR-RC reports the conclusions of the internal scrutiny to the eTDSG, requesting the report's endorsement by the UPL after a discussion and the unanimous agreement of the eTDSG members.

* The UPR-RC assists the UPL to develop a comprehensive schedule, in accordance with the general schedule developed by the ATLAS USC, to seek the necessary review process to baseline the schedule, to oversee the progress, and to take necessary corrective actions to ensure that the project remains on schedule.

* The schedule of each Level-2 and Level-3 deliverable shall include all the formal review steps outlined in Section 19.2, which are necessary tools for the UPR management to monitor and track that the design and/or production make adequate progress towards their completion.

* After each Level-2 and Level-3 deliverable's approval, it is the responsibility of the system and sub-system's coordinators to implement, execute, and track the progress of their project against the baselined schedule for their respective deliverables. They report regularly on the progress to the eTDSG.

* In the event of significant accumulated schedule variance, or in the event of technical issues emerging during the production phase, the UPR-RC organises a Production Assessment Review (PAR) at a production site to identify the root-causes of the technical issues and to propose corrective actions. The UPR-RC reports to the UPL who approves formally the proposed corrected actions coming from the PAR, following a discussion and agreement by the eTDSG.

* The UPR-RC may request the UPL's approval for a revision-controlled change of the baselined schedule in the event of a significant variance. The UPL approval is conditional to the majority agreement by the eTDSG members.

* The UPL reports regularly to the TDIB on the state of progress compared to the baselined schedule.

* The UPL informs immediately the TDIB and the ATLAS USC, in the event of becoming aware of a critical schedule variance.

* Upon request by the ATLAS Management, and, specifically by the UC, the UPL reports to the ATLAS USC on progress with respect to the baselined schedule. ATLAS Management through the UPO may also require periodic formal reviews of the status of the UPR, along the same lines as the ASSO reviews during the original construction phase.
## 19.2 Design, Production Plan and Hardware/Firmware Integration

The development and production process of each sub-system, which is broken down into tasks in the WBS, is mapped onto a few generic phases during the project lifetime. At the end of each phase a formal review establishes whether the phase has concluded successfully: for example, if it is compliant to the specifications, and satisfies all the quality and reliability requirements. If so, it is approved to progress to the next stage of the development/production cycle. The review may be an internal process within the TDAQ organisation, or an agreement with ATLAS Management to hold a formal ATLAS event organised either by the ATLAS Electronics Coordinator, in his/her function of chair of the ATLAS Electronics Review Office, or by a designated software responsible for software matters.

<figure><img src="image_284.png" /><figcaption>Figure 19.1: Sequence of activities and approval reviewing processes at different stages of the development and of the production cycle of any project's deliverable.</figcaption></figure>

Each coordinator shall define and include in the system/sub-system planning all the milestones and phases (from initial developments to the final installation), which are represented graphically in Figure 19.1 and described in the paragraphs that follow.
The statement is valid not only for the hardware deliverables, but for firmware and software components as well, as they shall be considered deliverables in the same way as their hardware counterparts.

The complexity and resources of single devices have grown by more than two orders of magnitude since the original ATLAS construction, even with respect to the large digital ASIC designs of some sub-systems. Recognising the significance of this, firmware activities shall be integrated into the UPR planning in a more structured and detailed fashion than has been done in the past. In particular, the firmware development and the hardware design shall be planned and scheduled coherently from the Requirements capture up to the Commissioning of each sub-system. The need for this approach is particularly true for the TDAQ UPR where most of the sub-systems contain technically challenging PCBs that host multiple high-performance FPGAs, in which complex algorithms are implemented which must fit the resource limits of the selected device.

There are four types of design reviews for each sub-system component:

* the Specification Review (SR),

* the Preliminary Design Review (PDR),

* the Final Design Review (FDR),

* the Production Readiness Review (PRR).

These reviews are done in agreement with ATLAS Management and are organised by the ATLAS Electronics Coordinator, in his/her function of chair of the ATLAS Electronics Review Office. Each of these reviews is described in a dedicated paragraph below.

For TDAQ components that contain a large effort on software development software reviews will be carried out at the time of SR, PDR, FDR, PRR. The scope of these reviews will be defined in a dedicated document.

A number of Internal TDAQ Reviews are also foreseen at various stages, for example at the end of the requirement capture, or just before installation of a subsystem. The scope of these reviews varies and they are briefly described in a single dedicated paragraph below.

## 19.2.1 Specification Review

The first step of the component development is the definition of a set of specifications and its proposed architecture. A Specification Document is prepared, based on a set of requirements, described in a "User Requirement Document" (URD) previously reviewed during an Internal TDAQ Review. The SR reviews the Specification Document that describes the required functionality and performance of the device, its interfaces to other devices, and reliability. The specified interfaces must be cross-checked for consistency with the corresponding components specifications. The approval of the SR starts the formal design phase of a prototype that captures all or a large fraction of the functions specified in the Specification Document. At the end of the design cycle a PDR is organised.
## 19.2.2 Preliminary Design Review

The PDR establishes whether the design meets all the aspects of the requirements and specifications, including all interfaces to other components. The prototype may not implement all functionality required in final system. If so, the prototype will need its own specifications, distinct from that reviewed previously in the SR. To avoid ambiguity, in this case, the PDR will address the prototype spec and any differences between this and the SR specification.

The review process is based upon the following documentation:

* The URD and the Specification Document to be used as benchmarks for the prototype design evaluation.

* Block diagrams - both at the board level and at the level of single FPGAs.

* Full Schematics-set of prototype PCB designs.

* PCB layout and PCB stack-up information when deemed necessary by the complexity of the component.

* High-level and/or SPICE-level simulations of the main functions.

* Preliminary estimate of the power consumption of the board appropriately dressed and configured.

* Preliminary thermo-mechanical analysis through a Finite Element Analysis program.

* Design and Verification of all the input/output interfaces.

PDRs should be held prior to the submission of any prototype design for fabrication or assembly.

The PDR shall also include in its scope the initial review of the FPGA firmware, held as a separate review if judged so by the Electronics Coordinator. The details of the scope of the firmware review at the time of the PDR will be described in a separate document.

## 19.2.3 Final Design Review

The FDR is the last step of the design phase of a component deliverable. The purpose of the review is to assess whether the design is mature enough for pre-production. All requirements and specifications are reviewed. A successful performance measurement of the prototype(s) that have been built allows the project to finalise the design in preparation for small scale pre-production, which will be approved upon a successful FDR review.

The review's scope and the documentation to be prepared for the hardware deliverables are similar to the PDR, emphasising the design modifications made since the design of the last prototype iteration (see Section 19.2.2).

The FDR shall also include in its scope a review of the FPGA firmware, held as a separate review if judged so by the Electronics Coordinator. The details of the scope of the firmware
review at the time of the FDR will be described in a separate document, but as a general
guideline, the following aspects related to the firmware implementation will be reviewed:

* FPGA resource utilisation.

* Firmware framework description

* External interface tests (such as interface link-speed tests, performance measurements of individual interfaces for serialiser/deserialise latency at a given baud rate and a defined data protocol, etc)

* Single board internal slice test (for the realtime path if applicable, otherwise for the readout path, with at least one-channel input, a single processing algorithm, and at least one-channel output).

* Power consumption based on real implementation of the most power-hungry functionalities

19.2.4 Production Readiness Review

The PRR shall address quality management (QA/QC), version control (hardware and firm-
ware) and any outstanding items from earlier reviews.

During the Production Readiness Review, the following items shall be considered:

* full functionality test with single board, including readout and monitoring path;

* system slice-test, where this project is part of a bigger test stand (many-channel in, many-channel out if applies, or a single processing algorithm, out of many, running on all input channels);

* test hardware and software needed for production testing;

* production-related issues (plan, monitoring of important parameters like yield);

* production test methodology (dedicated firmware, dedicated test bench, paths used as for example: JTAG chain coverage and the necessary functional tests where JTAG does not cover parts of the design for example);

* QA/QC plan.

At the level of single-board test, full system test or system integration test, a description
of the DAQ software necessary for configuration, test and monitoring, together with a de-
scription of its development and major milestones should be presented. Power estimates
need to be re-analysed during the review. Test coverage of all test benches should be eval-
uated, if they have changed in terms of input data and output checks, or if some new ones
have been added.

The PRR shall also include in its scope a review of the FPGA firmware, held as a separate
review if judged so by the Electronics Coordinator. The details of the scope of the firmware
review at the time of the PRR will be described in a separate document,
The PRR is the last ATLAS-wide review chaired by the ATLAS Electronics Coordinator. Additional Internal TDAQ reviews, described below, are held to monitor the component development up to Installation and Commissioning.

## 19.2.5 Internal TDAQ Reviews

The reviews described below are an internal process of TDAQ and will be organised within the context of the eTDSG. Internal TDAQ reviews are not always mandatory for deliverables, but are carried as needed.

**Requirement Capture and Internal Requirement Review.** The definition of a set of requirements is the first step of a design. A "User Requirement Document" (URD) that contains a list of short, well-defined statements shall be prepared. The URD contains:

* A brief introduction with a description of the UPR system and sub-systems to which the hardware deliverable belongs, or where firmware will be deployed;

* A list of the required functionalities;

* Physics Performance requirements and more generally Performance requirements for all functionalities;

* A list of all required interfaces to integrate the project into the UPR sub-system;

* A list of requirements that the project puts on the rest of the UPR systems and, more in general, on TDAQ.

A high-level simulation model of the component may be available at this stage to define these requirements. The URD is the document used for the definition of the specifications and will be part of the documentation required for the SR. This internal review is mandatory for all deliverables.

**Internal Follow-up Reports and Status/Progress Design Review.** During the construction of the prototypes and/or the evaluation of the Key Performance Indicators, the sub-system coordinator will report regularly to the eTDSG on the status of the prototype design, construction and, eventually, the evaluation of its performance. Once the performance evaluation is completed, a Progress Design Review may be organised to verify the compliance of the prototype design to the specifications. This review is only organised upon request of the UPR management and it might be an essential tool for UPL and his/her management team to verify whether the sub-system is on schedule, or to minimise any schedule and cost impacts deriving from potential design issues at an early stage. In the event that this review fails, a second prototyping cycle may be required through a Follow-up PDR.
**Production Advancement Review.** During the series production, it may be useful to schedule advancement reviews to monitor the progress of production through to its completion. Such a review may help to ensure that any non-conformities or schedule problems arising during production are addressed promptly. This internal review is not mandatory for all deliverables, but is carried as needed.

**Production completion and Installation Readiness Review.** Once the production of a hardware component of the UPR has completed, and all the hardware deliverables have been transported to CERN, its integration with the final, i.e. "production", firmware shall be done at the TMF facility and reviewed before the final deployment of the configured hardware in the USA15 counting room.

The following items shall be included in the scope of the "Installation Readiness Review":

* Full system test in the TDAQ TMF surface facility.

* Full realtime path test with all baseline algorithms implemented.

* System integration tests with detectors electronics.

* Monitoring tasks and other TDAQ software (databases, online).

**Firmware Review at the time of Installation.** A close-out review of a component firmware will be held at the end of the Production phase, to assess the suitability of the firmware for long-term support, covering documentation, version management and regression testing.

Other firmware reviews will be held after installation, during commissioning and during operation, to deploy new trigger algorithms for example, but they are outside the scope of this TDR and are not further described.

## 19.3 Production Schedule and Milestones

Figure 19.2 shows the top-level schedule of the UPR down to the Level-3 of the PBS/WBS.

In order to quantify the criticality of items for the schedule, the planned dates for completion of production for the primary deliverables have been compared to the required dates for their installation. The schedule float is defined as the number of working days available between the end of production and the required date of installation for each deliverable, after accounting for the time required for surface commissioning and testing. Sufficient float is available to accommodate unforeseen delays in all aspects of the project.

A full set of Gantt charts, for each Level-3 item in the WBS, is available in the Planning section of the UPR's EDMS structure [19.2], and will be included in the "Confidential material"
<figure><img src="image_285.png" /><figcaption>Figure 19.2: High-level schedule for the UPR deliverables at the Level-3 of the PBS. The schedule includes design, prototyping, pre-production, production, installation and commissioning.</figcaption></figure>
## 19.3 Production Schedule and Milestones

<figure><img src="image_286.png" /><figcaption>Figure 19.3: Example of expanded Gantt chart for the Level-0 Muon Trigger Barrel and Endcap SL.</figcaption></figure>
for the UCG review in Q1-2018. For illustrative purposes only the fully expanded planning of the Level-0 Muon Trigger (TGC and RPC) SL is shown in Figure 19.3.

## 19.3.1 Level-0 Trigger System's Milestones

Tables 19.1- 19.4 summarise the principal milestones for the Level-0 Trigger sub-systems during the construction.

<table><caption>Table 19.1: Principal milestones for the Level-0 Central Trigger sub-system that includes the following Level-3 deliverables: CTP, MUCTPI, and TTC.</caption><thead><tr><th>WBS</th><th>Descr.</th><th>Milestone</th><th>Date</th></tr></thead><tbody><tr><td>1.1.1</td><td>CTP</td><td>Specification Review</td><td>30.06.2020</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>30.06.2021</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>30.06.2022</td></tr><tr><td></td><td></td><td>Firmware Design Review</td><td>30.06.2023</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>30.06.2023</td></tr><tr><td>1.1.2</td><td>MUCTPI</td><td>Specification Review</td><td>31.12.2021</td></tr><tr><td></td><td></td><td>Firmware Design Review</td><td>30.06.2023</td></tr><tr><td>1.1.3</td><td>TTC</td><td>Specification Review</td><td>30.03.2020</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>31.12.2020</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>03.01.2022</td></tr><tr><td></td><td></td><td>Firmware Design Review</td><td>30.06.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>03.01.2023</td></tr></tbody></table>

<table><caption>Table 19.2: Principal milestones for the Level-0 Calorimeter Trigger sub-system.</caption><thead><tr><th>WBS</th><th>Descr.</th><th>Milestone</th><th>Date</th></tr></thead><tbody><tr><td>1.1.4</td><td>Level-0 Calorimeter</td><td>Specification Review</td><td>30.09.2020</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>03.05.2021</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>01.09.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>01.02.2023</td></tr><tr><td></td><td></td><td>Firmware Design Review (Phase-I FEXs)</td><td>01.01.2024</td></tr></tbody></table>
<table><caption>Table 19.3: Principal milestones for the Level-0 Muon Trigger sub-system (WBS: 1.1.5).</caption><thead><tr><th>WBS</th><th>Descr.</th><th>Milestone</th><th>Date</th></tr></thead><tbody><tr><td>1.1.5.1</td><td>TGC Sector Logic Board</td><td>Specification Review</td><td>02.07.2018</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>17.02.2020</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>25.05.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>24.07.2023</td></tr><tr><td>1.1.5.2</td><td>RPC Sector Logic Board</td><td>Specification Review</td><td>02.07.2018</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>11.02.2020</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>14.07.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>12.10.2023</td></tr><tr><td>1.1.5.3</td><td>MDT Processor Main Board</td><td>Specification Review</td><td>02.07.2018</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>28.02.2020</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>20.07.2021</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>22.12.2022</td></tr><tr><td>1.1.5.4</td><td>MDT Rear Transition Module</td><td>Specification Review</td><td>02.07.2018</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>28.02.2020</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>22.07.2021</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>19.12.2022</td></tr><tr><td>1.1.5.5</td><td>MDT Sector Processor Mezzanine</td><td>Specification Review</td><td>02.07.2018</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>28.02.2020</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>20.07.2021</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>16.12.2022</td></tr><tr><td>1.1.5.6</td><td>NSW Carrier</td><td>Specification Review</td><td>01.03.2020</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>02.10.2020</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>10.01.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>11.10.2022</td></tr><tr><td>1.1.5.7</td><td>NSW Mezzanine Card</td><td>Specification Review</td><td>01.03.2020</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>11.06.2021</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>15.09.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>21.09.2023</td></tr></tbody></table>
<table><caption>Table 19.4: Principal milestones for the Level-0 Global Trigger sub-system (WBS: 1.1.6).</caption><thead><tr><th>WBS</th><th>Descr.</th><th>Milestone</th><th>Date</th></tr></thead><tbody><tr><td>1.1.6.1</td><td>GCM</td><td>Specification Review</td><td>05.10.2019</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>05.10.2019</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>05.02.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>07.01.2023</td></tr><tr><td>1.1.6.2</td><td>PFM</td><td>Specification Review</td><td>25.04.2018</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>13.04.2019</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>04.01.2020</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>06.06.2020</td></tr><tr><td>1.1.6.3</td><td>Data Aggregator (Firmware)</td><td>Specification Review</td><td>10.12.2019</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>23.01.2021</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>10.12.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>08.12.2023</td></tr><tr><td>1.1.6.4</td><td>Trigger Framework (Firmware)</td><td>Specification Review</td><td>10.12.2019</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>09.01.2021</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>05.11.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>14.07.2023</td></tr><tr><td>1.1.6.5</td><td>Trigger Signature (Firmware)</td><td>Specification Review</td><td>10.12.2019</td></tr><tr><td></td><td></td><td>Preliminary Design Review</td><td>01.05.2021</td></tr><tr><td></td><td></td><td>Final Design Review</td><td>31.12.2022</td></tr><tr><td></td><td></td><td>Production Readiness Review</td><td>26.01.2024</td></tr></tbody></table>

## 19.3.2 DAQ System's Milestones

Table 19.5 summarises the principal milestones for the DAQ sub-systems during the construction.

## 19.3.3 EF System's Milestones

Tables 19.6 and 19.7 summarise the principal milestones for the two sub-systems in the EF system during the construction.
<table><caption>Table 19.5: *Principal milestones for the sub-systems of DAQ.*</caption><thead><tr><th>WBS</th><th>Descr.</th><th>Milestone</th><th>Date</th></tr></thead><tbody><tr><td>1.2.1</td><td>Detector Readout</td><td>Specification Review<br>Preliminary Design Review<br>Final Design Review<br>Production Readiness Review</td><td>28.08.2019<br>01.06.2020<br>22.06.2021<br>22.07.2022</td></tr><tr><td>1.2.2</td><td>Dataflow</td><td>Specification Review<br>Preliminary Design Review<br>Final Design Review<br>Production Readiness Review</td><td>05.09.2019<br>15.12.2020<br>18.08.2023</td></tr><tr><td>1.2.3</td><td>Network</td><td>Specification Review<br>Preliminary Design Review<br>Final Design Review</td><td>30.09.2021<br>31.12.2021<br>31.12.2022</td></tr><tr><td>1.2.5</td><td>Online software</td><td></td><td></td></tr><tr><td>1.2.5.1</td><td>EF Farm Orchestration</td><td>Selection of the cluster orchestrator system<br>Prototype System Review<br>Pre-production System Review<br>Production system fully operational</td><td>03.01.2020<br>10.12.2021<br>12.09.2022<br>31.05.2024</td></tr><tr><td>1.2.5.2</td><td>Physics Monitoring</td><td>Requirement Document Review<br>Prototype Test Completion and Review<br>Final Design Review<br>Production Deployment Review</td><td>13.10.2021<br>27.12.2022<br>06.04.2023<br>17.01.2024</td></tr></tbody></table>

<table><caption>Table 19.6: *Principal milestones for the HTT sub-system (WBS: 1.3.1).*</caption><thead><tr><th>WBS</th><th>Descr.</th><th>Milestone</th><th>Date</th></tr></thead><tbody><tr><td>1.3.1</td><td>HTT</td><td>System-wide Specification Review<br>Preliminary Design Review</td><td>01.06.2018<br>30.05.2019</td></tr><tr><td>1.3.1.1</td><td>TP</td><td>Final Design Review<br>Production Readiness Review</td><td>20.04.2022<br>29.03.2023</td></tr><tr><td>1.3.1.2</td><td>TP RTM</td><td>Final Design Review<br>Production Readiness Review</td><td>02.11.2021<br>29.03.2023</td></tr><tr><td>1.3.1.3</td><td>PRM</td><td>Final Design Review<br>Production Readiness Review</td><td>20.11.2021<br>20.03.2023</td></tr><tr><td>1.3.1.4</td><td>AM ASIC<br>AM08 (Prototype)<br>AM09pre (Preproduction)<br>AM09 (Production)</td><td>Preliminary Design Review<br>Final Design Review<br>Production Readiness Review</td><td>20.07.2018<br>17.07.2019<br>14.08.2020</td></tr><tr><td>1.3.1.5</td><td>TFM</td><td>Final Design Review<br>Production Readiness Review</td><td>28.01.2022<br>20.03.2023</td></tr></tbody></table>
<table><caption>Table 19.7: Principal milestones for the EFPU sub-system (WBS 1.3.2).</caption><thead><tr><th>WBS</th><th>Descr.</th><th>Milestone</th><th>Date</th></tr></thead><tbody><tr><td>1.3.2.1</td><td>Servers</td><td>Technology decision</td><td>01.04.2026</td></tr><tr><td rowspan="2">1.3.2.2</td><td rowspan="2">EF Compute Requirements</td><td>Updated EF farm compute requirements</td><td>10.04.2023</td></tr><tr><td>Final EF farm compute requirement for purchase</td><td>28.05.2025</td></tr><tr><td rowspan="2">1.3.2.3</td><td rowspan="2">Accelerator evaluation</td><td>PDR: Design of EF farm using accelerators</td><td>10.05.2024</td></tr><tr><td>FDR: Conclusion on use of accelerators for rolling replacement procurement</td><td>13.01.2025</td></tr><tr><td rowspan="4">1.3.3</td><td rowspan="4">Software</td><td>EF reconstruction software initial prototyping complete</td><td>01.10.2019</td></tr><tr><td>Software Design Reviews (all complete)</td><td>02.08.2024</td></tr><tr><td>Software ready for Run 4 commissioning</td><td>26.02.2025</td></tr><tr><td>Software ready for MC26a</td><td>26.02.2025</td></tr></tbody></table>

References

[19.1] The TDAQ Group, *TDAQ Phase-II Upgrade Schedule Management Plan*, Tech. Rep. ATL2-D-MG-0003, EDMS Id 1874385 v.1, CERN, Geneva, Nov, 2017. https://edms.cern.ch/document/1874385/1.

[19.2] The TDAQ Group, *TDAQ Phase-II Upgrade Project Management Plan Documents*, Tech. Rep. CERN-0000184878, CERN, Geneva, Nov, 2017. https://edms.cern.ch/project/CERN-0000184878.

[19.3] ATLAS Collaboration, *Technical Design Report for the Phase-I Upgrade of the ATLAS TDAQ System*, Tech. Rep. CERN-LHCC-2013-018. ATLAS-TDR-023, CERN, Geneva, Sep, 2013. https://cds.cern.ch/record/1602235.
<figure><img src="image_287.png" /></figure>
# 20 Resources Requirements and Institutional Responsibilities

This chapter documents the tasks and the associated required human resources for each system and sub-system, estimated bottom-up based on input from the institutes. In addition, the UPL and the TDIB chair are surveying the aspirations of the Institutes, and investigating with their Representatives whether the required resources are available locally among those Institutes that have expressed interest in the different systems and sub-systems. Many Institutes have already expressed their intention of contributing to the TDAQ UPR. This is an iterative process that will ultimately complete with an agreement between ATLAS and the Institutes, and by signing a "Memorandum of Understanding" (MoU) document for the TDAQ Phase-II Upgrade Project (UPR) construction after the approval of the UPR by the CERN Resource Review Board (RRB). The MoU will be signed by ATLAS and the Funding Agencies of the participating Institutes. Any formal commitment is taken only after the MoU's signing has been completed.

The chapter is organised in three main sections. An overview of the Resource Management Plan is presented in Section 20.1. A set of tables and charts, summarising the required effort at Level-3 of the PBS and WBS, is shown in Section 20.2. Note that for a given deliverable, the total required effort is estimated corresponding to the needed expertise and skill sets. However, in the tables the effort is divided into the typical professional categories consistent with the composition of the human resources available in those institutes that expressed their interests. More detailed information is being prepared for the UCG confidential material in the form of Excel spreadsheets. The last section of the chapter (Section 20.3) documents in a few tables the interests and the aspirations of the participating Institutes (within and outside the current ATLAS TDAQ collaboration) to participate in the different elements of the UPR program.

## 20.1 Resource Management Plan

As part of the ATLAS organisation, the TDAQ UPR management relies on the Institutes and the Funding Agencies participating in the project to provide and manage directly the financial and human resources required to produce locally, and deliver successfully to ATLAS, the items for which institutes have agreed to be responsible.
The Resource Management Plan is a process to develop and organise, through a set of documents, the resources needed based on the following:

* The UPR management through the UPR-RC and the Level-2/Level-3 managers of each system, determines the financial resources, the staffing and the staffing's competency required to complete the tasks in the project's WBS. An initial plan is prepared and documented for the approval of the process through the LHCC and UCG reviews.

* After the approval of the project, the UPL, assisted by the UPR-RC and the TDIB chair negotiate with the Institutes' Representative and the National Upgrade Programme Managers (or National Contact Physicists) responsibilities for specific tasks.

* The UPR-RC assists the UPL to verify that the resources and the staffing at a production site are sufficient to guarantee low risk of failures before the responsibility is assigned

* The final assignment of responsibilities to an Institute is a task of the UPL based on the CORE contribution, and shall be approved by the TDIB as part of the MoU document's preparation.

* The MoU documents and regulates not only the responsibilities of an Institution for each hardware deliverable, but also for firmware/software required to operate it.

* During the execution of the project, through specific reviews, the UPR-RC monitors the status and the progress of the design and of the construction, and verifies periodically the adequacy of the local resources.

* In case of failures or delays, the UPR management may decide, in concurrence with the TDIB chair, to intervene and negotiate with the Institutional Representative response strategies to recover schedule delays.

## 20.2 Required Manpower Estimate

### 20.2.1 Level-0 Trigger System

Table 20.1 summarises the required effort, in FTEs, for the Level-0 Trigger system in the years 2018-2026. The FTEs are subdivided in the following professional categories: scientists, electronics engineers, software engineers, technicians, and students. Figure 20.1 represents graphically, as stacked histograms, the same information as Table 20.1. Figures 20.4c-20.4f show the required effort for the six Level-3 sub-systems of the Level-0 Trigger system.

### 20.2.2 DAQ System

Table 20.2 summarises the required effort, in FTEs, of the DAQ system in the years 2018-2026 subdivided into the same personnel's categories as in Table 20.1. The same information
<figure><img src="image_288.png" /><figcaption>Figure 20.1: Estimated human resources required in the L0 Trigger system, expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).</figcaption></figure>

<table><caption>Table 20.1: Required effort in the Level-0 Trigger system expressed in FTEs and divided by professional category for the duration of the UPR's construction, installation and commissioning (2018-2026).</caption><thead><tr><th colspan="7">WBS: 1.1 Level-0 Trigger System</th></tr><tr><th>Year</th><th>Scientist [FTE]</th><th>Electronic Eng. [FTE]</th><th>Software Eng. [FTE]</th><th>Technician [FTE]</th><th>Student [FTE]</th><th>All [FTE]</th></tr></thead><tbody><tr><td>2018</td><td>5.7</td><td>12.2</td><td>0.0</td><td>0.1</td><td>5.7</td><td><strong>23.7</strong></td></tr><tr><td>2019</td><td>9.6</td><td>20.6</td><td>0.0</td><td>0.1</td><td>8.9</td><td><strong>39.1</strong></td></tr><tr><td>2020</td><td>7.5</td><td>23.4</td><td>0.0</td><td>0.1</td><td>10.6</td><td><strong>41.6</strong></td></tr><tr><td>2021</td><td>9.7</td><td>29.0</td><td>0.0</td><td>0.1</td><td>9.4</td><td><strong>48.2</strong></td></tr><tr><td>2022</td><td>10.7</td><td>26.4</td><td>0.0</td><td>0.1</td><td>9.9</td><td><strong>47.1</strong></td></tr><tr><td>2023</td><td>12.7</td><td>25.7</td><td>0.0</td><td>0.3</td><td>10.6</td><td><strong>49.3</strong></td></tr><tr><td>2024</td><td>12.1</td><td>15.1</td><td>0.0</td><td>1.4</td><td>7.6</td><td><strong>36.1</strong></td></tr><tr><td>2025</td><td>11.1</td><td>7.0</td><td>0.0</td><td>0.2</td><td>7.4</td><td><strong>25.8</strong></td></tr><tr><td>2026</td><td>7.6</td><td>4.0</td><td>0.0</td><td>0.0</td><td>6.0</td><td><strong>17.5</strong></td></tr></tbody><tfoot><tr><th>TOTAL</th><td><strong>86.6</strong></td><td><strong>163.4</strong></td><td><strong>0.0</strong></td><td><strong>2.4</strong></td><td><strong>76.1</strong></td><td><strong>328.4</strong></td></tr></tfoot></table>
is shown in Figure 20.2. Figures 20.5a-20.5d show the required effort in each year of the

<table><caption>Table 20.2: Required effort in the DAQ system expressed in FTEs and divided by professional category for the duration of the UPR's construction, installation and commissioning (2018-2026).</caption><caption>WBS: 1.2 DAQ System</caption><thead><tr><th>Year</th><th>Scientist [FTE]</th><th>Electronic Eng. [FTE]</th><th>Software Eng. [FTE]</th><th>Technician [FTE]</th><th>Student [FTE]</th><th>All [FTE]</th></tr></thead><tbody><tr><td>2018</td><td>1.4</td><td>1.2</td><td>0.7</td><td>0.0</td><td>1.1</td><td><strong>4.3</strong></td></tr><tr><td>2019</td><td>1.9</td><td>2.6</td><td>1.9</td><td>0.0</td><td>2.5</td><td><strong>9.0</strong></td></tr><tr><td>2020</td><td>2.0</td><td>1.9</td><td>1.9</td><td>0.0</td><td>2.9</td><td><strong>8.8</strong></td></tr><tr><td>2021</td><td>3.3</td><td>2.4</td><td>3.8</td><td>0.0</td><td>4.4</td><td><strong>13.9</strong></td></tr><tr><td>2022</td><td>6.6</td><td>2.8</td><td>3.6</td><td>0.0</td><td>5.4</td><td><strong>18.4</strong></td></tr><tr><td>2023</td><td>3.9</td><td>2.5</td><td>3.6</td><td>0.1</td><td>3.9</td><td><strong>14.0</strong></td></tr><tr><td>2024</td><td>2.6</td><td>1.1</td><td>3.4</td><td>0.9</td><td>2.5</td><td><strong>10.4</strong></td></tr><tr><td>2025</td><td>2.5</td><td>1.8</td><td>3.9</td><td>1.3</td><td>2.3</td><td><strong>11.8</strong></td></tr><tr><td>2026</td><td>0.7</td><td>0.3</td><td>0.6</td><td>0.2</td><td>0.4</td><td><strong>2.1</strong></td></tr></tbody><tfoot><tr><th>TOTAL</th><td><strong>24.9</strong></td><td><strong>16.6</strong></td><td><strong>23.4</strong></td><td><strong>2.4</strong></td><td><strong>25.4</strong></td><td><strong>92.8</strong></td></tr></tfoot></table>

<figure><img src="image_289.png" /><figcaption>Figure 20.2: Estimated human resources required in the DAQ system, expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).</figcaption></figure>

upgrade project's construction, installation and commissioning for the four (4) Level-3 subsystems in the WBS.
## 20.2.3 EF System

Table 20.3 summarises the required resources for the UPR's EF system expressed in FTEs in the years 2018-2026. The FTEs are subdivided by the same professional categories as in Table 20.1. It should be noted that the EF software development requires scientists

<table><caption>Table 20.3: Required effort in the EF system expressed in FTE and divided by professional category for the duration of the UPR's construction, installation and commissioning (2018-2026).</caption><thead><tr><th rowspan="2">Year</th><th colspan="5">Event Filter</th><th rowspan="2">All<br>[FTE]</th></tr><tr><th>Scientist<br>[FTE]</th><th>Electronic Eng.<br>[FTE]</th><th>Software Eng.<br>[FTE]</th><th>Technician<br>[FTE]</th><th>Student<br>[FTE]</th></tr></thead><tbody><tr><td>2018</td><td>14.2</td><td>5.6</td><td>0.0</td><td>3.4</td><td>1.2</td><td><strong>24.3</strong></td></tr><tr><td>2019</td><td>17.1</td><td>6.7</td><td>0.0</td><td>4.2</td><td>1.5</td><td><strong>29.6</strong></td></tr><tr><td>2020</td><td>23.3</td><td>7.3</td><td>0.0</td><td>4.3</td><td>2.1</td><td><strong>37.0</strong></td></tr><tr><td>2021</td><td>28.7</td><td>6.9</td><td>0.0</td><td>3.3</td><td>1.7</td><td><strong>40.6</strong></td></tr><tr><td>2022</td><td>22.0</td><td>7.9</td><td>0.0</td><td>2.8</td><td>2.6</td><td><strong>35.2</strong></td></tr><tr><td>2023</td><td>26.6</td><td>5.8</td><td>0.0</td><td>2.8</td><td>3.5</td><td><strong>38.6</strong></td></tr><tr><td>2024</td><td>25.4</td><td>5.9</td><td>0.0</td><td>1.8</td><td>3.2</td><td><strong>36.2</strong></td></tr><tr><td>2025</td><td>10.6</td><td>5.4</td><td>0.0</td><td>1.8</td><td>3.6</td><td><strong>21.4</strong></td></tr><tr><td>2026</td><td>0.6</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td><strong>0.6</strong></td></tr></tbody><tfoot><tr><td><strong>TOTAL</strong></td><td><strong>168.4</strong></td><td><strong>51.4</strong></td><td><strong>0.0</strong></td><td><strong>24.4</strong></td><td><strong>19.4</strong></td><td><strong>263.5</strong></td></tr></tfoot></table>

<figure><img src="image_290.png" /><figcaption>Figure 20.3: Estimated resources required in the EF system, expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).</figcaption></figure>
with strong software experience. In some cases the tasks may also be done by a software engineer. Figure 20.3 shows graphically the same information as in Table 20.3. Figures 20.6a, 20.6b show the required effort in each year of the upgrade project's construction, installation and commissioning for the two Level-3 sub-systems in the WBS.

<figure><img src="image_291.png" /><figcaption>Figure 20.4: Required human resources expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Histograms are shown for the different sub-systems of the Level-0 Trigger system. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).</figcaption></figure>
<figure><img src="image_292.png" /><figcaption>(c) TTC distribution (WBS 1.1.3)</figcaption></figure>

<figure><img src="image_293.png" /><figcaption>(d) Level-0 Calorimeter Trigger (WBS 1.1.4)</figcaption></figure>

Figure 20.4: Required human resources expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Histograms are shown for the different sub-systems of the Level-0 Trigger system. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).
<figure><img src="image_294.png" /><figcaption>Figure 20.4: Required human resources expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Histograms are shown for the different sub-systems of the Level-0 Trigger system. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).</figcaption></figure>
<figure><img src="image_295.png" /><figcaption>Figure 20.5: Required resources expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Histograms are shown for the different sub-systems of the DAQ system. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).</figcaption></figure>
<figure><img src="image_296.png" /><figcaption>Figure 20.5: Required resources expressed as FTEs for each year of the upgrade project’s construction, installation and commissioning. Histograms are shown for the different sub-systems of the DAQ system. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue) (cont.)</figcaption></figure>
<figure><img src="image_297.png" /><figcaption>(a) EF HTT sub-system (WBS 1.3.1)</figcaption></figure>

<figure><img src="image_298.png" /><figcaption>(b) EF Processing sub-system (WBS 1.3.2)</figcaption></figure>

Figure 20.6: Required resources expressed as FTEs for each year of the upgrade project's construction, installation and commissioning. Histograms are shown for the two sub-systems (HTT, EFPU) of the EF system. Four professional categories are stacked on top of each other: Scientists (blue), Electronics Engineers (orange), Software Engineers (grey), Technicians (yellow), and Students (light blue).
### 20.3 Participating Institute Responsibilities

The formal institutional responsibilities, and the level of participation and financial support by each Institute and Funding Agency to the programme of the TDAQ UPR, will be described in the MoU in preparation for the RRB meeting in the first half of 2018. In this TDR, we limit ourselves to listing the institutes participating in the TDAQ Phase-II UPR and their principal areas of interest.

Table 20.4 lists the Institutes interested in the upgrades of the Level-0 Trigger components. For each system and sub-system a ✓ indicates that the Institute plans to develop and/or provide hardware, firmware, and/or software deliverables. In this context any code that allows for the configuration, control, monitoring and operation of a hardware module during ATLAS data taking is categorised as software deliverable. For Phase-I legacy elements of the project, a ✓ indicates that the institute plans to maintain and/or upgrade firmware and software needed for them to operate in the HL-LHC environment.

<table><caption>Table 20.4: List of participating Institutes and their areas of interest within the Level-0 Trigger system.</caption><thead><tr><th rowspan="3"></th><th colspan="10">Level-0 Trigger</th></tr><tr><th colspan="3">Level-0 Calo</th><th colspan="2">Level-0 Muon</th><th colspan="2">Global</th><th colspan="2">Central</th></tr><tr><th>eFEX</th><th>gFEX</th><th>jFEX</th><th>fFEX</th><th>Fibre Mgmt./Infrastr.</th><th>Barrel SL</th><th>Endcap SL</th><th>MDT TP</th><th>NSW TP</th><th>GCM, MUX, GEP<br>PFM</th><th>Fibre Mgmt./Infrastr.</th><th>CTP</th><th>MUCTPI</th><th>TTC</th></tr></thead><tbody><tr><th rowspan="2">Argentina</th><td>Buenos Aires</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>La Plata</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><th rowspan="2">Germany-BMBF</th><td>Heidelberg KIP</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Mainz</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><th>Germany-MPI</th><td>MPI-Munich</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th rowspan="3">Israel</th><td>Technion</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Tel Aviv</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Weizmann</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

continue ...
...continued

<table><thead><tr><th rowspan="3"></th><th colspan="6">Level-0 Trigger</th></tr><tr><th colspan="2">Level-0 Calo</th><th colspan="2">Level-0 Muon</th><th colspan="2">Global</th></tr><tr><th>eFEX</th><th>gFEX</th><th>jFEX</th><th>fFEX</th><th>Fibre Mgmt./Infrastr.</th><th>Barrel SL</th><th>Endcap SL</th><th>MDT TP</th><th>NSW TP</th><th>GCM, MUX, GEP</th><th>PFM</th><th>Fibre Mgmt./Infrastr.</th><th>CTP</th><th>MUCTPI</th><th>TTC</th></tr></thead><tbody><tr><td>Country<br>Institute</td><td colspan="14"></td></tr><tr><td>Italy</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Napoli</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Roma I</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Roma II</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Japan</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>KEK</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Kobe</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Kyoto</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Nagoya</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Shinshu</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Tokyo ICEPP</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Tokyo MU</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Tokyo Tech</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Netherlands</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>NIKHEF</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Poland</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Cracow IFJ PAN</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Cracow AGH-UST/UJ</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Romania</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Bucharest</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>United Kingdom</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Birmingham</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Cambridge</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>London QMUL</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>RAL</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>USA-DOE</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Argonne</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>BNL</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

continue ...
...continued

<table><thead><tr><th rowspan="3"></th><th colspan="10">Level-0 Trigger</th></tr><tr><th colspan="4">Level-0 Calo</th><th colspan="2">Level-0 Muon</th><th colspan="2">Global</th><th colspan="2">Central</th></tr><tr><th>eFEX</th><th>gFEX</th><th>jFEX</th><th>fFEX</th><th>Fibre Mgmt./Infrastr.</th><th>Barrel SL</th><th>Endcap SL</th><th>MDT TP</th><th>NSW TP</th><th>GCM, MUX, GEP</th><th>PFM</th><th>Fibre Mgmt./Infrastr.</th></tr></thead><tbody><tr><th>Country<br>Institute</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>USA-NSF</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>Boston</th><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>Chicago</th><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td></tr><tr><th>Harvard</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><th>Indiana</th><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td></tr><tr><th>Michigan St.</th><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td></tr><tr><th>U Mass</th><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>Oregon</th><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td></tr><tr><th>Pittsburgh</th><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td></tr><tr><th>UC Irvine</th><td></td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>CERN</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>

Table 20.5 includes the Level-4 items of each DAQ sub-system in the PBS, and also elements of the ATLAS Online Software. The ATLAS Online Software is a large collection of software packages and libraries, which manages and controls the components of the DAQ system and the data-taking environment. Most of the software elements will be maintained and upgraded adiabatically in the next several years by the ATLAS DAQ Operation group. However, some of the core software infrastructure may be significantly redesigned to benefit from the availability of new technologies. Those elements are described in Section 11.7. In Table 20.5 a ✓ in the “Online” column indicates that either an institute is currently involved in operation, Phase-I or Phase-II upgrade tasks, or is planning to contribute in future.
<table><caption>Table 20.5: List of participating Institutes and their areas of interest within the DAQ system.</caption><thead><tr><th rowspan="3"></th><th colspan="6">Data Acquisition (DAQ)</th></tr><tr><th colspan="2">Readout</th><th colspan="2">Dataflow</th><th colspan="2">Networking</th></tr><tr><th>FELIX</th><th>Data Handlers</th><th>Storage Handler</th><th>Event Aggr.</th><th>Readout Net.</th><th>Dataflow Net.</th><th>EF Net.</th><th>Online</th></tr></thead><tbody><tr><td><strong>Australia</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Adelaide</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Sidney</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Czech Republic</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Prague</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Germany-BMBF</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Goettingen</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Wupperthal</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Israel</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Ben Gurion</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Tel Aviv</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><td>Technion</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Weizmann</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Italy</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Bologna</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Pavia</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Netherlands</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>NIKHEF</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Romania</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td></tr><tr><td>IFIN-HH Bucharest</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Russia</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td></tr><tr><td>JINR Dubna</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td></tr><tr><td>NRC Kurchatov Inst. - PNPI</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>✓</td></tr><tr><td><strong>Switzerland</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Bern</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>United Kingdom</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>London RHUL</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>London UC</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

continue ...
...continued

<table><thead><tr><th rowspan="3"></th><th colspan="6">Data Acquisition (DAQ)</th></tr><tr><th>Readout</th><th colspan="2">Dataflow</th><th colspan="2">Networking</th><th rowspan="2">Online</th></tr><tr><th>FELIX<br>Data Handlers</th><th>Storage Handler<br>Event Aggr.<br>Event Bld.</th><th>Readout Net.<br>Dataflow Net.<br>EF Net.</th><th></th><th></th></tr></thead><tbody><tr><th colspan="7" style="text-align: left;">Country<br>Institute</th></tr><tr><th colspan="7" style="text-align: left;">USA-DOE</th></tr><tr><td>Argonne</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>BNL</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Winsconsin</td><td></td><td></td><td></td><td></td><td></td><td>✓</td></tr><tr><th colspan="7" style="text-align: left;">USA-NSF</th></tr><tr><td>MSU</td><td></td><td></td><td></td><td></td><td></td><td>✓</td></tr><tr><td>UC Irvine</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td>✓</td></tr><tr><th colspan="7" style="text-align: left;">CERN</th></tr><tr><td></td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>

Table 20.6 lists the Institutes interested in the components of the HTT sub-system and the EF hardware and software. A ✓ in the EFPU column indicates that the institute is contributing financially to the purchase of EF CPUs. Table 20.6 also lists Institutes contributing to and/or planning to contribute to the core EF software and to the algorithms for the online reconstruction of trigger-object candidates and studies of trigger menus. It is to be noted that both the EF reconstruction developments and the studies of trigger menus are activities not under the sole responsibility of the TDAQ UPR, but they are shared with the ATLAS Upgrade Physics and the ATLAS Trigger Activity groups.

<table><caption>Table 20.6: List of participating Institutes and their areas of interest within the Event Filter (EF) system.</caption><thead><tr><th rowspan="3"></th><th colspan="6">Event Filter</th></tr><tr><th colspan="3">HTT</th><th colspan="3">EF</th></tr><tr><th>Trk. Processors<br>RTM, HTTIF, Infr.<br>PRM<br>AM ASIC<br>TFM<br>SW/Monit.<br>Sim./Perf.</th><th></th><th></th><th>EFPU<br>Core SW<br>Rec. SW/Menus</th><th></th><th></th></tr></thead><tbody><tr><th colspan="7" style="text-align: left;">Country<br>Institute</th></tr><tr><th colspan="7" style="text-align: left;">Argentina</th></tr></tbody></table>

continue ...