dataesr/scientific-paragraphs-categorization huggingface dataset:
The L2PU The L2PU is an application hosting the L2 event selection software. Typically one instance is running per CPU core or per hyper-thread; in October 2011 the total number of instances in the L2 farm was 6312. Each L2PU processes one event at a time. The L2PU is multi-threaded for most of its aspects except for the selection software which runs in a single thread. The L2PU obeys the standard run control commands. Trigger configuration, conditions and calibration data are read from a database using a proxy to avoid overloading the server  (2. 2. 4). Most of these parameters remain constant for the duration of a run, an exception are the trigger menu prescale values that can be changed dynamically while running, typically when the luminosity becomes smaller than a certain threshold. The prescale factors may vary between zero and one, making it possible to disable or enable complete trigger chains.
For each L1Id in the L2 decision message it is indicated whether the event is accepted or rejected. The fraction of accepted events as well as the number of L1Ids per L2 decision message and the frequency with which the event counter of the LTP module is read can be set in the configuration database. -31 - while the destinations of the requests are taken from randomized lists. This ensures successful event building even in case of temporary overload or problems in the network or in the ROS. Built events are dispatched on request to one of the Event Filter nodes  [70]  and, after reception is acknowledged, are deleted from the internal buffers. For commissioning and detector calibration purposes, the SFI can alternatively write the built events on the local disk, possibly registering the files in the SFO-Tier0 handshake database  (2.
JINST 11 P06008 The Tasks are combined to setup the required data flow paths according to the specification in the configuration database, the standard setup is shown in figure  21. The Input Task receives fully built events from the SFIs belonging to the sub-farm. One connection is opened for each SFI and each connection is handled by a dedicated thread. The incoming events are stored in the SharedHeap. References to debug type events are directly forwarded to the Output Task, bypassing the EFPU processing stage. The same route is followed by references to events that only belong to the calibration stream. All the other event references are delivered to the ExtPT Task. The ExtPT Task implements an UNIX domain socket server accessible by all the EFPUs running inside the node. On request of an EFPU the ExtPT Task returns the SharedHeap address of the event to be processed.
SFO-Tier0 handshake The data logging farm is expected to produce and transfer of the order of 5 million files per year. A dedicated mechanism, based on shared database tables, has been developed to guarantee correct bookkeeping and persistence of file meta data, to allow data integrity checks and to seed offline processing of the raw data files. The file meta data stored in the so-called SFO-Tier0 handshake tables includes file sizes, checksums, file locations, writing and transfer statuses as well as run numbers and streaming information. Both the SFO and Castor script access the database and verify that the life cycle of each file, from creation to deletion, is correctly represented in the tables. -40 - The flow of both data and meta data between the online and offline facilities is shown in figure  23.
Data are pushed from the data logging farm via two paths to the offline containers: the raw data files to the CERN mass storage infrastructure and the meta data to the online database cluster. The online database tables are automatically copied to the offline cluster using Oracle streams  (2. 2. 4). In case of offline service failures, raw data files can be buffered on the SFO disk volumes, while the online database cluster can retain the meta data. However, given the importance, and uniqueness in some cases, of the file meta data, a copy is also maintained in local SQLite  [44]  files on each node of the data logging farm. This allows the online database tables to be re-populated a posteriori if there is a failure of the online cluster.
HLT integration of online and offline software components As shown schematically in figure  24 , the HLT software is used in three domains: online, integration and offline. The integration domain is used to begin the validation of new software for online use. Each domain has its own specific requirements, use cases and applications that run HLT algorithms. To cover this wide range of use cases the HLT software is organized in layers. The HLT Steering  [83, 84] , which is part of the HLT Core Software, controls the HLT algorithms. Both the HLT algorithms and the HLT core software adhere to the architecture of the offline software framework Athena  [85] , which itself is based on the component framework Gaudi  [31]. Abstract interfaces ensure that each component can be replaced without changes in the client code.
The Steering Controller decodes the field and initiates the lookup to the trigger database (or conditions database in case of the beam spot) that holds the updated configuration keys (or parameters) for the new luminosity block. The parameters for the updated keys are read by the HLT Steering and applied to the HLT selection. A system of database proxy applications  [45]  ensures that the approximately simultaneous database access of the HLT processes does not give rise to long delays. The updates of the beam spot parameters are not only relevant for the HLT tracking algorithms but in fact critical for trigger algorithms that impose requirements on track impact parameters or decay lengths as these rely on the precise knowledge of the interaction point. The observed variations during a fill, or those from fill to fill, are often larger than the sensitivity of these algorithms.
To track such changes, a dedicated algorithm is executed as part of the L2 trigger that reconstructs primary vertices and publishes histograms of their distributions. These histograms are then aggregated across the farm (2. 12. 1) and analyzed by a separate application for every luminosity block. A resolution correction is applied that is derived online based on the observed displacement of split vertices. Through fitting, the position, size and tilt angles are extracted. The parameters that correspond to each luminosity block are loaded into the conditions database for further monitoring. In case of the HLT parameters, a sliding average over the last 10 luminosity blocks is used to increase the statistical precision without compromizing the time resolution. A separate module then compares this set of parameters with the ones currently stored for the HLT.
Changes are reported when (a) the x, y or z position of the centroid of the luminous region has moved by more than 10% of the corresponding luminous width; or (b) the luminous width has changed by more than 10% in either dimension; or (c) the error on any of the parameters has decreased by more than 50%. Whenever such a change is detected, the parameters are published to IS (2. 2. 2), and a command is sent to the CTP to instruct it that an update should be performed with the next luminosity block. Upon this command, the CTP fetches the parameters from IS, stores them in the conditions database, and then updates its data fragment with the relevant luminosity block number as described above. The same mechanism that is used for the prescales completes the feedback loop on the HLT processors as they receive the data fragment.
The AtlasTrigger and AtlasHLT projects The software packages which belong to the HLT framework form the so-called AtlasHLT software project whereas the AtlasTrigger project holds the HLT Core Software and the HLT algorithms. These software projects are structured like other offline software projects  [99]  and use the same version control, build and testing tools. Because of its nature as an interface between data flow and offline, the AtlasHLT software project is the only ATLAS software project which directly depends on data flow software and on all other ATLAS software projects. Therefore the releases of the AtlasHLT project have to be coordinated with trigger and data acquisition (TDAQ) releases and -45 - with offline releases. Further dependencies exist due to the common use of LCG software  [100]  and the common software project tdaq-common, which mainly hosts the code for the ATLAS raw event data format  [20].
Therefore performance monitoring tools have been developed which help to follow the evolution of execution time and memory usage of the code over the different release cycles in the offline, the AtlasTrigger and the AtlasHLT projects. Validating and deploying a new release is time consuming and storing many, almost identical, releases requires much disk space. To allow for fast bug fixes and the addition of minor new features, a so-called cache named AtlasP1HLT has been introduced. This can contain newer versions of any of the software packages in AtlasHLT, AtlasTrigger or other offline software projects. These will override the existing versions in the dynamic link order. In this way, the underlying code can be changed, but since only a few packages are recompiled, no changes can be made to public interfaces.
The cache uses the same software management and build infrastructure as full releases, but the contents are tightly controlled to ensure that the performance is not accidentally degraded. If required, a simple bug fix can be deployed using this mechanism in less than 24 hours; for new features and other non-trivial changes additional time is spent on testing. The same automatic tests as used for the AtlasHLT project are used for the AtlasP1HLT cache and catch most problems. In addition, before installing a new version of the cache in the online system, O(10 6 ) previously recorded events are processed through athenaMT. py, athenaPT. py and the offline event reconstruction. Besides testing the robustness, the results are also compared against those of previous processings and all differences are required to be understood before a cache is signed off for online deployment.
XPU racks. Two VLANs are defined on all XPU rack data switches: a VLAN for the DataCollection Network (VLAN A for switches attached to Core 1 and VLAN B for those attached to Core 2, see figure  30 ), and the default VLAN for the BackEnd Network (2. 10. 1). While the XPU nodes in the racks connect to ports which are part of both VLANs, the up-links are part of a single VLAN (VLAN A or B for the DataCollection up-link, and the default VLAN for the BackEnd up-link). Thus a single switch is used to emulate two separate "virtual" switches, one connecting the XPU nodes to the DataCollection Network, and the other to the BackEnd Network. This allows for fast migration of HLT processing power from the L2 to the EF system or vice versa, via changes in the configuration database that become active at the "configure step" prior to starting a new run  (2.
The online expert system  (2. 11. 4)  subscribes to the NSG to receive alarms generated by Spectrum. A database (known as the central database) is used to store information from various sources and supply it on demand to several presentation clients. This modular system  [106, 110, 111]   statistics gathering (imposed by the real-time nature of the DAQ/HLT system). Time-stamped data are typically stored in round-robin databases (RRD  [112] ) and consist of network traffic statistics, as well as various environmental and PC specific statistics (2. 14. 2). The network specific data sources are an SNMP (Simple Network Management Protocol) polling engine (apoll), a topology discovery engine and a flow analysis engine. The SNMP polling engine efficiently gathers statistics from all the network ports and stores the results in RRD files, with a polling interval of at most 30 seconds.
The flow analysis engine, based on sFlow  [113]  stores statistically sampled flow data. This proved to be extremely useful for a posteriori troubleshooting. Several flavors of presentation clients have been designed. Some of them are targeted mainly at the operator in charge of monitoring and control of the networking functionality, while others provide in depth information typically accessed by experts. The most complete of them is Net-IS (the Integrated System for Performance Monitoring of the ATLAS DAQ/HLT Network  [110, 111] ): a powerful interface which provides convenient access to all the data stored or referenced by the central database. The information can either be browsed by several predefined hierarchical trees, or it can be directly searched using regular expressions. Multiple time series can be displayed both overlaid on the same plot (see figure  74  for an example) and as an array of "mini-plots".
Overview and architecture The Configuration and Controls services provide support for a flexible description and smooth control of the distributed TDAQ system. They are designed following a layered component model as presented in figure  33 : at the very bottom are external packages and common base libraries -52 -  (described in 2. 2), higher up is a service layer, including the Configuration Database services and the Process Manager. The application layer contains the Run Control, the Diagnostics and Verification System, Online Recovery system, the Shifter Assistant and also a number of Graphical User Interface (GUI) applications. The Configuration and Controls components are used by many other TDAQ subsystems and they are designed in a framework approach, allowing the extension of the core functionality by the subsystems.
The RBAC data (users, roles, roles hierarchies, user assignment to role) are stored in an LDAP (Lightweight Directory Access Protocol)  [115]  database as this solution allows for easy interrogation from the access control software and offers the advantage of storing more information besides the user information. Since autumn 2008 the AM has actively controlled the access to the TDAQ system. The use of the AM has been extended to perform access control also at the operating system level. It controls login on the nodes and access from the outside network to the ATCN and defines rules to allow specific users to execute particular privileged commands on the nodes by means of the "sudo" command. Near the end of 2011 ∼3800 user accounts and ∼440 roles were defined and handled by an AM system consisting of 7 servers, one functioning as backup server.
An example of the load (i. e. number of authorization requests per second) of one server, able to manage ∼800 requests per second, is shown in figure  34. Resource Manager. The Resource Manager controls the usage of the available hardware and software resources, which are described in the configuration database. Most of the resources are allocated or deallocated automatically when an application is launched or terminated via the Process Management service. If resources requested by a process are not available, the service denies to start the process. A GUI application is available which allows resources to be browsed and manually handled. Process Manager. The Process Manager (PMG)  [116]  performs basic process control on behalf of the software components of the DAQ system. It is able to create and destroy software components distributed on the DAQ nodes, and to monitor their basic status (e.
JINST 11 P06008 to minimize the probability of failures. The Process Manager has a small system resource usage. In real life operations the average CPU utilization is less than 15% when launching processes (on a machine with two Intel Xeon 5150 2. 66 GHz CPUs running Scientific Linux CERN 4. 5) and is virtually zero during the running period. The average memory usage is around 10 MB for the Server and less than 4 MB for the Launcher.
Core services: configuration The Configuration service  [117]  provides access to the overall configuration of the DAQ system and to a partial description of the trigger and of the ATLAS detectors, which is stored in the Configuration Database. This database contains all information, from command line parameters to a specification of the overall organization of both hardware and software, needed for running the DAQ software. The service is provided by several components, which are described in more detail in the next sections. DAQ, trigger and detector groups contribute their own part of the configuration description and implement the code for configuration of the applications using common configuration service tools. The description is based on the core object schema (a graphical representation of a part of it is shown in figure  37a ) and can be extended by every group for their needs.
The configuration description is accessed simultaneously by every online application during the boot and configuration phases of a run. The configuration service notifies and reconfigures the clients (TDAQ applications) if any change of the configuration occurs during the run and archives the configurations used so that they later can be browsed by experts and accessed by applications performing offline event processing. The configuration service consists of several components. The configuration description and object persistency is based on the in-house developed package OKS ("Object Kernel Support")  (2. 11. 3. 1). A Remote Database service (RDB) allows OKS objects to be accessed in a distributed environment. The application programming interface includes various Data Access Libraries (DAL) accessing OKS objects and implementing user algorithms on top of configuration data  (2.
Storage and management of configuration data. OKS The main persistent storage for OKS is provided by XML files. The possibility to easily browse, modify and distribute these human readable files is one of the attractive points of OKS. Users can edit the schema (specifying classes) and data (specifying objects), in dedicated GUI applications (figure  37 ). However direct write access to OKS files is not allowed in the TDAQ system. To control the modifications of the database, a special mechanism (OKS Server) is deployed: before  committing a file into the ATLAS repository, the service checks the validity of the file for the ATLAS partition (the entire TDAQ system) and checks user permissions using the Access Manager. The database files are archived in a CVS  [119]  repository for future referencing and browsing of the history of changes, making use of the CORAL (2.
2. 4) interface. Incremental versioning is supported so that only differences between a complete base version of a database and its last revision are stored. OKS provides remote access to the configuration information via the RDB service, making use of CORBA (2. 2. 1),  (4. 7. 4 ). An RDB server provides its clients access to a single database, where the clients can be running on computers without a common file system. To address scalability issues, RDB servers can be organized in a tree-like structure: the master RDB server reads the XML repository and proxy RDB servers receive configuration information from the master. In the ATLAS TDAQ system, there is one RDB server per HLT rack. For efficiency reasons each RDB server keeps results of requests in a cache.
Configuration data access. The configuration data are accessed by applications implemented by developers from different groups. To ease software development the low-level algorithms for accessing the database data are completely hidden behind the high-level API of the DAL (Data Access Libraries) layer. The DAL layer uses the lower abstract config layer to map the OKS database schema to C++, Java and Python classes and to instantiate objects according to the class-definitions. A DAL can be automatically generated from the database schema (by the "genconfig" tool), with it user-defined algorithms facilitating and simplifying complex or typical database queries can be developed. The config layer provides an abstract interface for working with databases and for accessing configuration objects from different programming languages.
Resource Information Service. A TDAQ resource is a part of the ATLAS TDAQ system that can be individually disabled (i. e. masked out from the configuration) or enabled without stopping the data-taking process. Resources can be interconnected, such that changing the status of a single resource may change the status of other resources. Information on whether resources are or were enabled or disabled needs to be available during data-taking as well as during offline data analysis. Statically disabled resources are described in the configuration database. In case of problems during a run the Run Control stop-less recovery mechanism can disable some resources and inform the Resource Information service. This service maintains a list of enabled resources in the Information Service for online access.
For offline access it also stores the list in the conditions database (the interface has been implemented using the COOL framework  [42] ). Experts can browse the resources archive. PartitionMaker. The OKS editor can be used to create the XML files containing the configuration data needed to operate the TDAQ system. However, as entries have to be entered by hand it is a very tedious and error prone task to define large systems where a large number of objects (applications, configuration parameters, computers etc. ) need to be described. For instance, a typical setup of the -58 -.
JINST 11 P06008 High-Level Trigger farms contains about 1500 multi-core nodes and over 13000 applications running on them. To facilitate reliable creation of the OKS files needed for large setups like this the Python tool "PartitionMaker" was developed. The tool allows the details to be specified in an algorithmic way. It describes the system configuration in terms of Python objects directly derived from the database schema, keeping the names of classes, attributes and relationships identical to the schema. In combination with the support by OKS for a "templated" description of a set of applications a compact description of very large partitions is possible. For instance, the layout of the L2 or EF farm can be specified as a group of "racks", each comprising about 30 nodes with 8 cores each.
Diagnostic, testing and verification framework The Diagnostic and Verification System (DVS)  [122]  is based on the CLIPS expert system framework  (2. 11. 4). It allows a user to automate testing of complex TDAQ configurations and to diagnose problems found. The architecture of the system is shown in figure  42. It combines the information about available tests from the Test Repository (an OKS database which is a part of the partition configuration), diagnostics rules from its Knowledge Base and information about the current configuration of TDAQ and makes it possible to automate testing of the configuration using a GUI or a command-line application. It can diagnose problems detected during testing, provided that relevant knowledge is stored in the Knowledge Base. Subsystem experts can add subsystem-specific tests to the Test Repository and custom rules to the Knowledge Base.
The Test Repository database schema  [123]  allows a test (or a number of tests) for any component in the system to be defined and configured. The test is a small application which is launched via the Process Manager service and which verifies the functionality of a component. This functionality is used within the RC framework in two different ways: 1) in the initialization phase, all hardware components are tested before launching any applications on them; 2) in case of a failure, a subset of the system is tested to help localize and diagnose the problem. -62 - The Test Repository database contains a few dozen tests for basic TDAQ hardware (e. g. computers, NIC cards, ROBIN cards) and software components. During start up of the full ATLAS TDAQ system, in total O(10 4 ) test processes are executed in a few seconds.